A developer ran a GPT-2 model using only CSS and variable fonts

The CSS programming challenge

A developer set out to answer a hypothetical question: what is the most absurd programming language? They chose CSS, aiming to run a GPT-2 model without using any JavaScript or WebAssembly for the actual inference. The goal was to see if the browser's styling engine could handle the heavy lifting of a neural network.

The project began with a tiny transformer model featuring only 7 characters in its vocabulary. While this small version worked, the developer quickly realized that standard CSS variables were not suitable for large-scale math. In CSS, variables often just copy text expressions rather than calculating numbers, which would lead to a chain of operations too long for a browser to handle.

To solve this, the developer used the @property at-rule to register variables with specific types. This forced the browser to calculate the values once and store the actual numbers in memory. This allowed the developer to build a functional, albeit small, transformer that could predict the next character in a sequence.

Scaling to GPT-2 limits

Scaling the project from a few hundred parameters to the 124 million parameters of GPT-2 presented a massive hurdle. The developer initially tried to write every multiplication and addition as an explicit CSS expression. However, this approach would have resulted in a stylesheet between 3 and 6 GB in size, which is impossible to load on a standard webpage.

The developer then looked for a way to offload the math to the browser's internal engines. They discovered that variable fonts use a process that resembles a dot product. Specifically, the browser calculates the width of a glyph by multiplying a set of deltas by the axis values provided in the CSS.

By treating the font's delta tables as weight matrices and the CSS axis settings as input vectors, the developer could perform matrix multiplications for free. This method reduced the data size significantly, as the browser's text shaper handles the heavy arithmetic during the layout phase.

Bypassing browser constraints

The implementation faced technical limits, such as Chromium's refusal to load fonts with more than 256 axes. Since GPT-2 requires 768 dimensions, the developer had to split the input across three different fonts. Each font handled a third of the input, and the browser laid them out side-by-side to complete the calculation.

The developer also had to manage precision issues because font deltas cannot contain decimals. They used 9-bit integers to store the weights, which provided enough accuracy without exceeding the maximum width allowed for a single glyph. This required a scaling factor in the CSS to multiply the results back into the correct range.

As shubhampatil.dev reported, the developer used a Python script to pre-generate the stylesheets and font files. This ensured that while the final output was pure CSS, the complex setup of weights and deltas was handled before the user ever opened the page.

Reading the results back

One of the hardest parts of the project was getting the numbers back out of the CSS. There is no direct way to read the width of a glyph as a variable. The developer first tried container queries, but the nesting limits of the browser prevented this from working for the thousands of values needed.

The solution involved using scroll-driven animations to

The path to web trickery

The project was born from a curiosity about the limits of web technology. After seeing a post about how a configuration tool could be viewed as a programming language, the developer decided to push CSS to its absolute limit. They wanted to see if the most common styling language could be forced to perform the heavy lifting of a neural network.

Modern CSS has evolved to include several bells and whistles that bring it very close to turing-completeness. By utilizing these features, the developer was able to navigate the "Wild West" of web trickery. They were ecstatic to find a way forward after spending a long time struggling against the limitations of the stylesheet.

Complex math and animations

Certain operations, like LayerNorm, require all 768 numbers of a vector to be present on a single element to calculate a mean. Because CSS variables do not pass values horizontally to neighboring elements, the developer had to find a way to aggregate these values without hitting browser nesting limits.

The developer realized that animating a variable from 0 to 1 allows it to hold the exact progress of an animation at any given moment. By tying this progress to a scroll-driven animation, they could capture the width of a glyph as a numerical value. This clever use of the timeline allowed the browser to "report" the math results back into the system.