But I can't shake the feeling that by focusing so much on finding the lowest precision models, we are perhaps neglecting an equally-fundamental limitation of our current models: that they are too dense.

I have been struggling to articulate exactly what I mean by this for a while now, until my colleague pointed me to The Lottery Ticket Hypothesis, by Frankle and Carbin, a paper that is probably well known to folks who are more integrated into the machine learning community than I. In brief, the main thesis of the paper is:

A randomly-initialized, dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations.

The authors then show how by iteratively training a network, pruning away the lowest-weighted connections, and then retraining the resultant network from scratch, they can achieve similar performance on a few benchmarks with networks down to 1% of the size they began with. Compare this 99% savings to the gains that we can get from quantization: many models have already been quantized to 8 bits, so even assuming we can get them down to 1 bit, we would be saving only 87.% on what we have today. Worth doing? Sure. But the going is getting harder, and I don't think that we'll be getting all the way to 1 bit on every part of every model.

Back to Frankle and Carbin: they also show that randomly selecting sub-networks (instead of selecting based on results of training larger versions) does not work as well, producing worse accuracy and slower convergence. In their words:

The initialization that gives rise to a winning ticket is arranged in a particular sparse architecture. Since we uncover winning tickets through heavy use of training data, we hypothesize that the structure of our winning tickets encodes an inductive bias customized to the learning task at hand.

I find this very reminiscent of evolutionary methods like NEAT, which likewise attempted to build bespoke sparse networks, one neuron at a time. The difference of course, being that the Frankle and Carbin work backwards from a very dense network and NEAT builds up a network from scratch.

I see this as different from efforts to find sparse attention mechanisms, as sparse attention does not tend to affect the size of the model, but only how expensive it is to operate with it.

I hope that once we're done building larger models, we can get to work on building sparser models that more efficiently reflect the tasks they need to perform than a series of dense operations could. After all, our brains are not homogeneous meshes, but some sort of more contrived pattern of connections. I have this feeling that the optimal model for most problems exists in this sparse-yet-deep subspace, if only we could find it.

In other news: I'm continuing the pattern of keeping long pauses between posts, and changing the website's theme every other post. Bad habits.

I love the idea of using images to compress text. As Andrej points out, the tokenizer is kind of ugly, and it would be cool to have a more natural system. Intuitively, it makes sense that a more optimal way to parse written letters/characters might involve some amount of visual processing, since that's sort of how humans do it, with many people pattern-matching several words (or more) at a time when reading quickly, instead of parsing text as individual letters/tokens.

What I've been trying to figure out is how suprising this compression rate should be. From their abstract:

Experiments show that when the number of text tokens is within 10 times that of vision tokens (i.e., a compression ratio < 10x), the model can achieve decoding (OCR) precision of 97%. Even at a compression ratio of 20x, the OCR accuracy still remains at about 60%.

This sounds pretty good, but there are other ways to compress transformers. DeepSeek-OCR was released using BF16 weights, and since they didn't mention anything about their data precision in the paper, I'm assuming that's what they evaluated on.

On the other hand, weight quantization in the form of low-precision inference has been making substantial progress. I can't find much data on NVFP4, but it seems to promise 4-ish bits per weight and activation (a bit more due to scaling factors) for only slightly reduced accuracy. This means that the 10x-20x advantage claimed by the DeepSeek-OCR paper may just be a 2.5-5x advantage over a 4-bit quantized model. I would be very curious to see whether the image-token-to-text-token compression would remain as high as it is when this model is quantized to such low precision. Perhaps image tokens are just able to use this "extra space" better than text tokens.

Taking this a step further: if this advantage really does decrease from 10x-20x to 2.5x-5x when the model is quantized to 4 bit, then this might imply that text tokens should really be quantized to between 0.8 to 1.6 bits. It seems like a somewhat pleasing result that this line of reasoning should land in the vicinity of 1-bit quantization.

On the other hand, it is also interesting to ask which would better for performance: many 1-bit text tokens or 10-20x fewer 16 bit tokens. Or maybe something in the middle? It seems clear to me that fewer high-precision tokens would be desirable for low-latency decode, given its autoregressive nature, especially given that current GPU hardware targets 16-bit computations. Maybe the tradeoff would be different in other circumstances though.

Anyways, one more disclaimer that this has all been very speculative. I have not carried out any experiments of my own here.

I've been looking at some more bare-bones sites recently, and realizing that I tend to like the feel of these when used for blogs and the like. In particular, I recently ran across Miku's website, and I thought that it seemed great.

So I stole it. And then changed the colorscheme. And then added a basic blogging functionality to the site generator so that I can easily write posts.

If you like something about how this blog looks, please give all credit to Miku. I'm at fault for anything you don't like.

Maybe someday I'll actually use this blog for something other than writing about this blog.

Computer architects can't agree on a way to find the average.

For years, academic practitioners in this field have been arguing about the appropriate way to summarize the average performance of their designs [1]. That is: given \(n\) workloads, if system \(A\) outperforms system \(B\) by \(S_1, S_2, \ldots, S_n\) on each, how much faster should you say system \(A\) is, on average? I think this argument is kind of pointless through.

For the most part, people tend to use the arithmetic mean \(\left(\frac{1}{n} \sum_{i=1}^n S_i\right)\) or the geometric mean \(\left(\sqrt[n]{\prod_{i=1}^n S_i}\right)\). Henessey and Patterson's famous Computer Architecture: A Quantitative Approach advocates for the latter:

Using the geometric mean ensures two important properties:

1. The geometric mean of the ratios is the same as the ratio of the geometric means.
2. The ratio of the geometric means is equal to the geometric mean of the performance ratios, which implies that the choice of the reference computer is irrelevant.

Therefore the motivations to use the geometric mean are substantial, especially when we use performance ratios to make comparisons.

Other people disagree with H&P's reasoning, but I think it's just about as good as it gets.

All Means are Bad

Recently (well, over a year ago now), a paper appeared in IEEE Computer Architecture Letters with the title R.I.P. Geomean Speedup Use Equal-Work (Or Equal-Time) Harmonic Mean Speedup Instead. Its author, Eeckhout, argues that geomean is bad, and people should instead be using what he calls the Equal-Work Harmonic Speedup or the Equal-Time Harmonic Speedup. Eeckhout also presented this work at HPCA 2025 as a part of the Best of Computer Architecture Letters session.

The main thing that Eeckhout seems to dislike about the geometric mean is that it "lacks physical meaning." He claims that using one of his alternatives is better because they have physical meaning. One of the alternatives that he proposes is the Equal-Time Harmonic Speedup ((ETS)), which is just the harmonic mean of the speedups observed on each workload.

$$ETS = \frac{n}{\sum_{i=1}^n \frac{1}{S_i}}$$

Why use the Harmonic Mean instead of the Geometric Mean? Well, if every workload takes the same amount of time to run on the baseline system, the ETS is equal to the total speedup observed when running each of those workloads sequentially [2]. Eeckhout says that this physical meaning provides us with a compelling reason to use something like this over the geometric mean.

But this physical meaning doesn't matter! When I report a score for SPEC, I don't really care about how long it takes to run every single workload in that benchmark in a sequential fashion! It's not like I expect to run a suduko solver (exchange2), then immediately compile gcc, and then perform video compression (x264). I mean, I might run all of these at some point, but certainly not for the exact same amount of time [3]. Although the harmonic mean has a clear physical meaning, it's not one that really matters for many benchmark suites.

Admittedly, I don't really care about the geometric mean of these workloads either. I agree with Eeckhout when he says the geomean doesn't have a clear physical meaning. But it comes down to a choice between an average that doesn't have a clear physical meaning and one whose physical meaning isn't relevant in most situations.

So is there actually a good number to report?

Unless you actually know the precise mix of workloads being run in a real system, any number you report is going to fail to accurately predict the effect of your design on that system. Benchmarks like SPEC are useful insofar as they show general performance patterns, but no matter how you cut it, a single number is always going to fail to provide a perfect comparison between machines when using a general-purpose benchmark suite.

If you do know the particular applications that you care about, and you know their relative importance, then by all means, take their weighted average and you'll be set.

Otherwise, you might as well just using the geomean. It's easy to compare, and everyone else is familiar with it. Use another mean at your own risk: they'll all just be wrong in different ways.

Why are people still talking about this?

I really don't know. Seems like this argument should be over by now.

One of my former mentors once told me that he never looks at an academic paper's evaluation section. If the idea presented in the rest of the paper sounds reasonable, maybe he'll try to apply its innovations to the production design. If the idea sounds rediculous, or addresses a problem he's already solved in another way, then it's of no use, regardless of much speedup the authors might claim [4].

There are other problems that contribute to the industry perspective of academic evaluations. But I share this anecdote just to say: academic computer architects should spend more time coming up with new, inherently interesting ideas, and less time talking about which method of averaging is the least meaningless.

Footnotes

[1]: This argument goes back at least as far at 1986 with the paper How not to lie with statistics: The correct way to summarize benchmark results. Eeckhout provides a good account of this history in his paper.

[2]: Interestingly, even though we can assign a physical meaning to ETS, it can still provide non-intuitive results. For example, if machine \(A\) runs workload 1 twice as fast as machine \(B\) (\(S_1=2\)), but workload 2 only half as fast (\(S_2 = 0.5\)), then computing the ETS of \(A\) over \(B\) yields 0.8 (meaning a slowdown overall). But by symmetry, the ETS of \(B\) over \(A\) is also 0.8. How can both machines be "slower" than the other? Because unlike the geometric mean, the reference machine does matter for ETS! We're assigning different weights to the workloads depending on our starting point!

[3]: Personally, my machines have probably spent much more time running `x264` than compiling gcc or solving suduko. Thanks YouTube.

[4]: There are other problems that contribute to this perception of academic evaluations, beyond the relatively unimportant issue of averaging workload results. In particular, academic microarchitectural simulators are often inaccurate, and baseline systems are often poor comparison points.

I plan on writing mostly about technical topics, but who knows? These things can change.

This blog is currently using the Beautiful Jekyll theme, although the appearance may change slightly as I fine-tune.