A model’s score on a public benchmark is the product of two things: the model, and the way it was measured. Most of the conversation focuses on the first. In our experience, the second is where a surprising amount of the signal hides — and the only reliable way to see it is to run the evaluation yourself, end to end, on hardware you control.
So we did. We took Falcon-H1-34B-Instruct and ran it through a math-reasoning benchmark on-prem, on a single NVIDIA DGX Spark, and paid close attention to how the numbers were produced rather than just what they were.
Why on-prem, and why this setup
The whole evaluation ran on one DGX Spark — a compact GB10 Grace-Blackwell system with 128 GB of unified memory. The 68 GB model loads entirely on-device, no offloading. 3,000 problems across two full evaluation passes, about four hours of compute — no cluster, and nothing left the machine.
That last part matters more than it sounds. Running evaluation in-house means you control every variable: the serving stack, the decoding settings, the prompt format, the token budget. When a result looks surprising, you can open it up and find out why instead of guessing. That’s hard to do through a hosted API.
For the benchmark we used GSM-Symbolic, which regenerates grade-school math problems with fresh names and numbers (and optionally extra reasoning steps). Because the surface changes each time, it’s a cleaner read on reasoning than a fixed, widely-circulated test set.
A finding worth sharing
We ran each problem set two ways: once reproducing the common evaluation-harness defaults, and once giving the model more room to finish its working.
| Variant | Harness defaults | With more room |
|---|---|---|
| renamed problems | 77.6% | 98.2% |
| +1 reasoning step | 61.2% | 94.0% |
| +2 reasoning steps | 28.8% | 93.0% |
The left column looks like a model that struggles as problems get deeper. But that reading turns out to be mostly about the harness, not the model. The widely-used default caps output at 256 tokens — a sensible budget years ago, less so for today’s longer chains of thought. On the hardest problems, a large share of answers were simply cut off mid-calculation before the model could write down its result. Give it room to finish, and the apparent drop-off largely disappears.
We’d put it as a soft conclusion rather than a hard one: on these problems, a good deal of what can look like a reasoning ceiling may instead be a measurement ceiling. The practical lesson is that token budgets and prompt formats deserve as much scrutiny as the model itself — and you mostly only catch this by running the evaluation yourself.
Takeaway
None of this means the benchmark is wrong, or that the model is secretly better than reported. It means a single score compresses two different things — the model’s ability and the way it was measured — into one number, and you can’t tell them apart without looking at how it was produced. Running the evaluation where every setting is visible and adjustable is what makes that separation possible. On these problems, most of the apparent depth-driven decline turned out to be a token budget, not a reasoning limit.
Full methodology, serving configuration, and per-variant diagnostics are in the companion whitepaper: Evaluating Falcon-H1-34B-Instruct on GSM-Symbolic.