Most code-generation benchmarks report a single Python number. It’s a useful headline, but teams ship in Go, Rust, TypeScript, and plenty else — and a Python score, on its own, doesn’t say much about those. We wanted to see what a careful, multi-language evaluation looks like when you run the whole thing in-house, with full control over how it’s measured.
So we took Falcon-H1-34B-Instruct and benchmarked its code generation across six languages on a single NVIDIA DGX Spark.
Why on-prem, and why this setup
Everything ran on one DGX Spark — a compact GB10 Grace-Blackwell system with 128 GB of unified memory. We used MultiPL-E, which takes the same set of programming problems and translates them, with matching tests, into other languages. Because the problems are identical across languages, any difference in the score reflects the language, not the difficulty — a clean way to isolate one variable.
No cloud, no cluster — the model and every line of generated code stayed on the machine.
A finding worth sharing
| Tier | Language | pass@1 |
|---|---|---|
| Reference | Python | 85.4% |
| High | C++ | 76.4% |
| High | JavaScript | 65.8% |
| Medium | Rust | 54.4% |
| Medium | Go | 46.8% |
| Low | Racket | 47.2% |
| Low | Julia | 46.0% |
The broad pattern is one we’d expect from most code models, and it shows up cleanly here: quality tends to track how well-represented a language is in training data. The same problems that solve cleanly in Python and C++ get harder in the less common languages. It’s not a verdict on any single model so much as a property of how these systems are trained — and a good reason to measure the languages you actually care about rather than extrapolate from one.
Why running it yourself matters
The more interesting part, honestly, was a measurement issue we ran into along the way. For one language, a quirk in how the grader’s stop tokens interacted with the model’s output formatting was returning empty answers for a large share of problems — which would have dragged that language’s score down for reasons that had nothing to do with the model’s actual ability. Aligning the protocol fixed it and roughly doubled the score for that language.
That’s the kind of thing you only catch with hands-on, on-prem evaluation: the difference between “the model is weak here” and “the harness is misconfigured here” is easy to miss from the outside, and it can quietly change your conclusions. Getting it right is most of the work, and most of the value.
Takeaway
The practical point is narrow but useful: a model’s headline Python score doesn’t transfer evenly to other languages, and the gap is large enough to matter when you’re choosing a model for a Go or Rust codebase. Measuring the languages you actually use — rather than extrapolating from one — is the only way to know where you stand. And as the empty-answer issue showed, part of that work is making sure you’re measuring the model and not the harness.
Full methodology, per-language results, and the harness fix are in the companion whitepaper: Cross-Language Code Generation with Falcon-H1-34B-Instruct.