Summary
Coding models learn from public code, and public code is distributed very unevenly across languages and problem domains. This study asked whether that distribution predicts agent performance: is an agent measurably better at web work in TypeScript, or data work in Python, because so much more of that specific combination exists to learn from?
Part one measured the distribution. Using The Stack v2, the deduplicated permissively-licensed corpus assembled to train StarCoder2, we built a language × task-category matrix across ten languages and 3,000 sampled repositories, classified by dependency manifests and README content, with a validated classifier behind every number.
Part two tested the prediction. Twelve language-neutral programming challenges, four languages, three repetitions, 144 controlled runs of a headless coding agent, scored by hidden black-box tests written before any solution existed.
The central hypothesis was not supported. The agent solved essentially every challenge in every language (136 of 144 runs perfect), so task-specific corpus volume could not be shown to affect success at all. Cost did vary reliably (identical work cost roughly 24% more in TypeScript than in Python), but that variation correlates with language verbosity (ρ = +0.60) far better than with task-specific corpus share (ρ = −0.10), and runs opposite to raw language popularity. The one clean corpus effect appears in TypeScript, whose costs order almost perfectly by its own highly lopsided corpus (ρ = −0.90) and whose worst result is in machine learning, a domain comprising 0.0% of its public code.
We read this as evidence for a threshold rather than a dose-response relationship: above some volume of examples, more stops helping; near zero, the penalty is real and measurable.
1. Motivation and hypotheses
The practical question: when a coding agent will write most of the code, should language choice change?
The intuition behind “yes” is that models learn from public code, so competence on a task/language pair should track how much of that pair exists. Split apart, that intuition contains four distinct claims, which make different predictions:
- H1, volume. Agents are better in languages with more training code, full stop. Predicts one language wins everything.
- H2, task match. What matters is the volume of code in language L solving problems like T. Predicts the winner changes with the task.
- H3, uniformity. A language with one idiomatic way to do things produces a low-entropy corpus and therefore more consistent output.
- H4, fewer boundaries. A single language across a whole system reduces contract mismatches.
Part one produces the independent variable that H1–H3 need. Without it, “the language with the most code for this task” has no defined meaning.
Both parts were pre-registered: nine predictions written down before part one ran, ten before part two. Scoring against those predictions is in §7 and §11.
2. Part one: method
Six seeded, resumable pipeline phases:
- Frame. Stream all 64 shards (~33 GB) of The Stack v2
train-smol-idsmetadata, retaining repository name, pinned commit, per-file language and size, star events and creation date. Each shard is deleted after processing, so peak disk stays around 520 MB. Full-corpus pass, no sampling at this stage. - Sample. 300 repositories per language, uniform random from the eligible set, seeded. Ten languages → 3,000 (repository, language) pairs.
- Fetch. Dependency manifests and a bounded source sample from GitHub at each repository’s pinned commit, so the code classified is the code that was in the corpus rather than the current HEAD. Rate-limit aware, fully cached. 2,895 of 3,000 fetched successfully; the remainder were deleted or renamed since the 2023 cutoff.
- Classify. Deterministic scoring of manifests and imports against a committed 1,334-entry package→category map. Fourteen fine categories rolling up into seven coarse buckets. Originally specified with no LLM anywhere in this path.
- Validate. 200 repositories drawn stratified; README, file tree and entry-point file heads gathered as labelling materials; labels assigned blind from those materials only, never from classifier output; classifier scored against them.
- Analyze / report. Matrices in repositories, files and bytes; lift; focus (normalized entropy); framework concentration; seeded bootstrap intervals.
Units are reported in repositories, files and bytes. Bytes is the headline because models train on tokens, but repository counts are closer to how people think about “what gets built,” and the two are reported side by side.
3. Part one: results
Within-language share of bytes, coarse categories, hybrid classifier (§4):
| language | web | data_ml | tools | infra_sys | apps_other | library | unknown |
|---|---|---|---|---|---|---|---|
| Python | 13.6% | 22.2% | 13.6% | 7.1% | 7.7% | 18.3% | 17.5% |
| JavaScript | 65.6% | 0.1% | 3.4% | 4.7% | 2.3% | 6.5% | 17.4% |
| TypeScript | 77.0% | 0.0% | 1.8% | 0.9% | 9.2% | 8.1% | 2.9% |
| Go | 10.9% | 1.6% | 27.5% | 15.9% | 1.0% | 32.4% | 10.7% |
| Java | 6.9% | 0.0% | 0.3% | 4.2% | 6.4% | 1.8% | 80.4% |
| Rust | 5.0% | 0.8% | 14.2% | 43.2% | 3.8% | 20.6% | 12.4% |
| C# | 12.0% | 5.5% | 4.0% | 1.3% | 20.6% | 35.1% | 21.5% |
| Ruby | 31.9% | 0.8% | 9.3% | 12.6% | 1.6% | 39.4% | 4.5% |
| PHP | 31.1% | 11.3% | 0.8% | 0.1% | 0.8% | 18.2% | 37.7% |
| Kotlin | 3.2% | 0.1% | 0.7% | 0.4% | 83.4% | 8.2% | 4.0% |
3.1 Findings
Two derived measures appear throughout: lift is a cell’s observed share divided by the share expected if language and task were independent (1.0 = no association; 0.0 = the combination essentially does not occur), and focus is one minus the normalized entropy of a language’s category distribution (1.0 = a single-purpose language, 0.0 = perfectly even spread).
Python is the least focused language measured. Focus score 0.04, last of ten, with the next-lowest at 0.17. Its bytes distribute almost evenly across data/ML, library, web and tooling. This is the dilution mechanism from H2, measured directly: the token “Python” carries far less information about what follows than “TypeScript” does.
Data and ML are a near-monopoly. Lift for Java × data_ml is 0.00, Kotlin 0.02, JavaScript 0.03. Public data-science code is overwhelmingly Python, to a degree that leaves the other languages with essentially no representation.
Go’s corpus contradicts Go’s reputation. Measured: library 32.4%, tools 27.5%, web 10.9%. Two of the nine pre-registered predictions concerned Go and both failed: it is neither the most focused language (0.17, near the bottom) nor the highest-lift infrastructure language. Go is a language in which people publish libraries and command-line tools, and about which people talk as though it were a backend-services language.
Framework monoculture is also inverted from folklore. The tightest web monocultures are Java/Spring (83.3% of web repositories) and C#/ASP.NET (78.7%), not Rails (54.8%) or Laravel (66.7%). JavaScript web is the most fragmented (React at 56.7%, Herfindahl 0.39). For H3, the interesting uniformity contrast is Java/C# versus JavaScript, not Ruby versus JavaScript.
Pre-registered scoring: 4 supported, 2 not supported, 1 mixed, 1 insufficient data, 1 reserved for author judgment.
4. Part one: the classifier had a measurable ceiling
The validation step changed the study’s numbers materially, so it is reported in full.
The classifier was deliberately LLM-free: manifests, import regexes and a
committed YAML map, chosen so that any claim of the form “this repo is web
backend” could be audited by a reader. Validating it against 200 blind-labelled
repositories measured it at macro-F1 0.27, accuracy 0.44, with unknown
precision at 0.19.
The failure mode was systematic rather than random. A repository with no
dependency manifest produces no signal, so it lands in unknown even when its
README states its purpose in the first sentence. Django projects without a
requirements.txt, pygame games, machine-learning paper code: all unknown.
We then measured two additional signals against the same blind labels:
| track | macro-F1 | accuracy |
|---|---|---|
| dependency manifests only | 0.27 | 0.44 |
| README/description via a cheap LLM (Haiku) | 0.60 | 0.65 |
| hybrid (manifest primary, LLM fallback) | 0.60 | 0.68 |
GitHub Topics were evaluated as a third signal and rejected as a classifier: only 179 of 3,000 repositories carry any topics, and only 90 map to a category. Retained as an agreement check only (63% agreement with the LLM track).
The hybrid was adopted, which amends the study’s original “no LLM in classification” constraint. The justification is that the constraint existed to make claims auditable, and the hybrid preserves that: every manifest-derived classification still carries its matched packages, so framework-concentration analysis is unaffected, and each track’s accuracy is separately measured and reported rather than asserted.
Effect on the headline numbers. Python’s unknown share collapsed from
60.5% to 17.5%, exposing the 22% data/ML share that manifest-only classification
could not see. Go’s library share moved from 0% to 32.4% (Go modules carry no
structural library hint; a gap flagged on day one of the build and quantified
here). Per-language accuracy now ranges 0.45–0.89, with Go and Kotlin clearing
the study’s 0.85 target and the rest honestly reported as below it.
4.1 A cautionary note on bytes-weighting
apache/buildr alone constituted 24.5% of Ruby’s sampled bytes. A single
mis-weighted dev-dependency (jekyll, present only to build the project’s
documentation site) classified it as web frontend and moved Ruby’s entire
web share by roughly 25 percentage points until corrected. Bytes-weighted cells
at n=300 are highly sensitive to individual large repositories; repository-count
matrices are not, and are the safer basis for headline claims.
5. Part two: design
Languages (4): Python, TypeScript, Go, Ruby. Python as the H1/H2 crux
(largest volume, lowest focus); TypeScript as the specialist with volume;
Go as the folklore-versus-corpus test; Ruby chosen over C# because C#’s 21.5%
unknown rate made its corpus profile the least trustworthy of the candidates.
Kotlin was excluded because none of the testable buckets is mobile, where 83% of
its corpus lives.
Buckets (6): web, data_ml, tools, infra_systems, library, plus general:
pure algorithmic challenges with no ecosystem dependence, serving as a control.
Every bucket result is read as a delta against the same language’s general
baseline, so that language-level ability is separated from task-specific effects.
Scale: 2 challenges per bucket × 4 languages × 3 repetitions = 144 runs.
Agent: Claude Code headless (claude -p), Sonnet, one campaign, one model,
one machine.
5.1 Fairness controls
The design problem is running the same task four ways with nothing leaking between them.
- Identical prompts. One spec per challenge with the language name substituted; byte-identical otherwise. The SHA-256 of every rendered prompt is recorded per run.
- Hidden tests, written first. 401 tests across 12 challenges, written in
Python before any solution existed, exercising each solution only through
language-neutral interfaces: a
run.shthe agent must produce, plus stdin/stdout, files, HTTP, or a TCP socket. The identical suite scores all four languages. - Double validation of every challenge. Each suite must pass 100% against a
stdlib-only reference solution (proving the tests are satisfiable and agree
with the spec) and must fail against a do-nothing
run.sh(proving the tests discriminate). - Difficulty calibration, verified rather than assumed. An independently written naive solution to the booking-engine challenge passes all 25 functional tests and fails only the performance gate; a stream-slurping implementation of the JSON query tool passes everything except the streaming test; the churn model’s tiers were calibrated against measured AUCs of 0.774 (raw features), 0.872 (engineered) and 0.900 (true-logit ceiling).
- Isolation. Fresh workspace per run, located outside any directory tree containing the tests or the harness; no web tools in the agent’s toolset; operator customizations (CLAUDE.md, hooks, MCP servers) disabled. A transcript audit flags any reference to the tests directory: zero flags across 144 runs.
- Seeded interleaved run order, so no language clusters in wall-clock time.
Known confound: package installation during a run is permitted, since solutions must run offline afterward. Ecosystem richness is arguably part of what “language” means here, but it means these numbers are not pure language competence.
Metrics: hidden-test pass rate (primary as designed); billable tokens (input + output + cache writes), with cache reads recorded separately since they run ~27× billable in agentic loops; turns; wall-clock decomposed into API wait and local toolchain time; non-blank non-comment lines of code; and a fixed-rubric LLM judge scoring correctness-risk, idiomaticity, error handling and structure.
6. Part two: results
Campaign: 144/144 completed. No timeouts, no retries, no audit flags. $49.52, 5,559,505 billable tokens, 6.8 hours of cumulative agent time, 2h14m elapsed at concurrency 3.
6.1 Accuracy saturated
136 of 144 runs passed every hidden test. The worst run in the campaign scored 0.97. Pass rate is saturated, with effectively no variance, and the primary metric of the design is therefore uninformative: a ceiling effect from the instrument’s point of view; from any other point of view, simply a very capable model.
The saturation is not an artifact of easy challenges. The churn-model challenge was explicitly calibrated so that a raw-feature logistic regression scores AUC 0.774 and fails its top tier; all four languages cleared the top tier in all three repetitions, performing feature engineering unprompted.
Six of the eight imperfect runs form two perfectly reproducible failure patterns (the other two were isolated single-test misses), which makes them more interesting than their number suggests:
- TypeScript fails the expression-evaluator’s performance test in all three repetitions, identically: output correct for exactly 65,525 characters and then truncated. This is a 64 KB buffer boundary, Node’s asynchronous stdout not flushing before process exit. A deterministic, language-specific defect that the agent did not detect because the program appears to work.
- Python fails the task API’s cross-tenant authorization test in all three repetitions, returning 403 or 422 where the specification requires 404 so that resource ids do not leak existence. A specification-compliance failure rather than a capability one.
6.2 Cost differs reliably between languages
Each run normalized by its own challenge’s cross-language mean, so task difficulty cancels; 1.00 is the average language on that task. Bootstrap intervals, n=36 per language:
| language | relative cost | 95% CI | mean turns | mean LOC |
|---|---|---|---|---|
| Python | 0.88 | 0.80–0.98 | 14.1 | 215 |
| Ruby | 0.95 | 0.87–1.02 | 15.1 | 210 |
| Go | 1.07 | 1.00–1.16 | 16.1 | 388 |
| TypeScript | 1.09 | 1.01–1.18 | 23.2 | 285 |
Python and TypeScript both have intervals clear of 1.00; the gap between them is 24%. Per-challenge spreads range from 1.09× to 2.41×; in absolute terms, per-run means ranged from ~14,800 billable tokens (Python on the backup CLI) to ~72,200 (Go on the JSON query tool). TypeScript’s turn count, 64% above Python’s, is the sharpest single discriminator in the data.
By bucket (* = interval excludes the all-language average):
| bucket | Python | TypeScript | Go | Ruby |
|---|---|---|---|---|
| general | 1.01 | 1.09 | 1.11 | 0.79* |
| web | 0.84* | 0.98 | 1.21* | 0.98 |
| data_ml | 0.85 | 1.32* | 0.85 | 0.98 |
| tools | 0.89 | 1.11 | 1.08* | 0.92 |
| infra_systems | 0.82* | 1.03 | 1.19* | 0.96 |
| library | 0.90 | 1.02 | 1.01 | 1.07 |
7. Testing three explanations
7.1 Corpus match (H2): not supported
Spearman correlation between part one’s within-language corpus share and part two’s relative cost. Negative would support H2 (more corpus → cheaper):
| scope | ρ |
|---|---|
| pooled, all 20 (language, bucket) cells | −0.10 |
| within TypeScript | −0.90 |
| within Ruby | +0.40 |
| within Go | 0.00 |
| within Python | +0.50 (inverted) |
The extremes contradict H2 directly. Python’s cheapest bucket is infra_systems, where it holds 7.1% corpus share; Go’s most expensive is infra_systems, where it holds 15.9%: more than double, and it performs worse.
The exception is instructive. TypeScript × data_ml is the single most expensive cell in the study (1.32), and data_ml is 0.0% of TypeScript’s corpus. TypeScript is also the only language whose internal cost ordering tracks its own corpus almost perfectly.
7.2 Raw popularity: not supported, and inverted
Per GitHub Octoverse 2025, TypeScript is now the most-used language on GitHub by monthly contributors (2.64M, +66% year over year), having overtaken Python. It is the most expensive language in this study. Ruby, which JetBrains’ 2025 ecosystem survey places in long-term decline, is the second cheapest. Rank-correlating popularity against cost gives ρ = −0.40 in the direction of “more popular, more expensive,” though n=4 cannot support a real correlation in either direction. The qualitative point stands regardless: the two extremes of popularity are inverted relative to the two extremes of cost.
7.3 Verbosity and ceremony: supported
Mean lines of code written per run: Ruby 210, Python 215, TypeScript 285, Go 388. Correlation with relative cost: ρ = +0.60 across languages, and ρ = +0.58 across all 144 individual runs.
This is partly mechanical (more code means more output tokens), and we state that rather than hide it. What makes it non-trivial is that the cause of the line-count difference is not the model’s choice but the language’s ceremony: Go’s explicit error returns, its absence of comprehensions, TypeScript’s type declarations and configuration surface. These are stable properties of the languages, and they account for the data better than either alternative explanation.
The result has a clean structural reading: the two terse, dynamically typed languages occupy the two cheap positions on every effort metric collected (tokens, turns, and lines), while the two statically typed, structurally disciplined languages occupy the two expensive ones. With four languages this 2-versus-2 split cannot separate terseness itself from static typing (the properties co-vary perfectly in our sample), so we describe the combined bundle as ceremony and make no claim about which component drives the cost.
The finding sits in direct tension with a prominent industry narrative. GitHub’s Octoverse 2025 report attributes TypeScript’s rise to the top of its language rankings partly to typed languages making agent-assisted coding more reliable. At this task size we could not detect that reliability benefit anywhere: correctness saturated in all four languages, typed or not (§6.1), while the token cost of the added structure registered on every run. This does not refute the narrative at larger scales, where compiler feedback over a large codebase plausibly earns its keep; it does mean the benefit was unmeasurable exactly where the cost was unmissable.
7.4 Interpretation: threshold, not dose-response
The pattern across §7.1–7.3 is consistent with a threshold relationship rather than a linear one. Above some sufficient volume of examples, additional corpus stops improving performance; near zero, the penalty is real. Python clears the threshold in every bucket, so it performs well everywhere and its internal distribution ceases to matter, which explains its inverted ρ. TypeScript falls below the threshold in exactly one domain and is measurably penalized there.
This is interpretation, not demonstration: four languages cannot establish a threshold curve, and identifying where such a threshold sits would require languages sampled across a much wider range of per-domain volumes. It is offered as the reading that fits the observations, and as the hypothesis a follow-up study should be designed to test.
8. What did not discriminate
Reported because null results constrain interpretation.
The LLM judge could not distinguish the languages. Mean rubric scores: Python 7.57, TypeScript 7.51, Ruby 7.47, Go 7.33: a 0.24 spread on a 1–10 scale, with per-axis spreads no wider (correctness-risk 7.03–7.72, idiomaticity 7.47–7.64). A pre-registered prediction of a Python idiomaticity advantage from post-training skew is unsupported. Either the solutions genuinely are of similar quality, or a same-family judge on a fixed rubric is too blunt an instrument. The judge column supports no conclusion in this study.
Wall-clock time did not discriminate, despite tokens doing so: Ruby 161 s, Python 170 s, Go 174 s, TypeScript 180 s. That is a 12% spread against a 24% token spread. Wall-clock is dominated by API latency unrelated to the language, which is why tokens and turns are the effort metrics we lead with.
Toolchain time did not discriminate, and an earlier impression was wrong. A single pilot run suggested Go paid a heavy compilation penalty (127 s of a 206 s run). At n=36 this does not hold: Go has the lowest mean local time (10.8 s, 6.2% of wall) and Python the highest (26.0 s, 15.3%), though Python’s median is 2.2 s, so its mean is driven by a small number of runs re-executing expensive performance cases. Toolchain time is outlier-driven and separates nothing.
9. Threats to validity
- Ceiling effect. The binding limitation. Pass rate cannot test H2 when nearly every run is perfect. All accuracy-based conclusions are therefore “not demonstrated” rather than “disproven.”
- One model, one point in time. Sonnet, August 2026. These results describe this agent; capability changes rapidly.
- Task size. Well-specified programs of a few hundred lines. Nothing here extends to multi-file work on existing codebases, which is where most real agent use lives.
- Two challenges per bucket. Where a bucket result is driven by one challenge (most of TypeScript’s data_ml penalty comes from a single challenge), task idiosyncrasy and bucket effect are not separable. Run-to-run variance was small; task-to-task variance was large.
- Package installation is permitted, so ecosystem richness is inside the measurement rather than controlled out.
- The corpus is a proxy for training data, not training data. No lab publishes its mix. Models also learn from documentation, forums, licensed and synthetic code, and are post-trained on task distributions that skew heavily toward Python.
- Permissive licenses only, September 2023 cutoff. GPL-heavy domains are under-represented; post-cutoff trends are absent.
- Blind labels are LLM-proposed and not yet human-reviewed, so part one’s reported F1 figures inherit that error.
- Java’s corpus row is the least trustworthy at 80.4%
unknowneven after the hybrid classifier: enterprise Java in this corpus genuinely lacks READMEs. - The judge is same-family with the agent and cannot be blind to language, since the code reveals it.
10. What we would do differently
- Calibrate difficulty against the agent, not against a reference solution. Every challenge was verified hard enough that a naive implementation fails, but “naive” was our definition, and the agent is not naive. One throwaway run against the real agent before authoring the remaining eleven challenges would have exposed the ceiling in an hour rather than after a full campaign.
- Match the control for difficulty, not just for ecosystem-independence.
The
generalbucket was intended as a neutral baseline but turned out to contain the most expensive tasks in the study, so every bucket appears “cheaper than control” for every language. - Record cheap metrics from the start. Pass rate was the designed headline and had no variance; token cost, added mid-study, carried the entire analysis. Metrics cannot be added retroactively to runs already paid for.
- Ask the naive comparison first. The initial analysis compared each language against its own control and nearly produced a “no difference” conclusion. The direct language-versus-language comparison on identical tasks (the question anyone actually asks) was implemented later and showed a clean 24% gap.
- Prefer more challenges over more repetitions. Run-to-run variance was small relative to task-to-task variance; the marginal third repetition bought less than a third challenge per bucket would have.
- Add multi-stage projects. Every task here was single-shot, which is the scenario ceremony is not designed for. Structured languages bet that their discipline pays off as a codebase grows and must survive repeated extension, a bet our tasks never allowed to pay out, so the structured pair bore the cost with no opportunity to collect the benefit. A staged design (build, then extend, then extend again across sessions) is the fair test of that bet, and the plausible place for the cost ranking to compress or invert.
11. Conclusions
Scoring the ten pre-registered part-two predictions: three clearly supported, two partly, five not supported. The design’s central prediction, that corpus match would move the numbers, did not hold.
| # | prediction (abridged) | verdict |
|---|---|---|
| 1 | Python’s data/ML advantage the most positive of any language; others negative | not supported: accuracy saturated; on cost, Python and Go tie cheapest there |
| 2 | TypeScript peaks on web, with the largest web advantage of the four | partly: web is TypeScript’s own best bucket, but Python is cheaper there |
| 3 | Go performs better on tools than on web | supported: 1.08 vs 1.21 relative cost |
| 4 | Go shows a positive library effect | not supported: flat (1.01) |
| 5 | Ruby’s web result positive and better than Go’s | partly: Ruby 0.98 vs Go 1.21, but not positive against its own baseline |
| 6 | Control-bucket spread across languages within 15 points | supported: pass rates all ≥ 0.99 |
| 7 | At least one bucket’s winner differs from the control’s winner | supported: Ruby wins the control; Python wins web and infra |
| 8 | Judge idiomaticity favors Python | not supported: judge spreads are indistinguishable (§8) |
| 9 | Token cost lower in each language’s corpus-strong bucket | not supported: pooled ρ = −0.10 |
| 10 | Go terse per unit of work on tools | not supported: Go is the most verbose language measured |
What the study does support:
- For well-specified tasks of this size, language choice did not determine whether the agent succeeded. It succeeded in all four languages, including on a machine-learning task calibrated to be hard. That near-universal success is itself the study’s clearest capability finding: the models are simply good. A frontier agent now reliably produces working, externally verified software across ecosystems in a single pass at this task size. The conventional reasons to choose a language (team familiarity, deployment target, maintenance horizon) are not displaced by agent-assisted development; for an individual practitioner, the dominant factor this leaves is familiarity, since the human’s ability to read and steer the output is the constraint the agent does not remove.
- Language choice does affect cost, by roughly 24% between the extremes, and the split is structural: the terse, expressive languages beat the ceremonial, structured ones on every effort metric collected. Cost tracks verbosity (ρ = +0.60) far better than corpus composition (ρ = −0.10) or popularity (inverted). Python’s advantage, on this reading, is coverage without ceremony: above the sufficiency threshold in every tested domain, nothing foreign to it, while being among the tersest languages measured. Neither property is about the size or diversity of its ecosystem as such; a hypothetical language with the same coverage and terseness but a tenth of the volume would, on this account, perform the same. Meanwhile the widely claimed agent-reliability benefit of static typing was undetectable at this task size, while its cost was visible on every run (§7.3). Cost should not be read as quality: the expensive languages’ output passed the same tests and scored indistinguishably on the quality rubric (§8). Higher cost is purely more effort to reach an equivalent result.
- The corpus effect is real but confined to the tail. It is invisible in the middle of the distribution and clear at the edge: the single most expensive cell in the study is the language-domain pair with 0.0% corpus share. Practically: taking a language somewhere its ecosystem never goes is the case where this matters.
- Reproducible, language-specific failure modes exist and are invisible without external tests. The Node stdout truncation reproduced in three of three attempts and produced output that looked correct.
Scope of the recommendation. Agent effort is one line item in a language decision, and this study measured only that line. A language is also its runtime performance and resource behaviour, its compilation and deployment story, how well a team can work in it concurrently, the available hiring pool, and the maintenance horizon of the code it produces. The very ceremony that costs tokens here (explicit error paths, static types, enforced structure) exists to serve several of those other line items, and this study is silent on whether it succeeds at larger scales. Python winning the measured column does not make it the right choice for a latency-critical service, an embedded target, or a large long-lived team codebase; notably, the agent built production-shaped concurrent daemons in Go perfectly well, at a modest token premium a team might rationally pay.
Future work
The ceiling is the binding constraint. A meaningful test of H2 requires tasks on which a frontier agent still fails often enough to discriminate: multi-file changes against existing codebases, longer horizons, ambiguous specifications requiring design decisions, or genuine domain depth. The staged-complexity design of §10.6 doubles as the fair test of ceremony: if structure’s benefits exist, they should appear as the same codebase is extended across sessions. A threshold-shaped hypothesis (§7.4) also wants languages sampled across a much wider range of per-domain corpus volumes than four mainstream languages provide.
12. Reproducibility
The materials are not currently published, but the design is fully specified here and in the sections above; a competent practitioner can rebuild either part from this document. The load-bearing details:
Part one. Corpus: The Stack v2, train-smol-ids variant, pinned to a single
dataset revision (hub snapshot of 2026-08-03; collection cutoff September 2023).
Ten languages, 300 repositories per language, uniform random from repositories
meeting a minimum per-language byte floor, all randomness derived from one
integer seed. Classification: dependency manifests and import statements scored
against a fixed package→category map (1,334 entries across eight package
ecosystems; fourteen fine categories rolling into seven coarse buckets), with a
README-based LLM fallback only where the manifest signal is absent, and each
track’s accuracy measured against 200 blind-labelled repositories as reported
in §4. Repositories are fetched at the corpus-pinned commit, never at HEAD.
Part two. Twelve challenges as described in §5, each defined by a
language-neutral specification that states every tested behaviour (formats,
error text, exit codes, rounding, ordering), a hidden test suite exercising the
solution only through a run.sh entry point plus stdin/stdout, files, HTTP, or
TCP, and a reference solution used solely to validate the suite. Agent: Claude
Code in non-interactive mode, Sonnet, file and shell tools only (no web), fresh
workspace per run, wall-clock capped, operator customizations disabled. All
run-level telemetry (token usage split by input/output/cache, turns, durations,
per-test outcomes, rubric scores) captured per run; every aggregate in this
paper is computed from those records with seeded bootstrap resampling for the
intervals. Prompts differ across languages only in the language name, verified
by hash.
Generated datasets (the performance workloads and the churn corpus) come from seeded generators calibrated as described in §5.1, so equivalent datasets can be reconstructed from the stated distributions and difficulty targets.