I spend most of my working life with AI coding agents these days, across a
bunch of languages and a bunch of very different projects, and one question has
been quietly nagging me the whole time: how much does the language actually
matter? I know roughly how these models get made, and knowing that is exactly
what planted the suspicion. It seemed likely to me that a model would be
stronger in some languages for some kinds of work, and that if that was real,
the reason would be simple: it comes down to how much code the model has
watched doing that kind of task in that particular language.
Two pieces of background explain the suspicion, if this isn’t your daily
world. First, how these things learn: coding models are trained on enormous
piles of public code, much of it scraped from places like GitHub, and they get
good at things roughly in proportion to how often they’ve seen them. Nobody
hand-writes rules about how a web server should look; the model absorbs a few
hundred thousand web servers and develops an opinion.
Second: programming languages have day jobs. In principle any language can do
anything, but in practice ecosystems snowball. One good library attracts
users, users write tutorials and more libraries, and a decade later the
language has a reputation and a gravitational pull. Python ended up doing the
world’s data science and machine learning, plus a long tail of scripts and
glue. TypeScript and JavaScript run the web, overwhelmingly. Go built itself a
home in cloud tooling and infrastructure. Ruby, these days, mostly means Rails
web apps. So the pile of code a model learns from isn’t just big. It’s
lopsided, and lopsided in a different direction for every language.
Put those two things together and you get a really tempting intuition. Here’s
the concrete version of it I’d have bet money on: TypeScript should be better
at building web apps than Python, because nearly all of TypeScript’s corpus
(the body of public code a model learns from) is web applications, while
Python’s is scattered across half of software. More relevant examples in
training, better output at the keyboard. Right?
I wanted to know if that intuition was actually true. So I designed a study to
test it, and had Claude Code help me make it happen: first measure what each
language’s public code really contains, then hand an agent identical work in
four languages and watch what happens.
What public code is actually made of
For the measuring step, I used The Stack v2, the deduplicated public-code
dataset used to train StarCoder2 and about as close as you can get to “what a
code model actually saw,” and classified 3,000 sampled repositories by what
they depend on and what they say about themselves. Django means web backend.
Pandas and Airflow mean data. Cobra means CLI tool.
What each language's public code is actually made ofshare of bytes by category · The Stack v2, 3,000 sampled repositories
webdata/MLtoolsinfraappslibraryunknown
(Rows don’t add to 100: the remainder is code that defied categorization.)
See the numbers
web
data/ML
tools
infra
apps
library
Python
14%
22%
14%
7%
8%
18%
TypeScript
77%
0%
2%
1%
9%
8%
Go
11%
2%
28%
16%
1%
32%
Ruby
32%
1%
9%
13%
2%
39%
The corpus had surprises in it before I ran a single test. TypeScript is 77%
web and rounds to zero percent machine learning. Data science is essentially
a Python monopoly; the amount happening in other languages is statistical
dust. And Go, famously the backend language, is actually a corpus of
libraries and command-line tools. The backend-services reputation is something
people say about Go, not something people mostly do with it.
The experiment
Then the test. Twelve programming challenges, two each across six categories:
web, data/ML, CLI tools, infrastructure, libraries, and a control group of pure
algorithm puzzles, LeetCode-medium-to-hard type stuff, with no ecosystem
attached at all. The control matters: if a language is just better at
everything, it shows up there, and you can subtract it out.
Each challenge is a plain-English spec with a hidden test suite, written
before any solution existed, that pokes the finished program strictly from
the outside: a shell command, an HTTP endpoint, a TCP socket. Identical tests
for every language, 401 in total. I handed each spec to a coding agent (Claude
Code, headless) four times over: Python, TypeScript, Go, Ruby. Three
repetitions apiece, isolated workspaces, no web access. 144 runs, about two and
a quarter hours, $49.52, five and a half million tokens. I pre-registered ten
predictions beforehand so I couldn’t quietly move the goalposts. Three
survived.
The first result flattened my plan: the agent just did all of it. 136 of
144 runs passed every hidden test; the worst run in the study scored 97%. You
can’t rank things that are all tied, and it wasn’t that the challenges were
easy. One was a churn-prediction model graded on ROC-AUC, calibrated so a basic
logistic regression fails the top tier. Every language cleared the top tier
every time. The agent did feature engineering in Go. In Ruby. Unprompted.
So before anything else, one finding stands on its own: at this size of task,
language choice did not determine whether the agent succeeded. And sit with
that for a second, because it’s quietly the best news in the study: these
models are just good now. They rarely got stuck, they shipped working
software in every language I tried, and the code held up against tests written
specifically to catch them out. More good software, built more easily, by more
people. That’s the floor this data establishes, before we argue about anything
else.
What did vary, reliably and repeatably, was the bill. I normalized every run
against the average for that specific challenge, so task difficulty cancels
out and 1.00 means “the average language on this task”:
What identical work cost, by languagetoken cost normalized per challenge · 95% bootstrap CI · n=36 runs eachSee the numbers
language
relative cost
turns per run
lines written
Python
0.88
14.1
215
Ruby
0.95
15.1
210
Go
1.07
16.1
388
TypeScript
1.09
23.2
285
Identical work cost about 24% more in TypeScript than in Python, and the
gap is statistically solid. TypeScript needed 64% more back-and-forth turns to
reach the same finish line. Go wrote nearly double Python’s lines for the same
behaviour, in every category.
To be clear about what that table does and doesn’t say: the expensive languages
did not produce worse code. Everything passed the same tests, and a
code-quality rubric scored all four within a whisker of each other. The
difference isn’t what came out. It’s how much work it took to get there.
Why? Not the reason I bet on
I had three candidate explanations, and the data killed the two I liked.
Three explanations, one pair of axesy is always relative cost · only the x-axis changes
PythonTypeScriptGoRuby
1 · corpus share All 20 language × task cells: no pattern (ρ −0.10). Only TypeScript tracks its own corpus (ρ −0.90).2 · popularity GitHub rank vs cost: the most popular language is the most expensive.3 · verbosity Lines per run vs cost: a clean upward slope (ρ +0.60). The boring answer survived.
My original theory, corpus match, failed almost completely. Correlating each
language’s corpus share against its cost across all twenty language-task cells
gives −0.10, which is nothing. Worse, the details point the wrong way: Python’s
cheapest categories were infrastructure and web, two of its smallest
corpus slices. Whatever makes Python cheap, it is not “Python has seen more of
this.”
Popularity failed too, and backwards in a way I enjoy: TypeScript just became
the #1 language on GitHub by monthly contributors, and it’s the most expensive
language in my study. Ruby, which every industry survey politely describes as
declining, came second cheapest.
What survived is the boring answer: verbosity. The correlation between how
many lines a language needs and what a run costs is +0.60 across languages,
+0.58 across all 144 individual runs. Look at the shape of the cost table
again: the two terse, say-what-you-mean languages first and second, the two
strict, structured, ceremony-heavy languages third and fourth, on every effort
metric I collected.
Part of that is mechanical: more code is more tokens. But why does Go take
388 lines where Ruby takes 210? That’s not the model’s choice. That’s the
language. Explicit error handling on every call, no comprehensions, no
shortcuts. TypeScript’s types, interfaces, and config are the same bet. That
ceremony exists, honourably, to make large codebases legible to large teams.
Real benefits, paid for in keystrokes. When an agent does the typing,
keystrokes are tokens, and tokens are the bill.
Which collides with the industry story. GitHub, in the same report that
crowned TypeScript, credits its rise partly to typed languages being better
for AI coding, because the compiler catches the agent’s mistakes. Maybe, at
scale. But at this task size I couldn’t detect that benefit anywhere:
correctness was maxed out in all four languages, types or no types, while the
cost of the structure showed up on every single run. The safety rail did
nothing measurable; the toll booth charged anyway.
One thread of my original theory did survive, though, and it’s the useful one.
TypeScript, the language with the most lopsided corpus, followed the corpus
script faithfully: cheapest on web, its 77% category, and most expensive on
data/ML, its 0% category and the single priciest cell in the entire study.
Correlate TypeScript’s own costs against TypeScript’s own corpus and you get
−0.90, about as clean as this kind of data gets. So my theory wasn’t wrong so
much as it only applies at the extremes. It’s a threshold, not a dial: once a
language has enough public code doing a kind of work, more doesn’t help.
What hurts is having approximately none. Python clears the threshold
everywhere, so nothing is ever foreign to it. TypeScript falls off a cliff in
the one place its ecosystem has simply never gone.
That’s also the honest version of why Python does well, and it’s not the
version I used to tell. It isn’t “huge diverse ecosystem makes the model
smarter.” It’s two much less glamorous things stacked: no blind spots, and
very few lines. Coverage without ceremony.
expression evaluator · performance test · expected vs emitted output, three TypeScript runs
So, is Python the best language for your AI projects?
The answer this study earned is: it depends, but it’s likely a good choice.
Here’s the whole decision, compressed.
The models are good, and that swamps everything else. They shipped working,
verified software in every language, on every kind of task, almost every time.
Which means the language that matters most is yours: you’re the one reading
what the agent writes, steering it, catching what the tests don’t. Comfort
beats every number in this post.
If you don’t have a comfort zone, because coding was never your thing until
an agent made it possible (what people now call vibe coding, and I mean that in
its honest sense: you describe, it builds, you steer), then yes: Python. It was cheapest overall, never
worst at anything, and it’s the one language with no blind spots, from web
servers to machine learning (which, remember, barely exists outside it). You
will never be the first person solving your problem in Python, and when an
agent is doing the writing, that’s exactly the road you want: paved, lit, and
going everywhere.
And when something concrete pushes you elsewhere, let it. A web shop living in
TypeScript should stay: web is its home turf and it’s perfectly competitive
there. Just don’t drag it off-turf (“we’re a TS shop, just do the ML pipeline
in TS” bought the worst cell in my entire study). If the running software
matters more than the writing of it, Go is still a fine choice: the agent
built production-shaped concurrency daemons in it without breaking a sweat,
the ~20% token premium is paid once, and Go’s runtime advantages recur every
day the thing runs. Because a language is always more than how efficiently an
agent emits it: runtime, memory, deploys, teams, hiring. My study measured one
line on the invoice. It’s a real line. It’s not the whole invoice.
Two things I’d do differently, and one I want to do next. I calibrated
challenge difficulty against reference solutions I wrote myself, but the agent
is a much better programmer than my “naive baseline,” and one throwaway run
against the real thing would have shown me the ceiling in an hour.
Well-specified few-hundred-line programs are, as of now, simply solved; any
benchmark that hasn’t noticed is measuring nothing. And the experiment I want
to run next is multi-stage projects: build, extend, extend again, the same
codebase growing across sessions. All of Go and TypeScript’s ceremony is a bet
on exactly that scenario, and my single-shot tasks never let the bet pay out.
Maybe at stage four Python’s flexibility starts generating the mess that Go’s
rigidity prevents, and the cost ranking flips. I genuinely don’t know, which
is exactly why it’s the next thing worth measuring.
One model (Claude Sonnet), one machine, one point in time. These numbers will
drift as models change, though I’d bet on the ceremony effect outliving any
particular model version.
If you want the full methodology (the corpus pipeline, the challenge specs,
the statistics, the pre-registered predictions and how few survived) the
complete study write-up is here: Language × Task Affinity.
It’s written for people who want to check my work rather than take my word for
it.
Two-part study August 2026 By Colin Moffat
Language × Task Affinity
Does the composition of a language's public corpus predict how well an AI coding agent performs in it?
Summary
Coding models learn from public code, and public code is distributed very
unevenly across languages and problem domains. This study asked whether that
distribution predicts agent performance: is an agent measurably better at web
work in TypeScript, or data work in Python, because so much more of that
specific combination exists to learn from?
Part one measured the distribution. Using The Stack v2, the deduplicated
permissively-licensed corpus assembled to train StarCoder2, we built a
language × task-category matrix across ten languages and 3,000 sampled
repositories, classified by dependency manifests and README content, with a
validated classifier behind every number.
Part two tested the prediction. Twelve language-neutral programming
challenges, four languages, three repetitions, 144 controlled runs of a headless
coding agent, scored by hidden black-box tests written before any solution
existed.
The central hypothesis was not supported. The agent solved essentially every
challenge in every language (136 of 144 runs perfect), so task-specific corpus
volume could not be shown to affect success at all. Cost did vary reliably
(identical work cost roughly 24% more in TypeScript than in Python), but that
variation correlates with language verbosity (ρ = +0.60) far better than with
task-specific corpus share (ρ = −0.10), and runs opposite to raw language
popularity. The one clean corpus effect appears in TypeScript, whose costs order
almost perfectly by its own highly lopsided corpus (ρ = −0.90) and whose worst
result is in machine learning, a domain comprising 0.0% of its public code.
We read this as evidence for a threshold rather than a dose-response
relationship: above some volume of examples, more stops helping; near zero, the
penalty is real and measurable.
1. Motivation and hypotheses
The practical question: when a coding agent will write most of the code, should
language choice change?
The intuition behind “yes” is that models learn from public code, so competence
on a task/language pair should track how much of that pair exists. Split apart,
that intuition contains four distinct claims, which make different predictions:
H1, volume. Agents are better in languages with more training code, full
stop. Predicts one language wins everything.
H2, task match. What matters is the volume of code in language L
solving problems like T. Predicts the winner changes with the task.
H3, uniformity. A language with one idiomatic way to do things produces a
low-entropy corpus and therefore more consistent output.
H4, fewer boundaries. A single language across a whole system reduces
contract mismatches.
Part one produces the independent variable that H1–H3 need. Without it,
“the language with the most code for this task” has no defined meaning.
Both parts were pre-registered: nine predictions written down before part one
ran, ten before part two. Scoring against those predictions is in §7 and §11.
2. Part one: method
Six seeded, resumable pipeline phases:
Frame. Stream all 64 shards (~33 GB) of The Stack v2 train-smol-ids
metadata, retaining repository name, pinned commit, per-file language and
size, star events and creation date. Each shard is deleted after processing,
so peak disk stays around 520 MB. Full-corpus pass, no sampling at this stage.
Sample. 300 repositories per language, uniform random from the eligible
set, seeded. Ten languages → 3,000 (repository, language) pairs.
Fetch. Dependency manifests and a bounded source sample from GitHub at
each repository’s pinned commit, so the code classified is the code that
was in the corpus rather than the current HEAD. Rate-limit aware, fully
cached. 2,895 of 3,000 fetched successfully; the remainder were deleted or
renamed since the 2023 cutoff.
Classify. Deterministic scoring of manifests and imports against a
committed 1,334-entry package→category map. Fourteen fine categories rolling
up into seven coarse buckets. Originally specified with no LLM anywhere in
this path.
Validate. 200 repositories drawn stratified; README, file tree and
entry-point file heads gathered as labelling materials; labels assigned
blind from those materials only, never from classifier output; classifier
scored against them.
Analyze / report. Matrices in repositories, files and bytes; lift; focus
(normalized entropy); framework concentration; seeded bootstrap intervals.
Units are reported in repositories, files and bytes. Bytes is the headline
because models train on tokens, but repository counts are closer to how people
think about “what gets built,” and the two are reported side by side.
3. Part one: results
Within-language share of bytes, coarse categories, hybrid classifier (§4):
language
web
data_ml
tools
infra_sys
apps_other
library
unknown
Python
13.6%
22.2%
13.6%
7.1%
7.7%
18.3%
17.5%
JavaScript
65.6%
0.1%
3.4%
4.7%
2.3%
6.5%
17.4%
TypeScript
77.0%
0.0%
1.8%
0.9%
9.2%
8.1%
2.9%
Go
10.9%
1.6%
27.5%
15.9%
1.0%
32.4%
10.7%
Java
6.9%
0.0%
0.3%
4.2%
6.4%
1.8%
80.4%
Rust
5.0%
0.8%
14.2%
43.2%
3.8%
20.6%
12.4%
C#
12.0%
5.5%
4.0%
1.3%
20.6%
35.1%
21.5%
Ruby
31.9%
0.8%
9.3%
12.6%
1.6%
39.4%
4.5%
PHP
31.1%
11.3%
0.8%
0.1%
0.8%
18.2%
37.7%
Kotlin
3.2%
0.1%
0.7%
0.4%
83.4%
8.2%
4.0%
3.1 Findings
Two derived measures appear throughout: lift is a cell’s observed share
divided by the share expected if language and task were independent (1.0 = no
association; 0.0 = the combination essentially does not occur), and focus is
one minus the normalized entropy of a language’s category distribution (1.0 = a
single-purpose language, 0.0 = perfectly even spread).
Python is the least focused language measured. Focus score 0.04, last of
ten, with the next-lowest at 0.17. Its bytes distribute almost evenly across
data/ML, library, web and tooling. This is the dilution mechanism from H2,
measured directly: the token “Python” carries far less information about what
follows than “TypeScript” does.
Data and ML are a near-monopoly. Lift for Java × data_ml is 0.00, Kotlin
0.02, JavaScript 0.03. Public data-science code is overwhelmingly Python, to a
degree that leaves the other languages with essentially no representation.
Go’s corpus contradicts Go’s reputation. Measured: library 32.4%, tools
27.5%, web 10.9%. Two of the nine pre-registered predictions concerned Go and
both failed: it is neither the most focused language (0.17, near the bottom)
nor the highest-lift infrastructure language. Go is a language in which people
publish libraries and command-line tools, and about which people talk as though
it were a backend-services language.
Framework monoculture is also inverted from folklore. The tightest web
monocultures are Java/Spring (83.3% of web repositories) and C#/ASP.NET (78.7%),
not Rails (54.8%) or Laravel (66.7%). JavaScript web is the most fragmented
(React at 56.7%, Herfindahl 0.39). For H3, the interesting uniformity contrast
is Java/C# versus JavaScript, not Ruby versus JavaScript.
Pre-registered scoring: 4 supported, 2 not supported, 1 mixed, 1 insufficient
data, 1 reserved for author judgment.
4. Part one: the classifier had a measurable ceiling
The validation step changed the study’s numbers materially, so it is reported
in full.
The classifier was deliberately LLM-free: manifests, import regexes and a
committed YAML map, chosen so that any claim of the form “this repo is web
backend” could be audited by a reader. Validating it against 200 blind-labelled
repositories measured it at macro-F1 0.27, accuracy 0.44, with unknown
precision at 0.19.
The failure mode was systematic rather than random. A repository with no
dependency manifest produces no signal, so it lands in unknown even when its
README states its purpose in the first sentence. Django projects without a
requirements.txt, pygame games, machine-learning paper code: all unknown.
We then measured two additional signals against the same blind labels:
track
macro-F1
accuracy
dependency manifests only
0.27
0.44
README/description via a cheap LLM (Haiku)
0.60
0.65
hybrid (manifest primary, LLM fallback)
0.60
0.68
GitHub Topics were evaluated as a third signal and rejected as a classifier:
only 179 of 3,000 repositories carry any topics, and only 90 map to a category.
Retained as an agreement check only (63% agreement with the LLM track).
The hybrid was adopted, which amends the study’s original “no LLM in
classification” constraint. The justification is that the constraint existed to
make claims auditable, and the hybrid preserves that: every manifest-derived
classification still carries its matched packages, so framework-concentration
analysis is unaffected, and each track’s accuracy is separately measured and
reported rather than asserted.
Effect on the headline numbers. Python’s unknown share collapsed from
60.5% to 17.5%, exposing the 22% data/ML share that manifest-only classification
could not see. Go’s library share moved from 0% to 32.4% (Go modules carry no
structural library hint; a gap flagged on day one of the build and quantified
here). Per-language accuracy now ranges 0.45–0.89, with Go and Kotlin clearing
the study’s 0.85 target and the rest honestly reported as below it.
4.1 A cautionary note on bytes-weighting
apache/buildr alone constituted 24.5% of Ruby’s sampled bytes. A single
mis-weighted dev-dependency (jekyll, present only to build the project’s
documentation site) classified it as web frontend and moved Ruby’s entire
web share by roughly 25 percentage points until corrected. Bytes-weighted cells
at n=300 are highly sensitive to individual large repositories; repository-count
matrices are not, and are the safer basis for headline claims.
5. Part two: design
Languages (4): Python, TypeScript, Go, Ruby. Python as the H1/H2 crux
(largest volume, lowest focus); TypeScript as the specialist with volume;
Go as the folklore-versus-corpus test; Ruby chosen over C# because C#’s 21.5%
unknown rate made its corpus profile the least trustworthy of the candidates.
Kotlin was excluded because none of the testable buckets is mobile, where 83% of
its corpus lives.
Buckets (6): web, data_ml, tools, infra_systems, library, plus general:
pure algorithmic challenges with no ecosystem dependence, serving as a control.
Every bucket result is read as a delta against the same language’s general
baseline, so that language-level ability is separated from task-specific effects.
Scale: 2 challenges per bucket × 4 languages × 3 repetitions = 144 runs.
Agent: Claude Code headless (claude -p), Sonnet, one campaign, one model,
one machine.
5.1 Fairness controls
The design problem is running the same task four ways with nothing leaking
between them.
Identical prompts. One spec per challenge with the language name
substituted; byte-identical otherwise. The SHA-256 of every rendered prompt is
recorded per run.
Hidden tests, written first. 401 tests across 12 challenges, written in
Python before any solution existed, exercising each solution only through
language-neutral interfaces: a run.sh the agent must produce, plus
stdin/stdout, files, HTTP, or a TCP socket. The identical suite scores all
four languages.
Double validation of every challenge. Each suite must pass 100% against a
stdlib-only reference solution (proving the tests are satisfiable and agree
with the spec) and must fail against a do-nothing run.sh (proving the tests
discriminate).
Difficulty calibration, verified rather than assumed. An independently
written naive solution to the booking-engine challenge passes all 25
functional tests and fails only the performance gate; a stream-slurping
implementation of the JSON query tool passes everything except the streaming
test; the churn model’s tiers were calibrated against measured AUCs of 0.774
(raw features), 0.872 (engineered) and 0.900 (true-logit ceiling).
Isolation. Fresh workspace per run, located outside any directory tree
containing the tests or the harness; no web tools in the agent’s toolset;
operator customizations (CLAUDE.md, hooks, MCP servers) disabled. A
transcript audit flags any reference to the tests directory: zero flags
across 144 runs.
Seeded interleaved run order, so no language clusters in wall-clock time.
Known confound: package installation during a run is permitted, since
solutions must run offline afterward. Ecosystem richness is arguably part of
what “language” means here, but it means these numbers are not pure language
competence.
Metrics: hidden-test pass rate (primary as designed); billable tokens
(input + output + cache writes), with cache reads recorded separately since they
run ~27× billable in agentic loops; turns; wall-clock decomposed into API wait
and local toolchain time; non-blank non-comment lines of code; and a
fixed-rubric LLM judge scoring correctness-risk, idiomaticity, error handling
and structure.
6. Part two: results
Campaign: 144/144 completed. No timeouts, no retries, no audit flags.
$49.52, 5,559,505 billable tokens, 6.8 hours of cumulative agent time, 2h14m
elapsed at concurrency 3.
6.1 Accuracy saturated
136 of 144 runs passed every hidden test. The worst run in the campaign scored
0.97. Pass rate is saturated, with effectively no variance, and the primary
metric of the design is therefore uninformative: a ceiling effect from the
instrument’s point of view; from any other point of view, simply a very capable
model.
The saturation is not an artifact of easy challenges. The churn-model challenge
was explicitly calibrated so that a raw-feature logistic regression scores
AUC 0.774 and fails its top tier; all four languages cleared the top tier in
all three repetitions, performing feature engineering unprompted.
Six of the eight imperfect runs form two perfectly reproducible failure
patterns (the other two were isolated single-test misses), which makes them
more interesting than their number suggests:
TypeScript fails the expression-evaluator’s performance test in all three
repetitions, identically: output correct for exactly 65,525 characters and
then truncated. This is a 64 KB buffer boundary, Node’s asynchronous stdout
not flushing before process exit. A deterministic, language-specific defect
that the agent did not detect because the program appears to work.
Python fails the task API’s cross-tenant authorization test in all three
repetitions, returning 403 or 422 where the specification requires 404 so
that resource ids do not leak existence. A specification-compliance failure
rather than a capability one.
6.2 Cost differs reliably between languages
Each run normalized by its own challenge’s cross-language mean, so task
difficulty cancels; 1.00 is the average language on that task. Bootstrap
intervals, n=36 per language:
language
relative cost
95% CI
mean turns
mean LOC
Python
0.88
0.80–0.98
14.1
215
Ruby
0.95
0.87–1.02
15.1
210
Go
1.07
1.00–1.16
16.1
388
TypeScript
1.09
1.01–1.18
23.2
285
Python and TypeScript both have intervals clear of 1.00; the gap between them is
24%. Per-challenge spreads range from 1.09× to 2.41×; in absolute terms,
per-run means ranged from ~14,800 billable tokens (Python on the backup CLI) to
~72,200 (Go on the JSON query tool). TypeScript’s turn count, 64% above
Python’s, is the sharpest single discriminator in the data.
Relative cost per challenge × languagemean of 3 runs per cell · 1.00 = average language on that challenge
cheaper more expensive
Anatomy of the same run, four times overevery agent step · URL-shortener web challenge, first repetition · block width = output tokens
writerunrespond
By bucket (* = interval excludes the all-language average):
bucket
Python
TypeScript
Go
Ruby
general
1.01
1.09
1.11
0.79*
web
0.84*
0.98
1.21*
0.98
data_ml
0.85
1.32*
0.85
0.98
tools
0.89
1.11
1.08*
0.92
infra_systems
0.82*
1.03
1.19*
0.96
library
0.90
1.02
1.01
1.07
7. Testing three explanations
7.1 Corpus match (H2): not supported
Spearman correlation between part one’s within-language corpus share and part
two’s relative cost. Negative would support H2 (more corpus → cheaper):
scope
ρ
pooled, all 20 (language, bucket) cells
−0.10
within TypeScript
−0.90
within Ruby
+0.40
within Go
0.00
within Python
+0.50 (inverted)
The extremes contradict H2 directly. Python’s cheapest bucket is
infra_systems, where it holds 7.1% corpus share; Go’s most expensive is
infra_systems, where it holds 15.9%: more than double, and it performs worse.
The exception is instructive. TypeScript × data_ml is the single most
expensive cell in the study (1.32), and data_ml is 0.0% of TypeScript’s corpus.
TypeScript is also the only language whose internal cost ordering tracks its own
corpus almost perfectly.
7.2 Raw popularity: not supported, and inverted
Per GitHub Octoverse 2025, TypeScript is now the most-used language on GitHub by
monthly contributors (2.64M, +66% year over year), having overtaken Python. It
is the most expensive language in this study. Ruby, which JetBrains’ 2025
ecosystem survey places in long-term decline, is the second cheapest.
Rank-correlating popularity against cost gives ρ = −0.40 in the direction of
“more popular, more expensive,” though n=4 cannot support a real correlation in
either direction. The qualitative point stands regardless: the two extremes of
popularity are inverted relative to the two extremes of cost.
7.3 Verbosity and ceremony: supported
Mean lines of code written per run: Ruby 210, Python 215, TypeScript 285,
Go 388. Correlation with relative cost: ρ = +0.60 across languages, and
ρ = +0.58 across all 144 individual runs.
Lines written vs billable tokens, all 144 runseach dot is one run · trend across all runs shown in gray
PythonTypeScriptGoRuby
This is partly mechanical (more code means more output tokens), and we state
that rather than hide it. What makes it non-trivial is that the cause of the
line-count difference is not the model’s choice but the language’s ceremony:
Go’s explicit error returns, its absence of comprehensions, TypeScript’s type
declarations and configuration surface. These are stable properties of the
languages, and they account for the data better than either alternative
explanation.
The result has a clean structural reading: the two terse, dynamically typed
languages occupy the two cheap positions on every effort metric collected
(tokens, turns, and lines), while the two statically typed, structurally
disciplined languages occupy the two expensive ones. With four languages this
2-versus-2 split cannot separate terseness itself from static typing (the
properties co-vary perfectly in our sample), so we describe the combined bundle
as ceremony and make no claim about which component drives the cost.
The finding sits in direct tension with a prominent industry narrative. GitHub’s
Octoverse 2025 report attributes TypeScript’s rise to the top of its language
rankings partly to typed languages making agent-assisted coding more reliable.
At this task size we could not detect that reliability benefit anywhere:
correctness saturated in all four languages, typed or not (§6.1), while the
token cost of the added structure registered on every run. This does not refute
the narrative at larger scales, where compiler feedback over a large codebase
plausibly earns its keep; it does mean the benefit was unmeasurable exactly
where the cost was unmissable.
7.4 Interpretation: threshold, not dose-response
The pattern across §7.1–7.3 is consistent with a threshold relationship
rather than a linear one. Above some sufficient volume of examples, additional
corpus stops improving performance; near zero, the penalty is real. Python
clears the threshold in every bucket, so it performs well everywhere and its
internal distribution ceases to matter, which explains its inverted ρ.
TypeScript falls below the threshold in exactly one domain and is measurably
penalized there.
This is interpretation, not demonstration: four languages cannot establish a
threshold curve, and identifying where such a threshold sits would require
languages sampled across a much wider range of per-domain volumes. It is offered
as the reading that fits the observations, and as the hypothesis a follow-up
study should be designed to test.
8. What did not discriminate
Reported because null results constrain interpretation.
The LLM judge could not distinguish the languages. Mean rubric scores:
Python 7.57, TypeScript 7.51, Ruby 7.47, Go 7.33: a 0.24 spread on a 1–10
scale, with per-axis spreads no wider (correctness-risk 7.03–7.72, idiomaticity
7.47–7.64). A pre-registered prediction of a Python idiomaticity advantage from
post-training skew is unsupported. Either the solutions genuinely are of similar
quality, or a same-family judge on a fixed rubric is too blunt an instrument.
The judge column supports no conclusion in this study.
Wall-clock time did not discriminate, despite tokens doing so: Ruby 161 s,
Python 170 s, Go 174 s, TypeScript 180 s. That is a 12% spread against a 24%
token spread. Wall-clock is dominated by API latency unrelated to the language,
which is why tokens and turns are the effort metrics we lead with.
Toolchain time did not discriminate, and an earlier impression was wrong.
A single pilot run suggested Go paid a heavy compilation penalty (127 s of a
206 s run). At n=36 this does not hold: Go has the lowest mean local time
(10.8 s, 6.2% of wall) and Python the highest (26.0 s, 15.3%), though Python’s
median is 2.2 s, so its mean is driven by a small number of runs re-executing
expensive performance cases. Toolchain time is outlier-driven and separates
nothing.
9. Threats to validity
Ceiling effect. The binding limitation. Pass rate cannot test H2 when
nearly every run is perfect. All accuracy-based conclusions are therefore
“not demonstrated” rather than “disproven.”
One model, one point in time. Sonnet, August 2026. These results describe
this agent; capability changes rapidly.
Task size. Well-specified programs of a few hundred lines. Nothing here
extends to multi-file work on existing codebases, which is where most real
agent use lives.
Two challenges per bucket. Where a bucket result is driven by one
challenge (most of TypeScript’s data_ml penalty comes from a single
challenge), task idiosyncrasy and bucket effect are not separable. Run-to-run
variance was small; task-to-task variance was large.
Package installation is permitted, so ecosystem richness is inside the
measurement rather than controlled out.
The corpus is a proxy for training data, not training data. No lab
publishes its mix. Models also learn from documentation, forums, licensed and
synthetic code, and are post-trained on task distributions that skew heavily
toward Python.
Permissive licenses only, September 2023 cutoff. GPL-heavy domains are
under-represented; post-cutoff trends are absent.
Blind labels are LLM-proposed and not yet human-reviewed, so part one’s
reported F1 figures inherit that error.
Java’s corpus row is the least trustworthy at 80.4% unknown even after
the hybrid classifier: enterprise Java in this corpus genuinely lacks READMEs.
The judge is same-family with the agent and cannot be blind to language,
since the code reveals it.
10. What we would do differently
Calibrate difficulty against the agent, not against a reference solution.
Every challenge was verified hard enough that a naive implementation fails,
but “naive” was our definition, and the agent is not naive. One throwaway run
against the real agent before authoring the remaining eleven challenges would
have exposed the ceiling in an hour rather than after a full campaign.
Match the control for difficulty, not just for ecosystem-independence.
The general bucket was intended as a neutral baseline but turned out to
contain the most expensive tasks in the study, so every bucket appears
“cheaper than control” for every language.
Record cheap metrics from the start. Pass rate was the designed headline
and had no variance; token cost, added mid-study, carried the entire
analysis. Metrics cannot be added retroactively to runs already paid for.
Ask the naive comparison first. The initial analysis compared each
language against its own control and nearly produced a “no difference”
conclusion. The direct language-versus-language comparison on identical tasks
(the question anyone actually asks) was implemented later and showed a clean
24% gap.
Prefer more challenges over more repetitions. Run-to-run variance was
small relative to task-to-task variance; the marginal third repetition bought
less than a third challenge per bucket would have.
Add multi-stage projects. Every task here was single-shot, which is the
scenario ceremony is not designed for. Structured languages bet that their
discipline pays off as a codebase grows and must survive repeated extension,
a bet our tasks never allowed to pay out, so the structured pair bore the
cost with no opportunity to collect the benefit. A staged design (build,
then extend, then extend again across sessions) is the fair test of that
bet, and the plausible place for the cost ranking to compress or invert.
11. Conclusions
Scoring the ten pre-registered part-two predictions: three clearly supported,
two partly, five not supported. The design’s central prediction, that corpus
match would move the numbers, did not hold.
#
prediction (abridged)
verdict
1
Python’s data/ML advantage the most positive of any language; others negative
not supported: accuracy saturated; on cost, Python and Go tie cheapest there
2
TypeScript peaks on web, with the largest web advantage of the four
partly: web is TypeScript’s own best bucket, but Python is cheaper there
3
Go performs better on tools than on web
supported: 1.08 vs 1.21 relative cost
4
Go shows a positive library effect
not supported: flat (1.01)
5
Ruby’s web result positive and better than Go’s
partly: Ruby 0.98 vs Go 1.21, but not positive against its own baseline
6
Control-bucket spread across languages within 15 points
supported: pass rates all ≥ 0.99
7
At least one bucket’s winner differs from the control’s winner
supported: Ruby wins the control; Python wins web and infra
8
Judge idiomaticity favors Python
not supported: judge spreads are indistinguishable (§8)
9
Token cost lower in each language’s corpus-strong bucket
not supported: pooled ρ = −0.10
10
Go terse per unit of work on tools
not supported: Go is the most verbose language measured
What the study does support:
For well-specified tasks of this size, language choice did not determine
whether the agent succeeded. It succeeded in all four languages, including
on a machine-learning task calibrated to be hard. That near-universal
success is itself the study’s clearest capability finding: the models are
simply good. A frontier agent now reliably produces working, externally
verified software across ecosystems in a single pass at this task size. The
conventional reasons to choose a language (team familiarity, deployment
target, maintenance horizon) are not displaced by agent-assisted
development; for an individual practitioner, the dominant factor this leaves
is familiarity, since the human’s ability to read and steer the output is
the constraint the agent does not remove.
Language choice does affect cost, by roughly 24% between the extremes,
and the split is structural: the terse, expressive languages beat the
ceremonial, structured ones on every effort metric collected. Cost tracks
verbosity (ρ = +0.60) far better than corpus composition (ρ = −0.10) or
popularity (inverted). Python’s advantage, on this reading, is coverage
without ceremony: above the sufficiency threshold in every tested domain,
nothing foreign to it, while being among the tersest languages measured.
Neither property is about the size or diversity of its ecosystem as such;
a hypothetical language with the same coverage and terseness but a tenth of
the volume would, on this account, perform the same. Meanwhile the widely
claimed agent-reliability benefit of static typing was undetectable at this
task size, while its cost was visible on every run (§7.3). Cost should not
be read as quality: the expensive languages’ output passed the same tests
and scored indistinguishably on the quality rubric (§8). Higher cost is
purely more effort to reach an equivalent result.
The corpus effect is real but confined to the tail. It is invisible in
the middle of the distribution and clear at the edge: the single most
expensive cell in the study is the language-domain pair with 0.0% corpus
share. Practically: taking a language somewhere its ecosystem never goes is
the case where this matters.
Reproducible, language-specific failure modes exist and are invisible
without external tests. The Node stdout truncation reproduced in three of
three attempts and produced output that looked correct.
Scope of the recommendation. Agent effort is one line item in a language
decision, and this study measured only that line. A language is also its
runtime performance and resource behaviour, its compilation and deployment
story, how well a team can work in it concurrently, the available hiring pool,
and the maintenance horizon of the code it produces. The very ceremony that
costs tokens here (explicit error paths, static types, enforced structure)
exists to serve several of those other line items, and this study is silent on
whether it succeeds at larger scales. Python winning the measured column does
not make it the right choice for a latency-critical service, an embedded
target, or a large long-lived team codebase; notably, the agent built
production-shaped concurrent daemons in Go perfectly well, at a modest token
premium a team might rationally pay.
Future work
The ceiling is the binding constraint. A meaningful test of H2 requires tasks on
which a frontier agent still fails often enough to discriminate: multi-file
changes against existing codebases, longer horizons, ambiguous specifications
requiring design decisions, or genuine domain depth. The staged-complexity
design of §10.6 doubles as the fair test of ceremony: if structure’s benefits
exist, they should appear as the same codebase is extended across sessions. A
threshold-shaped hypothesis (§7.4) also wants languages sampled across a much
wider range of per-domain corpus volumes than four mainstream languages provide.
12. Reproducibility
The materials are not currently published, but the design is fully specified
here and in the sections above; a competent practitioner can rebuild either
part from this document. The load-bearing details:
Part one. Corpus: The Stack v2, train-smol-ids variant, pinned to a single
dataset revision (hub snapshot of 2026-08-03; collection cutoff September 2023).
Ten languages, 300 repositories per language, uniform random from repositories
meeting a minimum per-language byte floor, all randomness derived from one
integer seed. Classification: dependency manifests and import statements scored
against a fixed package→category map (1,334 entries across eight package
ecosystems; fourteen fine categories rolling into seven coarse buckets), with a
README-based LLM fallback only where the manifest signal is absent, and each
track’s accuracy measured against 200 blind-labelled repositories as reported
in §4. Repositories are fetched at the corpus-pinned commit, never at HEAD.
Part two. Twelve challenges as described in §5, each defined by a
language-neutral specification that states every tested behaviour (formats,
error text, exit codes, rounding, ordering), a hidden test suite exercising the
solution only through a run.sh entry point plus stdin/stdout, files, HTTP, or
TCP, and a reference solution used solely to validate the suite. Agent: Claude
Code in non-interactive mode, Sonnet, file and shell tools only (no web), fresh
workspace per run, wall-clock capped, operator customizations disabled. All
run-level telemetry (token usage split by input/output/cache, turns, durations,
per-test outcomes, rubric scores) captured per run; every aggregate in this
paper is computed from those records with seeded bootstrap resampling for the
intervals. Prompts differ across languages only in the language name, verified
by hash.
Generated datasets (the performance workloads and the churn corpus) come from
seeded generators calibrated as described in §5.1, so equivalent datasets can be
reconstructed from the stated distributions and difficulty targets.