AI coders think they’re 20% faster — but they’re actually 19% slower
AI coders think they’re 20% faster — but they’re actually 19% slower
pivot-to-ai.com
AI coders think they’re 20% faster — but they’re actually 19% slower
You're viewing a single thread.
108 comments
-
The N=16 keeps getting buried. Deliberate?
-
You're acting like this is a gotcha when it's actually probably the most rigorous study of AI tool productivity change to date.
-
Paragraph 2:
METR funded 16 experienced open-source developers with “moderate AI experience” to do what they do.
-
... and just a few paragraphs further down:
The number of people tested in the study was n=16. That’s a small number. But it’s a lot better than the usual AI coding promotion, where n=1 ’cos it’s just one guy saying “I’m so much faster now, trust me bro. No, I didn’t measure it.”
I wouldn't call that "burying information".
-
<vapid statement>
. Debate me bro? (jk)
-
-
this user has been removed for commenting without reading the article
being from programming dot dev is just the turd on top
-
programming.dev: statistical sampling excellency (worst edition)
-
programmers learned what N means in statistics and immediately realized that “this N is too small” is a cool shortcut to sounding smart without reading the study, its goals, or its conclusions. and you can use it every time N is smaller than the human population on earth!
-
This N is too small: N
-
Skill issue - this N is even smaller:
-
The colon-space-subscript bothers me Immensely
-
-
-
-
-
108 comments