AI and The Redmonk rankings
Every year I look forward to seeing the Redmonk language rankings. They are the only language rankings I pay any attention too, cough Tiobe. In recent years there has been very little movement in their top 20. It features all the usual candidates. Java, Javascript, Python, C# etc... the languages you may possibly get a job in. My interest lies in the outliers, seeing random languages fly up the chart, often to fizzle out.
This year what caught my interest were three observations the writer made on the impact of AI. Or rather the lack of impact.
an anomalously low second half’s worth of pull requests from GitHub
This comment came after the author mentioned the declining relevance of StackOverflow, which makes up one axis on the popularity grid. Github making up the other.
The author expected a decline in StackOverflow tags, but was puzzled when observing a decline in pull requests. This also surprised me. I see alot of "hype" online about the supposed productivity gains related to generative AI in software engineering. The lack of data to reflect this suggests to me that there is little substance to the hype. Which would reflect my personal experience. And before you suggest that Github pull requests are not a good measure of productivity. Let me stop you. The sheer amount of money spent on code generation should have increased pull requests and every other metric related to "lines" of code.
The author surmised that it could be bad data, or that code creation went up while code committed went down. The latter would suggest to me a decrease in productivity. Are programmers just generating slop to get a dopamine hit without actually producing anything of value? I suspect this is often the case. But it's pure speculation. The data says pull requests have declined. That is a fact.
given that coding assistants make developers’ familiarity with a language less relevant and the tools’ propensity to reflect the biases inherent in their training data, it would be logical to expect some meaningful change in language usage... however, these have not manifested
Again, this also surprised me. When I used generative AI, I no longer do, I used it with languages I was not familiar. For example generating Ada code, and trying out Zig. So my usage was exactly the usage the author expected to be reflected in the charts. The problem I had was that the code generated often didn't work, and I didn't have the knowledge of the langauge to fix it. For me, when people write about code generation with AI they often seem to assume or suggest the code generated is perfect. I have not found this to be the case. It was pot luck whether it worked or not.
Using AI to write a language you are not familiar with also robs you of learning that language. So even if you got AI to spit out Haskell code, why would you bother. You would just be left with code that might work, in a language you can't understand. So maybe it's not a surprise that developers are not jumping from one language to another just cause they have a bot that can vomit up syntax.
coding assistance tools, in theory, should be lowering some of the barriers to entry with Rust. If that’s happening, however, it’s not observable yet.
I don't care what anyone says, Rust is super hard. I've made multiple attempts to write Rust and have been put off by the compiler constantly saying NO. One might think that the strictness would lend itself to LLM's being able to generate working Rust code. I've not found that to be the case and it looks like the data supports that finding.
So what are the benefits of generative AI for software engineers? The real benefits. The ones that show up in data. Well going off my favorite twice annual report from Redmonk. The data suggests that generative AI helps programmers make less pull requests, avoid Rust, and doesn't help them learn new languages. What a tool. A great waste of time and money.
Hip. Hip. Hooray. For AI.