[DISCUSS] IBM using LLMs to convert COBOL to Java

And doing it manually is probably cheaper in the long run, especially considering that COBOL tends to power some very mission critical tasks, like financial systems.

The process should be:

1. set up a way to have part of your codebase in your new language
2. write tests for the code you're about to port
3. port the code
4. go to 2 until it's done

If you already have a robust test suite, step 2 becomes much easier.

We're doing this process on a simpler task of going from Flow (JavaScript with types) to TypeScript, but I did a larger transition from JavaScript to Go and Ruby to Python using the same strategy and I've seen lots of success stories with other changes (e.g. C to Rust).

If AI is involved, I would personally use it only for step 2 because writing tests is tedious and usually pretty easy to review. However, I would never use it for both step 2 and 3 because of the risk of introducing subtle bugs. LLMs don't understand the code, they merely spot patterns and that's absolutely not what you want.

Yeah, I read the article.

They’re MASSIVELY handwaving a lot of detail away. Moreover, they’re taking the “we’ll fix it in post” approach by suggesting “we can just run an armful of security analysis software on the code after the system spits something out”. While that’s a great sentiment, you (and everyone considering this approach) needs to consider that complex systems are pretty much NEVER perfect. There WILL be misses. Add this to the fact that a ton of organizations that still use COBOL are banks - which are generally considered fairly critical to the day-to-day operation of our society, and you can see why I am incredibly skeptical of this whole line of thinking.

I’m sure the IBM engineers who made the thing are extremely good at what they do, but at the same time, I have a lot less faith in the organizations that will actually employ the system. In fact, I wouldn’t be terribly shocked to find that banks would assign an inappropriately junior engineer to the task - perhaps even an intern - because “it’s as simple as invoking a processing pipeline”. This puts a truly hilarious amount of trust into what’s effectively a black box.

Additionally, for a good engineer, learning any given programming language isn’t actually that hard. And if these transition efforts are done in what I would consider to be the right way, you’d also have a team of engineers who know both the input and output languages such that they can go over (at the very, very least) critical and logically complex areas of the code to ensure accuracy. But since this is all about saving money, I’d bet that step simply won’t be done.

Even if it only converts half of the codebase, that’s still a huge improvement.

The problem is it'll convert 100% of the code base but (you hope) 50% of it will actually be correct. Which 50%? That's left as an exercise to the reader. There's no human, no plan, no logic necessarily to how it was converted also so it can be very difficult to understand code like that and you can't ask the person who wrote why stuff is a certain way.

Understanding large, complex codebases one didn't write is a difficult task even under pretty ideal conditions.

Counter counterpoint: The longer you let it sit the more obsolete the language becomes and the harder it becomes to fix it when something does break.

This is essentially preventative maintenance.