AI scope creep & productivity

I was recently asked to look at a change one of our developers made to RavenDB. He was chasing a particularly slow query with spatial sorting (find me all the nearest Thai restaurants style of query). He found the root cause, and had a fix ready. The fix gave us a 25% improvement in query processing time and the change was about 15 lines of code.

A few years ago, I would have said: great job, merge it, ship it, let customers enjoy the faster queries. But I got a hunch that we can do something different. We asked the AI model to try it out. It took a non-trivial amount of work to even test that, and in the past, it would be a few days just to try it out.

The model was able to complete the spike while we were talking about why this should be a good approach and the architectural level aspects that were driving it.

The result of that spike was an improvement of 12x. Not 12%. Twelve times faster. A 25% improvement is a wow, a 1,200% improvement is damn. In essence, we took a shortcut a couple of decades ago, and had to pay a lot more than we could have.

Note that a 15-line fix is local, easy to understand and verify. A structural change like this is… not. We also have to deal with backward compatibility concerns (we had to change the index format), verifying that we get the same output in both cases, etc.

Along the way, we found three separate bugs that had been in the product for 10 to 15 years. They're all in very weird edge cases. What happens when you sort on a spatial field that doesn't exist?

If a document has no spatial value, we would treat the missing value as zero distance, which means it is the closest result. I can argue that this is the correct behavior, but not convincingly.

The PR went from 15 lines of code to about a thousand. Review time for that went from under 10 minutes to over an hour.

Comparing this to the same task four years ago, a few  hours to isolate and understand the problem. A couple of hours to figure out how we can optimize this, writing the code, etc. The overhead of actually reviewing the code and merging it would be pretty minimal.

Today, it took an hour to reproduce the problem, another with the model to find & propose the original 15 lines fix. Then we dove deeper into this and did the rest in another couple of hours. The 1K LoC PR would take two hours to review.

Interestingly enough, I don’t think if we were working manually we would have hit on the 15 lines fix. That took advantage of a library function I didn’t even know existed. We would have headed directly to the proper fix we ended up with.

That leads to a few observations about the process. The quick fix with the AI would have delivered an inferior end result. It is funny when I say that a 25% improvement is inferior, but it would have been in this case.

The other aspect is that each decision we made along the way made sense, but we ended up with something that was big. To the point that you need to meditate over the code to understand it.

This is a very small and focused change, mind. And it grew from 15 lines to 1K. We are seeing the size of PRs grow from a few hundreds to many thousands of lines of code.

All of that pushes the complexity into the review process. And for long-lived software, review is not something you can skimp on. RavenDB has been around since 2009. Code we merge today is code we will be supporting and reasoning about for the next decade.

There is a wide spectrum of opinions on this right now. Some people say they never read the code, some have an agent review it. Others insist on going through every line, and that is where I stand.

The code that goes into RavenDB gets a manual review by a person who understands everything about the system. Not just whether the code is correct, but how it interacts with all the other stuff we have, the history behind it, and the things that are not in the diff.

Productivity? Where is it at?

As an industry, we expected AI to make us dramatically faster. If AI writes 90% of the code, sometimes 100%, shouldn’t we be shipping more often? And better software as well, no?

Every developer I've talked to says the same thing: I cannot not use AI. The benefits are too high to give up. I have seen people who left the IDE entirely and some who just use AI to explain and reason about the code, but everyone is using AI in some capacity. But we aren't seeing the change in productivity that everyone predicted.

I think a big part of the answer is in the story above. We aren't doing the same work faster. We're doing more work, arguably better work. We find opportunities we would never have considered, because the cost of checking a hunch dropped significantly. The cost of accepting the answer didn't change, in fact, I would argue that it was made much heavier.

To be clear about the scale of this: RavenDB did not suddenly get 12 times faster across the board. As much as I wish a 1K PR could do that. A single scenario got dramatically better. That is great, but it is also more code, more behavior to understand, and more surface area to maintain.

When we keep going like that, we end up with better software overall, but also a much higher mass of code to work with. It also means much higher effort from the people who actually have to deliver it.

I think that we’ll see major shifts in how we build and maintain software, but at the same time, I think that the key fundamentals haven’t really changed. In the past quarter, RavenDB (the company, not the software) hired more R&D staff than we did in the past two years, to give you some context.

There is going to be higher need for developers than ever, and even more digitalization of the world.