How to think about AI progress

The Zvi has a good survey post on what is going on with the actual evidence.  I have a more general point to make, which I am drawing from my background in Austrian capital theory.

There are easy projects, and there are hard projects.  You might also say short-term vs. long-term investments.

The easier, shorter-term projects get done first.  For instance, the best LLMs now have near-perfect answers for a wide range of queries.  Those answers will not be getting much better, though they may be integrated into different services in higher productivity ways.

Those improvements will yield an ongoing stream of benefits, but you will not see much incremental progress in the underlying models themselves.  Ten years from now, the word “strawberry” still will have three r’s, and the LLMs still will tell us that.  There are other questions, such as “what is the meaning of life?” where the AI answers also will not get much better.  I do not mean that statement as AI pessimism, rather the answers can only get so good because the question is not ideally specified in the first place.

Then there are the very difficult concrete problems, such as in the biosciences or with math olympiad problems, and so on.  Progress in these areas seems quite steady and I would call it impressive.  But it will take quite a few years before that progress is turned into improvements in daily life.  Again, that does not have to be AI pessimism.  Just look at how we run our clinical trials, or how long the FDA approval process takes for new drugs, or how many people are reluctant to accept beneficial vaccines.  I predict that AI will not speed up those processes nearly as much as it ideally might.

So the AI world before us is rather rapidly being bifurcated into two sectors:

a) progress already is extreme, and is hard to improve upon, and

b) progress is ongoing, but will take a long time to be visible to actual users and consumers

And so people will complain that AI progress is failing us, but mostly they will be wrong.  They will be the victim of cognitive error and biases.  The reality is that progress is continuing apace, but it swallows up and renders ordinary some of its more visible successes.  What is left behind for future progress can be pretty slow.