It would take too long to sum up the dialogue, by this point you are either following the discussion or not. A few points:
1. Bryan Caplan asks a good question: "You know now that the price of oil will be flat for five years, then
fall by 10% per year every year thereafter. Everyone else thinks the
price will be flat forever." Can you profit? Yes, by rolling over your short positions and constructing a synthetic long-term bet, even if the current futures markets extend for only three years or for that matter one year. Fischer Black said so, so in other words you do have a chance to put your money where your mouth is.
2. Arnold Kling wonders why so many commodity prices have risen at the same time. I’ll repeat that fundamental value — and thus the concepts of speculation and bubble — are trickier and vaguer with commodities than in stock markets. I’ll say that "expectations" have driven the general rise in commodity prices. If those expectations turn out to be wrong, we can call it all a bubble; if they turn out to be right, then it hasn’t been a bubble. What should we call it in the meantime? We’re not going to solve that problem in any factual way. Make your bets, as they say.
3. Bryan Caplan notes that commodity prices always have fallen back down in the past and argues that is likely to happen again in the future. I say no, the current price is your best (rough) estimate of scarcity (adjusting for storage costs), don’t expect mean-reversion, future returns (but not prices) are a random walk, and extrapolation is a dangerous method to apply to financial time series. (For instance every time the stock market has fallen it has bounced back up again but that does not mean you can earn supernormal returns by buying on the downticks; even Shiller finds only small gains here.) I love Julian Simon too but don’t let him overrule Eugene Fama.
4. Mark Thoma has an exhaustive post on convenience yield. The models used are too piecemeal and they allow "inventories," "convenience yield," and "speculation," to serve as free-floating, not necessarily attached concepts. The discussion here pays insufficient attention to Holbrook Working, who knew that convenience yield was front and center of the entire analysis, just as "the demand for money" is the centerpiece of the quantity theory. Working himself didn’t even think that "speculation" was a well-defined concept in commodities markets; even if he went too far there the concept remains murky. The current discussions are mixing fundamental conceptual definitions with some broader institutionally-motivated definitions and thus none of the results quite match up.
5. Contra Paul Krugman, invoking convenience yield should not be thought of as an Ptolemaic epicycle or a fudge factor. The demand to hold oil is the starting point of the whole analysis, see also Jeffrey Williams’s work. The upshot is that if speculation were driving the current price, it would be consistent with either a premium of the futures price over the spot or vice versa; invoking convenience yield to explain the relatively cheap futures is what you might expect in the first place, speculation or not, bubble or not.
6. Interfluidity has the most careful and accurate exposition of the relevant market relationships, mostly because he sticks closely to the Holbrook Working tradition.
7. The bottom line is that when it comes to the key substantive questions about the oil market – why are prices so high — the correct answer is the Lachmannian one: "expectations." If you push one step further on that, and try to evaluate or "source" those expectations, the correct answer is "we don’t know." Jim Hamilton hints at some of this — and the imprecision of the "inventories" term — in this insightful post.
Addendum: On other practical matters, this new Op-Ed by Paul Krugman is essentially correct, although his claim that speculation is impossible in the iron ore market shows, better than anything else, the oddity of his semantic choices.