I can think of a few candidate theories:
1. His views are the right views, more or less, and American voters recognized this.
2. A quite significant percentage of America is very directly racist. I don’t mean statistical discrimination here, I mean “downright racist.”
3. Give Ray Fair (NYT) his Nobel Prize right here and now, economic conditions truly predict election results at the national level.
4. The “third term Party fatigue” effect is stronger in national elections than we had thought.
5. Hillary Clinton is a weaker candidate than many people had thought. Maybe so, but that has to be unpacked a bit more. I would try “the Democratic national establishment doesn’t understand why much of America trusts it so little, so it keeps on doing and saying unpopular things. Those things include elevating some candidates and also encouraging them to take particular stances.”
6. As Robert D. Putnam suggested, ethnic diversity can lower the quality of governance, and this is one step along that path toward greater fractiousness. This may blend into racism, but much of it is simply “fear of being in the losing coalition.” The common claim that the electorate is more polarized than before fits into this. You might try Ezra Klein’s podcast with Arlie Hochschild.
7. America is not ready for a woman president. Or maybe it has to be a different kind of woman president, noting that Hillary, while she has passed through many filters, has not passed through the “truly popular with normal voters filter” in the same way that say Thatcher and Merkel did. And no, New York isn’t normal, sorry people.
8. The Democrats have plenty of policy proposals, but only the Republicans are running on ideas. And very often an idea beats no idea, even if the idea on the table is a bad one.
I don’t agree with #1, and while #4 sounds like a plausible part of the story to me, as a truly major explanation I find it hard to square with Obama’s continuing popularity. #3 kicks in but as a dominant force, it seems hard to elevate when median household income just grew at 5.2%, inflation is low, there is no major war, gas prices are low, and asset prices are high.
On #2, I see #5 as a more convincing statement of related ideas, while admitting #2 is a factor. How well the Democrats do in the Senate might give us some bead on the relative import of #5.
Overall I am seeing a lot of room for #5 and #6 and #7 and #8. Presumably 5, 6, and 8 are hard for many Democrats to admit, and I genuinely wonder how their thoughts run in the quiet of their homes. Some are plugging hard for an extreme version of #2, but, as long as we are considering matters of prejudice, I find the gender bias of #7 easier to swallow. We did after all just elect Obama for two terms in a row, and we have never ever had a woman president or even a serious contender before.
If, I wish to stress that word if. But that he is still in the running, and making it close, is reason enough to ponder these questions.