Sarah Constantin replies on MetaMed

Not long ago I linked to this Robin Hanson blog post on MetaMed.  I was sent this reply, which I will put under the fold:

I noticed you linked Robin Hanson’s article on MetaMed on Marginal Revolution.  I’m the VP of research at MetaMed, and I just wanted to tell you a little bit more about us, because if all you know about us is the Overcoming Bias article you might get some misleading impressions.

Medical practice is basically a mass-produced product. Professional and regulatory bodies (like the AMA) put out guidelines for treatment.  At their best, these guidelines follow the standards of evidence-based medicine, which means that on average they will produce the best health outcomes in the general population.  (Of course, in practice they often fall short of that standard.  For example, checklists are overwhelmingly beneficial by an evidence-based medicine standard, and yet are not universally used.)

But even at their best, the guidelines that are best from a population-health standpoint need not be optimal for an individual patient.  If you have the interest and the willingness to pay, investigating your condition in depth, in the context of your entire medical history, genetic data, and personal priorities, may well turn up opportunities to do better than the standardized medical guidelines which at best maximize average health outcomes.

That’s basically MetaMed’s raison d’etre.  And it’s a pretty conservative hypothesis, in fact.  We may harbor a few grander ambitions (for example, I come from a mathematical background and I’m working on some longer-term projects related to algorithmically automating parts of the diagnostic process, and using machine learning principles on biochemical networks in novel ways) but fundamentally the thing we claim to be able to do is give you finer-grained information than your doctor will.  We’re, of course, as yet unproven in the sense that we haven’t had enough clients to provide empirical evidence of how we improve health outcomes, but we’re not making extraordinary claims.

Robin Hanson seems to be implying that MetaMed is claiming to be useful only because we’re members of the “rationalist community.”  This isn’t true.  We think we’re useful because we give our clients personalized attention, because we’re more statistically literate than most doctors, because we don’t have some of the misaligned incentives that the medical profession does (e.g. we don’t have an incentive to talk up the benefits of procedures/drugs that are reimbursable by insurance), because we have a variety of experts and specialists on our team, etc.

The “rationalist” sensibility is important, to some degree, because, for instance, we’re willing to tell clients that incomplete evidence is evidence in the Bayesian sense, whereas the evidence-based medicine paradigm says that anything that yet hasn’t been tested in clinical trials and found a 5% p-value is completely unknown. For instance, we’re willing to count reasoning from chemical mechanisms as (weak) evidence. There’s a difference in philosophy between “minimize risk of saying a falsehood” and “be as close to accurate as possible”; we strive to do the latter.  So there’s a sense in which our epistemic culture allows us to be more flexible and pragmatic.  But we certainly aren’t basing our business model on a blanket claim of being better than the establishment just because we come from the rationalist community.


Metamed is a great idea. When I visit a doctor, I would like him to give me a probabilistic evaluation of diagnoses (ex: 80%: toxoplasmosis: 18%: mononucleosis; 2%: lymphoma) with references in the medical literature that I can check to verify his calculations. We are very far from this. When you visit your doctor, ask him to write down the diagnoses and assign probabilities to each. He can't, he does not think that way. This is one of the reasons many unnecessary procedures are carried out. Major reforms are needed in the execution of first line health care, and Metamed is leading the way.

I would guess, based on my experience with various professions, that some doctors think readily in probabilistic terms, and some don't. For those who don't, the question might have to be rephrased as "What if that doesn't work, what will we try next?" or "If that's not the answer, what is your next best guess?" I would be very surprised if you found any doctors who couldn't give useful answers when the question is phrased in strategic or analytic terms, though those rephrasings are of course isomorphic to the probabilistic framing.

Doctors constantly think in probabilistic terms. One of the central skills is making a "differential diagnosis" - thinking of all the possible etiologies, from most likely to least likely. You may have heard the common doctor expression: "when you hear hoofbeats, think horses instead of zebras." I suspect most patients, who have very poor understanding of statistics, don't want to hear their doctor reel off some statistics - they want to know what's wrong and what to do about it.

MetaMed, like IBM's Watson and "decision-support" software in EMRs, all seem to misunderstand the problem. Making diagnoses isn't the hard part, despite what TV would have you believe. In outpatient, it's all about management of multiple chronic diseases. In the hospital, it's managing the consequences of various disorders that are inherently difficult to treat (if they were easy to treat, you wouldn't need to be in the hospital).

If only all companies were both OPEN and ABLE (i.e. competent enough) to responding like this, the world would be a better place.

Every time that I have checked MetaMed's Team page in the last several weeks, the number of employees listed has shrunk; there used to be more than double the number currently listed. Does this reflect actual changes in employment, or just increasingly strategic selection of which employees to highlight?

Does MetaMed still employ Hank Gardstein, who appears to have lost his medical license for selling prescription drugs?

We are no longer working with Hank Gardstein.

And the shrinking employee numbers?

Now, if only there were a few competitors. It's nice to be able to have a second opinion from another doctor...can I get a second opinion from a different MetaMed? Right now, I don't think so. I hope the market grows, though, and that some new entrants make this market more robust to bias

I reply in an added to my original post here:

Wow. I'm totally blown away. I have no idea if what she is saying about MetaMed is true (I've never heard of MetaMed before), but just the fact that a corporate executive can write something that is intelligent and statistically literate has made my day. Certainly the premise - using all the relevant information together with the best statistical techniques to improve patient outcomes - should, if correctly implemented, result on average in better outcomes than a generic policy that uses only a subset of the relevant info and does not employ the best statistical methodology. I'm definitely going to check this out, as well as Hanson's comments.

I'm all for the things motivating metamed, but I doubt metamed is going to demonstrate much in the way of comparative advantage compared to 'bog standard' medical care.

Constantin's reply seems to be imply Metamed will do better than 'bog standard medicine' in two ways.

1) Better personalization than 'one size fits all'
2) Better interrogation of research than the average doctor.

I doubt both.

The 'improved personalization' case doesn't fly: in the vast majority of cases, medical science has not got to the stage where we are aggregating/discarding personalized data re. family history, ethnic background, medical history, etc. in making our guidelines. Rather, so much of our practice is based on poorly evidenced tradition and fad awaiting cursory evidentiary input (e.g. "Does this drug we give for heart failure *increase* or *decrease* mortality?" "Does this drug we prescribe to everyone with Atrial Fibrilation *increase* survival times", "Do patients intubated pre-hospital by paramedics in cardiac arrest fare better than those who don't - and emergency med and cardiology are relatively *good* in terms of evidence based medicine!)

I just don't see the low hanging fruit along the lines of "Treatment X is generally good for those with condition Y, but for patients like yourself with A, B, and C, do Z instead!" that Metamed will pluck (let alone with high enough frequency to tempt punters to drop $5000 on it).

Worse, although it is contrarian, medicine should probably be *more* 'one size fits all': experienced doctors tend to veer from guidelines too often by thinking factors they divine make their patients exceptions from 'standard treatment'. So strict guideline following generally has better outcomes than expert judgment.

re. 2) although individual doctors cannot compete with Metamed with time spent looking at a condition, most are wise enough to pool their abilities: resources like NICE (UK NHS) or Uptodate (Private firm) have groups of domain experts regularly reviewing the state of the evidence about condition X re. recommended treatments and actions to take, and most doctors in the UK use both (I gather US medics have similar sorts of things). I would take these over Metameds non-experts, and I don't think the average NICE technology appraisal is far off the mark in a way rationalists are going to improve upon. I look forward to being mistaken.

But hey, maybe I'm wrong (and being a med student, perhaps I'm biased towards the establishment). There are plenty of ways Metamed can demonstrate their superiority compared to bog standard medicine (case series of superior outcome, reports of number of times their recommendation differs from original medical recommendation, or best practice according to guidelines). Until then, I'm sceptical.

I don't know about that so much. My wife had an apparently rare issue a few years back, rare enough that 4-5 different doctors specializing in that system didn't have any idea what was wrong. We eventually found a doctor that had dealt with the issue before, and he got it fixed up after a few month's treatment. The lesson I took away is that most doctors are just like any other professional: only familiar with the most common issues they deal with plus a few that interest them in particular. Only a few will have a very broad knowledge if only because not many people get the rare things, and since there doesn't seem to be a lot of incentive for doctors to really dig into a specific patient's problem instead of sending them somewhere else and seeing more patients in that time, I doubt many get much farther than "the usual stuff."

all the controversy is only and exclusively about the price. People hate ripoffs. Hanson should write about that.

I agree with Tyler's original post that Metamed is not something I would shell out for. But I see the comparative advantage lying not in Metamed's knowledge base or the customization of their answers but in the fact Metamed does not profit by, say, recommending surgery whereas your doctor does.

I have a hard time believing MetaMed is not incentivized to recommend new, zoomy therapies and/or screening tests. After all, what's the use of the service if they're just recommending society guidelines to follow. And, in a world where successful randomized trials have failed to hold up, drawing conclusions from non-significant results seems like a particularly poor idea.

It's easy to do everything- its hard to choose the right things to do.

Many of the practices of the medical establishment are designed to minimize legal risk related to malpractice. Communicating nuanced evaluations of someone's health needs a sophisticated patient as much as a sophisticated practitioner.

How much medical malpractice liability insurance do they hold?

If I have health issues and lots of disposable income, I wonder what the better investment is: a personalized treatment plan or personal genomics?

It's not either-or. Somebody has to evaluate the credibility and risk of the alleles "associated" with different traits or chances of disease. GWAS studies are notoriously hard to replicate. If you have a few thousand dollars, I'd spend part of it on a SNP chip ($100-300) and then get a report on what your genes actually mean, over full sequencing ($5000) and no interpretation. Of course, if you only have a $100 budget, you probably have more time than money so you should get the chip and do your own reading.

Comments for this post are closed