Not quite ready for prime time

This sounds like a great innovation, using Raman spectroscopy to detect melanoma.  But let’s look a little deeper at the preliminary results.

First, the developers claim that in 274 known cases of melanoma, the device detected all 274 of them, which gives and estimated test sensitivity of 100%, which is patent bullshit; nothing in this life is guaranteed perfect.  A reasonable Bayesian shrinkage estimator might be 275/276 = 0.9964, which is still pretty good.  Second, the developers conveniently omitted any statistics on specificity (how well cases of NO melanoma are correctly identified); I’m sure this was simple oversight*.  Third, and finally, there’s no mention of the prevalence of melanoma in the general population.  Without all three of these numbers, we have no way of evaluating the utility of the device.

But what the heck, let’s do a back of the envelope** calculation.  We have the estimate for sensitivity, and we’ll be optimistic and assume that the other 726 subjects were all melanoma-free and tested negative, so we’ll get a shrinkage estimate of 727/728=0.9986  (which should make it a pretty good test).  Then look up the prevalence of melanoma (also called incidence rate) to get 1/5074 = 0.000197.  Then plug into Bayes’ Theorem to get the posterior predictive value:

PPV = (sensitivity x prevalence) / (sensitivity x prevalence + (1-specificity)x(1-prevalence)) = 0.1251

So if you test positive with this gizmo, you have about a 12.5% chance ( 1 in 8 ) of having melanoma.  I think we need more test data before buying any stock.

Tip from the Instapundit, who should calm down.

* I’m sure there’s an Easter Bunny, too.

** OK, back of the spreadsheet

Update (8 February).  Jan Nordgren has the perfect quote to describe this situation.  I wish I had used it for my title.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s