First, the developers claim that in 274 known cases of melanoma, the device detected all 274 of them, which gives and estimated test sensitivity of 100%, which is patent bullshit; nothing in this life is guaranteed perfect. A reasonable Bayesian shrinkage estimator might be 275/276 = 0.9964, which is still pretty good. Second, the developers conveniently omitted any statistics on specificity (how well cases of NO melanoma are correctly identified); I’m sure this was simple oversight*. Third, and finally, there’s no mention of the prevalence of melanoma in the general population. Without all three of these numbers, we have no way of evaluating the utility of the device.
But what the heck, let’s do a back of the envelope** calculation. We have the estimate for sensitivity, and we’ll be optimistic and assume that the other 726 subjects were all melanoma-free and tested negative, so we’ll get a shrinkage estimate of 727/728=0.9986 (which should make it a pretty good test). Then look up the prevalence of melanoma (also called incidence rate) to get 1/5074 = 0.000197. Then plug into Bayes’ Theorem to get the posterior predictive value:
PPV = (sensitivity x prevalence) / (sensitivity x prevalence + (1-specificity)x(1-prevalence)) = 0.1251
So if you test positive with this gizmo, you have about a 12.5% chance ( 1 in 8 ) of having melanoma. I think we need more test data before buying any stock.
Tip from the Instapundit, who should calm down.
* I’m sure there’s an Easter Bunny, too.
** OK, back of the spreadsheet
Update (8 February). Jan Nordgren has the perfect quote to describe this situation. I wish I had used it for my title.