2. BASE RATE FALLACY

Eight Rules 1. False Precision 3. Correlation Not Causation 4. Single Variable 5. Substantial Differences 6. Longitudinal Study 7. Different From Chance 8. Dose-Response

An Organization Suspects Syping Decides To Give Everyone A Polygraph       —   Polygraph Accuracy is 90%       —   [Actual Number Is More Like 70%]       —   90% Accuracy ==> Should Detect 90% of Spies Who Would Object To This?   It Seems Quite Reasonable . . .

IF YOU LEARN JUST ONE THING IN THIS COURSE, IT SHOULD BE THIS

EXAMPLE 10,000 people in organization 10 spies (.1%) Polygraph detects 9 of the spies (90%) Again, who would object to this?   It seems quite reasonable . . .

TOP

EXAMPLE If there are 10,000 people in the organization       —   9 of the 10 spies will be identified       —   90% accuracy ==> 10% failure (false positive)       —   10% of remaining 9991 ==> 999 identified as spies       —   Total number identified as spies = 999 + 9 = 1008 Actual accuracy = (actual spies)/(number id'd as spies) = 10/1008 = 1%

LET'S TRY AGAIN WITH 99% POLYGRAPH ACCURACY . . . If there are 10,000 people in the organization       —   9.9 of the 10 spies will be identified       —   99% accuracy ==> 1% failure (false positive)       —   1% of remaining 9990 ==> 99 identified as spies       —   Total number identified as spies = 99 + 10 = 109 Actual accuracy = (actual spies)/(number id'd as spies) = 10/109 = 9%

TOP

Next, we can generalize to show the relationship between the advertised accuracy and the true accuracy:

True Accuracy   =   B/(B   +   F)   =   1/(1 + F/B)

where F is the failure rate of the test and B is the Base Rate

What do you do to improve accuracy of the test?   Apply test a second time   Not feasible for all tests   —   Drug tests - yes   —   Polygraph - no   Apply a more accurate test   Not feasible for all tests   —   Drug tests - yes   —   Polygraph - no

TOP

Next