Steven's Random Scribes

Steven's Random Scribes

Prosecutors Fallacy

What is Prosecutors Fallacy?

In its simplest form is faulty reasoning, usually statistical, used by prosecutors. If you understand the words prosecutor and fallacy the term is simple to understand. The fallacy derives from the court system when one side (there’s also the concept of defence attorney’s fallacy) makes an argument that over simplifies a statistical inference and incorrectly states the odds of guilt (or innocence).

Examples

Conditional Probability

Let’s assume that the owner of a convenience store wins the lottery from a ticket that came from her store. Fishy eh? Well the Crown through it was fishy too. So the charged the store own with fraud. The prosecutor argues that because the chances of winning the lottery are so small, let’s say 1 out of a 10 million, that the store owner must have modified her chances to win. Given the number of people who play the lottery, the probability of someone winning it is quite low.

Berkson’s Paradox

In a famous British case. The mother of two children who both died as early infants (11 weeks and 8 weeks) was accused of murdering them. She was convicted, in part because of the expert testimony of paediatrician that testified that the probability of two children in the same family dying from SIDS is about 1 in 73 million. This was based off the assumption that the deaths were independent (e.g. having one child die of SIDS doesn’t increase the probability of having another child die of SIDS). It turns out that there isn’t independence. Parents who have one child die of SIDS have an increase likeliness to have another die. There’s also the argument that having two infants murdered is very unlikely, just like both of them dying from SIDS. Neither of these two arguments were brought up at trial.

The woman was convicted, but appealed and was acquitted after three years in jail. Tragically, she developed numerous psychiatric problems and died of alcohol poisoning.

Multiple Comparisons

DNA evidence can also lead to the prosecutor’s fallacy. Let’s say DNA evidence is compared to a database that contains a 100,000 samples. In this example a match is made. An expert witness testifying that the probability of a matching profile being made is 1 in 50,000. This database had 100,000 observations. So that means that 100,000 possible matches. We’d expect that there would even be 2 matches. The probability of getting at least 1 match in the database would have been:

1 - (1 - (1 / 50000)) ^ 100,000 = 86.47%

Without other evidence, this match could have been do to chance.

Business Examples

While researching this topic I struggled to come up with business examples. It’s because at its heart, this fallacy is just incorrect reasoning used be prosecutors. The examples that I would be looking for would be ways that business use faulty reasoning. In healthy business you don’t have cases where you have a prosecutor and a defendant. With that in mind I did come up with a few scenarios that I could see occurring:

HR

Let’s say that 1 out of 100 people lie about their work history on their resume. HR hires a new tech startup that automatically reads and filters resumes that are untruthful. They have a 5% false positive rate. HR was enthusiastic with the results. They stopped themselves from hiring 10 potential candidates in the first month. The director of HR was so enthralled, he wanted to run their existing employees through the system and discipline any of those that lied. He tested 200 existing employee resumes. He flagged Kelly’s resume. What’s the probability that Kelly actually lied on her resume?

Well because of the 5% false positive rate, we’d expect 10 people (200 * .05) to be incorrectly flagged as lying. Since only 1 out of 100 people lie about their work history, only 2 employees (.01 * 200) actually did. So even if Kelly is flagged she still only has a 20% (2 / 10) chance of actually lying on her resume.

Marketing

1 out of every 10 leads are big fish. Big Fish have an average annual contract value of $1,000,000. Regular leads have an annual contract value of only $50,000. The company’s goal is to increase annual revenue by $10 M. 5% of leads close. According to this information, one would expect to hit the $10 M goal with:

10,000,000 / ((.1 * 1,000,000 + .9 * 50,000) * .05) = 1,380 leads (round up)

At the end of a year, marketing provided 1,500 leads to sales. Unfortunately, the company missed their sales target by a staggering $4.15 M. The CMO and CRO got into an argument over who is at fault over the missed sales target. The CRO eventually concedes having no evidence to refute marketings claims. She returns to the sales floor expecting that her time with the company may be coming to an end. A couple of interns in the sales department heard about the numbers that marketing came up with. They were shocked, everything seemed to be going well on the sales floor. Close rate hasn’t dipped, their annual contract value even increased by 20% compared to last year. They realize a missing piece in marketing’s reasoning, not all leads close at the same rate. Big Fish close at a rate of 1%. They update the calculation:

10,000,000 / ((.1 * 1,000,000 * .01 + .9 * 50,000 * .05)) = 3,077 leads (round up)

They report their findings to the CRO, immediately lifting the CRO’s spirit. She never liked the CMO anyway.

Wrap-up

As you may have noticed, the examples that I have provided are just fallacies. Prosecutors Fallacy is a fallacy that’s done in an accusatory way.

Much of this post was based on the Wikipedia Article on Prosecutor’s Fallacy