Intuitive explanation of bayes theorem - Part 1/2

One of the most important formula in statistics is perhaps the bayes theorem.

Many people see this formula as somewhat magical and struggle to understand its math or getting the intuition of what's going on. In fact, once you understand it, it may seem very trivial and I suspect that teachers don't spend the time needed to really dive into it, instead they just give the formula and ask you to know it by heart. As the formula is quite simple, students find it easy to memorize and don't really spend time to get into it either. Knowing a formula by heart may be useful to pass exams but it won't help you in your scientist journey :)
In the contrary, understanding the underlining concept will make you able to derive the formula instead of memorizing it.

I personnaly like to spend time on "easy" and "trivial" things because I consider them as the most important.
Furthermore, there are many ML models and concepts built on top of the bayes rule so I garantee you that the time spent is worth it.

Without wasting any more time, let's begin !

Conditional probabilities

As Bayes theorem is all about conditionnal probabilities, we need to understand first what is a conditionnal probability (wink wink captain obvious).

Let's consider a population of cars in your neighborhood.

Suppose that the distribution of the colors is like so:

red cars: 30

blue cars: 50

black cars: 100

yellow cars: 20

TOTAL: 200

If I ask you the question what's the probability of seeing a blue car within your neighborhood, you would answer:

P(blue car) = 50/200 = 1/4

This is a probability in its most simple form, it is simply a way to quantify how likely something is to happen.

What are conditional probabilities then ?

As their name suggest, it's when we are adding a condition or some may say a restriction.

Let's now add a condition:
There are taxi drivers among your neighbors and in your city taxis can be either black or yellow.
For now, we consider that the ensemble taxi is formed by yellow cars + black cars.

Now what is the probability of seeing a yellow car KNOWING that it's a taxi ?

This is written:

P(yellow car | taxi)

the vertical bar stands for "knowing that"

As the population is now restricted to only black and yellow cars we are no longer taking into account the whole population but only a subset. We answer the question "What's the amount of yellow cars within the taxis ?"

P(yellow car | taxi) = 20/ (20+100) = 1/6

Still pretty intuitive isn't it ?

The general form of a conditional probability can be written

P(A | B) = P(A ∩ B) / P(B)

the character ∩ symbolizes the intersection,
in our case A ∩ B = yellow cars ∩ taxis = yellow cars

and P(B) = P(taxi)

Once again conditional probabilities are nothing more than calculating probabilities on a subset of the population.

Let's stick with Taxis a bit more

In fact, not all yellow and black cars are taxis, to be fair taxis are also a subset of the ensemble yellow cars + black cars.
To make things clearer let's use this little diagram.

Let's play a bit with conditionnal probabilites in a visual way.

If a car is at the same time a taxi and a yellow car then it belongs to taxis ∩ yellow cars and it visually corresponds to this:

If we want the probability of a car being a taxi knowing that's it's a yellow car then we simply take the subset of yellow cars and see among it what is the proportion of taxis. Visually, it would look like that:

Got it ?

Sometimes it's more interesting to have the probability the other way around, that is to say having the probability of a yellow car knowing it's a taxi. No problem, within the subset of the taxis, what is the proportion of yellow cars ?

Let's put those two bad boys together in a single image so that we can compare them easily :)

Notice how these two probabilities share the same quantity in the numerator. In a way we can say that P(Taxi | Yellow car) and P(Yellow car | Taxi) are in fact "the same quantity seen through different glasses".

• P(Taxi | Yellow car) sees taxis ∩ yellow cars from yellow cars point of view
• whereas P(Yellow car | Taxi) sees taxis ∩ yellow cars from Taxi point of view

To me this is the key point to understand ! From there, we are one step away from "discovering the bayes theorem".

Bayes Theorem

The idea of Bayes Theorem is to have a simple formula that links P(A | B) to P(B | A).
This would be darn handy because in real life you sometimes have easily access to P(A | B) but your ultimate goal is to find P(B | A) so you want to be able to switch from one quantity to the other.

Let's get back to the previous image, what if we get rid of the denominators of each quantity ? To do so, we simply multiply each quantity with their denominator:

These quantities are now equal !

The last step is to divide each side by P(Yellow car)

Ladies and gentleman, I'm very happy to introduce you Bayes theorem :)

It's general form is :

Should I memorize the formula ?

Hello no ! If someday you forget the formula, just remember that the formula is designed to link P(A | B) with P(B | A) and that they share the same quantity P(A ∩ B) then the following steps are straight forwards as A ∩ B is equivalent to B ∩ A. I just used a little tip by introducing P(A) and P(B) in the equality.

An example

• We know the proportion of yellow cars among the taxis: P(Yellow | Taxi) = 0.5
• It is also given that 5% of the cars in the neighborhood are taxis.

What is the probability of a car being a taxi knowing that it's a yellow car (P(Taxi | Yellow)) ?

Let's solve this

Let's calculate first the numerator and the denominator of the formula.

Numerator = P(Yellow | Taxi) x P(Taxi)= 0.5 x 0.05

Denominator = P(Yellow) = 20/200 = 0.1

Finally,

P(Taxi | Yellow) = N/D = 0.5 x 0.05 / 0.1 = 0.25

P(Taxi | Yellow) = 25%

Extended version of the formula

Sometimes you'll see the bayes formula used with a longer denominator.
In fact it's still the same formula, it's just an "extended version".

Consider this situation:

we divided Taxi into two new groups: big taxis and small taxis.
in order to make it easier to write down things let's consider that:

• big taxis = A1
• small taxis = A2

A1 and A2 form a partition of the space Taxis. This means that the sum of A1 + A2 forms exactly Taxis, in other words there's no overlap between these two, or you can also say that they are mutually exclusive.

Now that you are confortable with the concept of partition, let's form a partition of Yellow cars.

Note that the c above Taxi in the last quantity stands for "not", i.e it represents all the cars that are not taxis.

Thus

Thus

To simplify writing let's say that not taxi = A3

Finally

Now let's apply bayes theorem to the probability of A1 given Yellow (P(A1 | Yellow)):

Let's plug into the formula the "extended version of P(Yellow)"

And voila !

The general form can be written like so:

Remember ! This holds true only if the events (A1,...,An) form a partition of the space, that is to say they are mutually exclusive and collectively exhaustive.

Merci ! This is the end of the article, I hope I made Bayes formula a little bit easier to understanded. :)