Bayes’ Theorem

Bayes’ Theorem: Statement and Formula

Introduction and Purpose

Bayes' Theorem, named after the 18th-century English statistician and theologian Thomas Bayes, is a fundamental theorem in probability that provides a framework for updating the probability of a hypothesis based on new evidence or information. It describes how to revise existing probabilities (prior probabilities) in light of new data to arrive at updated probabilities (posterior probabilities).

In essence, Bayes' Theorem allows us to calculate a conditional probability $P(E_k|A)$ (the probability of event $E_k$ given that event $A$ has occurred) using the "reverse" conditional probabilities $P(A|E_i)$ (the probability of $A$ given various conditions $E_i$) and the initial probabilities of those conditions $P(E_i)$.

It answers the question: Given that a particular outcome or piece of evidence ($A$) has been observed, what is the revised probability that it was caused by or is associated with a specific initial event or hypothesis ($E_k$)?

Setup and Statement of Bayes' Theorem

To state Bayes' Theorem formally, we first need to define a partition of the sample space and the event representing the new evidence.

Let $\{E_1, E_2, \dots, E_n\}$ be a set of $n$ events that form a **partition** of the sample space $S$. This means these events are:

• **Mutually Exclusive (Pairwise Disjoint):** No two events can occur simultaneously in a single trial. $E_i \cap E_j = \phi$ for all $i \neq j$.
• **Exhaustive:** Their union covers the entire sample space. $E_1 \cup E_2 \cup \dots \cup E_n = S$.
• **Non-Zero Probability:** Each event $E_i$ has a positive probability, $P(E_i) > 0$ for all $i=1, 2, \dots, n$. (This condition is assumed to avoid division by zero in the formula).

Let $A$ be any event associated with the sample space $S$ such that $P(A) > 0$. Event $A$ represents the observed evidence or outcome.

**Bayes' Theorem states** that for any specific event $E_k$ from the partition (where $k$ is one of $1, 2, \dots, n$), the conditional probability of $E_k$ occurring given that event $A$ has occurred is given by the formula:

And by using the Multiplication Rule ($P(E_k \cap A) = P(E_k) P(A|E_k)$) for the numerator and the Law of Total Probability ($P(A) = \sum_{i=1}^{n} P(E_i) P(A|E_i)$) for the denominator, the formula becomes:

This expanded form is the most commonly used version of Bayes' Theorem in applications.

Derivation of Bayes' Theorem

Bayes' Theorem can be easily derived from the basic definition of conditional probability and the Law of Total Probability.

1. Start with the definition of conditional probability for $P(E_k|A)$:

   $$P(E_k|A) = \frac{P(E_k \cap A)}{P(A)} \quad (\text{assuming } P(A) > 0)$$

   ... (iii)

2. Apply the general Multiplication Rule (Formula 2 from Section I1) to the numerator, $P(E_k \cap A)$:

   $$P(E_k \cap A) = P(E_k) P(A|E_k) \quad (\text{assuming } P(E_k) > 0)$$

   ... (iv)

3. Apply the Law of Total Probability (Formula 2 from Section I4) to the denominator, $P(A)$. Since $\{E_1, E_2, \dots, E_n\}$ is a partition of $S$, the overall probability of event $A$ is the sum of the probabilities of $A$ occurring under each $E_i$:

   $$P(A) = P(E_1) P(A|E_1) + P(E_2) P(A|E_2) + \dots + P(E_n) P(A|E_n)$$

   $$P(A) = \sum\limits_{i=1}^{n} P(E_i) P(A|E_i)$$

   ... (v)

4. Substitute the expression for the numerator from step 2 (equation iv) and the expression for the denominator from step 3 (equation v) back into the formula from step 1 (equation iii):

   $$P(E_k|A) = \frac{P(E_k) P(A|E_k)}{\sum\limits_{i=1}^{n} P(E_i) P(A|E_i)}$$

   ... (vi)

This completes the derivation of Bayes' Theorem.

Terminology in Bayes' Theorem

The terms in Bayes' Theorem have specific interpretations, particularly in the context of updating beliefs:

• Prior Probability ($P(E_k)$):

  This is the initial probability assigned to event $E_k$ being true, **before** any new evidence (event $A$) is observed. It represents our initial belief or knowledge about the likelihood of $E_k$.

• Likelihood ($P(A|E_k)$):

  This is the conditional probability of observing the specific evidence $A$, **given** that event $E_k$ is true. It measures how likely the evidence is under the assumption that $E_k$ is the case. Note that this is $P(A|E_k)$, not $P(E_k|A)$.

• Evidence ($P(A)$):

  This is the overall (marginal) probability of observing the evidence $A$. It is calculated in the denominator using the Law of Total Probability, summing up the probabilities of $A$ occurring under all possible scenarios $E_i$. It acts as a normalizing constant, ensuring the posterior probabilities sum to 1.

• Posterior Probability ($P(E_k|A)$):

  This is the updated probability of event $E_k$ being true, **after** observing the evidence $A$. It represents our revised belief about the likelihood of $E_k$ having accounted for the new information provided by event $A$.

Bayes' Theorem provides a formal mechanism for learning from data – starting with a prior belief and updating it based on observed evidence to arrive at a posterior belief.

Applications of Bayes’ Theorem (Solving Problems)

Solving Problems using Bayes' Theorem

Bayes' Theorem is a powerful tool for solving problems where we are given the results of an experiment (the evidence) and want to determine the probability of a specific cause or condition that could have led to that result. These are often referred to as "inverse probability" problems because we are inferring back to the cause ($E_k$) based on the effect ($A$).

The general strategy for applying Bayes' Theorem to solve problems is outlined below:

1. Define the Events Clearly:

   • Identify and define the set of mutually exclusive and exhaustive events ($E_1, E_2, \dots, E_n$) that represent the different possible scenarios, causes, or hypotheses that could exist or occur at the first stage of the problem. These events must form a partition of the sample space ($S$).
   • Identify and define the event $A$ that represents the observed evidence, outcome, or result obtained from the experiment.

2. Assign the Known Probabilities:

   Determine the numerical values for the probabilities required:

   • **Prior Probabilities ($P(E_i)$):** The probabilities of each of the partition events $E_i$ occurring *before* the evidence $A$ is known. These should sum to 1 ($\sum P(E_i) = 1$).
   • **Conditional Probabilities (Likelihoods, $P(A|E_i)$):** The probabilities of observing the evidence $A$ *given* that each of the partition events $E_i$ has occurred.

3. State the Goal:

   Clearly identify which posterior probability you need to calculate. This will always be of the form $P(E_k|A)$ for a specific event $E_k$ from the partition, given the evidence $A$.

4. Calculate the Denominator ($P(A)$):

   Calculate the overall probability of the evidence $A$ occurring. This is typically done using the Law of Total Probability:

   $$P(A) = P(E_1) P(A|E_1) + P(E_2) P(A|E_2) + \dots + P(E_n) P(A|E_n) = \sum\limits_{i=1}^{n} P(E_i) P(A|E_i)$$

   ... (1)

   This sum serves as the denominator in Bayes' formula.

5. Calculate the Numerator:

   Identify the specific term in the sum from step 4 that corresponds to the event $E_k$ whose posterior probability $P(E_k|A)$ you want to find. The numerator of Bayes' formula for $P(E_k|A)$ is $P(E_k) P(A|E_k)$.

   Numerator for $P(E_k|A) = P(E_k) P(A|E_k)$

   ... (2)

6. Apply Bayes' Formula:

   Divide the numerator (step 5) by the denominator (step 4) to get the desired posterior probability:

   $$P(E_k|A) = \frac{P(E_k) P(A|E_k)}{P(A)}$$

   ... (3)

   Or, writing out the denominator fully:

   $$P(E_k|A) = \frac{P(E_k) P(A|E_k)}{\sum\limits_{i=1}^{n} P(E_i) P(A|E_i)}$$

   ... (4)

7. Interpret the Result:

   State your final answer in the context of the original problem, explaining what the calculated posterior probability means.

Example

Answer:

Answer: