Probability 101: The Ultimate Beginner’s Guide to Mastery

What is probability?

Probability is a branch of mathematics that deals with quantifying uncertainty in events. Let us assume a classic example a coin toss. When a coin is tossed, it can either be head or tail. Probability helps us understand uncertainty; it quantifies information. 

In coin toss:

probability of Heads = probability of Tails = 0.5

We have said probability and quantified it, but what does that mean?

When I say the probability of heads is 0.5, it means a 50% chance is there for heads to occur. When I flip a coin it’s either heads or tails, we cannot say it’s 50% heads. Probability says that when I repeat the experiment repeatedly many times, 50% of those iterations will be Heads and 50% of them will be Tails. 

If I do a coin toss 10 times the outcome can be 7 tails and 3 heads. If I do it 100 times the outcome can be 65 tails and 35 heads, not exactly 50%. But as I keep increasing the number of iterations the occurrence rate will be much closer to 50%. That’s precisely what probability indicates.

Event, Sample Space and Favourable Outcomes

In probability, the terms event, sample space, and favourable outcomes are fundamental concepts that help define and calculate the likelihood of an occurrence.

Event:

An event is a specific outcome or a set of outcomes we are interested in. It is a subset of the sample space. For example, if you’re rolling a die, an event could be rolling an even number (2, 4, or 6).

• Example: Rolling a 4 on a die is an event. So is rolling an even number or getting heads on a coin toss.

Note: In continuous distributions, probability will be assigned to a subset of sample space, rather than a point. This subset is called an event.

Ex: Probability of student’s weight to be between 65 and 70

Types of Events:

• Simple Event: Event with a single outcome
• Compound Event: Event with multiple outcomes
• Independent Events: Two Events are said to be independent if the outcome of one event doesn’t depend on the result of the other event.
• Mutually Exclusive: If one outcome occurs, another outcome from the set cannot happen in an experiment.
  

Sample Space(Ω):

The sample space is the set of all possible outcomes of an experiment. It includes every potential result that could occur. For example, when flipping a coin, the sample space consists of two outcomes: {Heads and tails}. Sample space should be mutually exclusive and collectively exhaustive.

Collectively Exhaustive: The outcome of an experiment must be available in sample space. No other outcome should be possible

Example: For a die, the sample space is {1, 2, 3, 4, 5, 6} because these are all the possible numbers you can roll. Refer here for a better understanding of rolling a dice.

Favourable Outcomes:

Favourable outcomes are the specific outcomes that satisfy the event of interest. They are the outcomes that lead to the success of an event. The number of favourable outcomes is used to calculate probability.

• Example: If the event is rolling an even number on a die, the favourable outcomes are {2, 4, 6}, because these are the outcomes that satisfy the condition of the event.

Probability Formula

Example:

Let’s take the example of rolling a die.

• Sample space = {1, 2, 3, 4, 5, 6}
• Event: Rolling an even number
• Favorable outcomes = {2, 4, 6}

The probability of rolling an even number would be:

P(Even number)=3/6=0.5 or 50%

In summary:

• The sample space contains all possible outcomes.
• The event is the outcome or set of outcomes you are interested in.
• The favourable outcomes are the outcomes within the sample space that satisfy the event.

Probability Axioms

Axioms are rules that probability should follow.

1. Probability should be Non-negative
2. The probability of sample space is 1
3. The probability of disjoint sets is 0, A∩B = ø (Null set) then p(A∩B) = 0
   →p(A∪B) = p(A) + p(B) + p(A∩B) → p(A∪B) = p(A) + p(B) 

These are the only three rules, that need to be followed by any probability model to be legit.

Well, some of you may ask, shouldn’t there be a new condition: the probability of an event should be between 0 and 1? 

Yes, but it has been defined by the above axioms:

1. Probability should be non-negative → P > 0
2. The sample space probability is 1, and we know that sample space is mutually exclusive.
   1. Let sample space = {event1, event2,……, eventn)
3. By using the third axiom, and taking into consideration sample space is mutually exclusive:
   1. p(Sample Space) = p(event1) + p(event2) + …… + p(eventn)

So p(sample space) is 1, p(event)>0, so by law p(event) should be less than 1 to satisfy the axioms. In this way, all the more rules you can think of on probability are explained by just these 3 axioms. 

The 4 stages of probability calculation

1. Identify the Sample Space

Begin by defining the sample space, which is the set of all possible outcomes in the experiment. The sample space includes every possible result, ensuring you have a collectively exhaustive set.

Example: If you’re rolling a fair die, the sample space is {1, 2, 3, 4, 5, 6}.

2. Specify the Probability Law

Next, determine the probability law that governs how likely each outcome in the sample space is. This could be based on a uniform distribution (where all outcomes are equally likely) or a different distribution based on the problem’s specifics.

Example: For a fair die, each number has an equal chance of occurring, so the probability of each outcome is 1/6. It is Discrete uniform law(More on that is available later in the article)

3. Identify the Event of Interest

Now, identify the specific event you want to calculate the probability for. The event is a subset of the sample space that consists of the outcomes you’re interested in.

Example: If you’re looking for the probability of rolling an even number, the event of interest is {2, 4, 6}.

4. Calculate the Probability

Finally, calculate the probability of the event by dividing the number of favourable outcomes (those in your event) by the total number of possible outcomes (in the sample space). This ratio gives you the probability of the event occurring.

Example: For rolling an even number on a die, there are 3 favourable outcomes (2, 4, 6) out of 6 total possible outcomes, so the probability is:

P(Even number)=3/6​=0.5 or 50%

Now, that we have seen how to calculate probability, let’s understand some probability laws and rules that define how probability works in different cases.

The Guide: Probability Laws and Rules

1. Law of Large Numbers

The Law of Large Numbers is one of the foundational principles in probability theory. It states that as the number of trials in an experiment increases, the average results will converge to the expected value (the theoretical probability). If you repeat an experiment many times, the empirical probability (what you observe) will get closer to the true probability.

Explanation:

Think of flipping a coin. If you flip a fair coin just 10 times, you may not get exactly 5 heads and 5 tails. However, if you flip it 1,000 or 10,000 times, the ratio of heads to tails will be much closer to the theoretical probability of 0.5 for each.

Example of Application:

• Stock Market: Investors use the Law of Large Numbers to analyze stock returns over a long period. Short-term returns can be volatile, but over time, they average out to provide a more stable picture of an asset’s performance.

Gambling: Casinos rely on this law because, while players might win or lose in the short run, over time, the house always comes out ahead due to fixed probabilities.

2. Addition Rule of Probability

The Addition Rule of Probability applies when you want to calculate the probability of the union of two events (i.e., the probability that at least one of the events occurs). There are two versions of this rule depending on whether the events are mutually exclusive (cannot happen at the same time) or not.

Explanation:

• For mutually exclusive events (events that cannot both happen at the same time):
  • P(A∪B)=P(A)+P(B)-P(A ∪ B) = P(A) + P(B) – P(A ∪ B) = P(A) + P(B)
• For non-mutually exclusive events (events that can happen together):
  • P(A∪B)= P(A)+P(B)−P(A ∩ B) 

The extra term P(A∩B) subtracts the overlap, so you don’t double-count when both events occur.

Example of Application:

• Weather Forecasting: If you want to find the probability that it will rain or snow tomorrow, you can use the addition rule. If rain and snow can happen together (non-mutually exclusive), you would subtract the overlap.

Cards: When drawing a card from a deck, the probability of drawing a heart or a king (non-mutually exclusive, since the king of hearts belongs to both groups) requires using the addition rule with overlap subtraction.

3. Multiplication Rule of Probability

The Multiplication Rule is used when you want to determine the probability of the intersection of two events (i.e., both events happening). Like the addition rule, it comes in two forms depending on whether the events are independent or not.

Explanation:

• For independent events (one event does not affect the probability of the other):
  • P(A∩B)=P(A)×P(B)
• For dependent events (the occurrence of one event affects the probability of the other):
  • P(A∩B)=P(A)×P(B∣A)
  • Where P(B∣A) is the conditional probability of B given that A has occurred.

Example of Application:

• Rolling Dice: If you roll two dice, the probability of getting a 4 on the first die and a 6 on the second die is 1/6×1/6=1/36. These are independent events.

4. Complementary Rule

The Complementary Rule states that the probability of an event not happening is 1 minus the probability of the event happening. This rule is useful when it’s easier to calculate the probability of something not happening than the probability of it happening.

Explanation:

P(not A)=1−P(A)

Example of Application:

• Product Defects: If the probability of a manufactured product being defective is 0.02, then the probability of it being non-defective is:
  • P(non-defective)=1−0.02=0.98
• Weather Forecasting: If the chance of rain tomorrow is 30%, the complementary probability of no rain is:
  • P(no rain)=1−0.3=0.7 or 70%

5. Conditional Probability

Conditional Probability measures the probability of one event occurring given that another event has already occurred. It is denoted as P(A∣B), the probability of A happening given that B has already happened.

Explanation:

P(A∣B)=P(A∩B)*P(B)​

This formula helps you adjust the probability of A occurring based on the knowledge that B has already taken place.

Example of Application:

• Medical Diagnosis: In diagnosing a disease, the probability that a patient has the disease given a positive test result is calculated using conditional probability.

Marketing: If a customer has already bought a product, the probability they will purchase related products (cross-selling) can be determined using conditional probability.

6. Discrete Uniform Law

The Discrete Uniform Law describes a situation where each possible outcome of a discrete random variable is equally likely. This rule is often used in cases like rolling a die, where each number has the same chance of appearing.

Explanation:

For a discrete random variable X with n equally likely outcomes, the probability of each outcome is:

P(X=x)=1/n

Example of Application:

• Dice Games: If you roll a standard six-sided die, the probability of any one number appearing is 1/6.
• Lottery Systems: In some simple lotteries, where each ticket has an equal chance of winning, the Discrete Uniform Law applies.

7. Bayes’ Theorem

Bayes’ Theorem is a powerful rule used to calculate conditional probabilities, updating our beliefs based on new evidence. It is especially important in fields like machine learning and statistics.

Explanation:

P(A∣B)=P(B∣A)×P(A)/P(B)

This formula allows you to reverse conditional probabilities, providing insights based on prior knowledge.

Example of Application:

• Spam Filters: Email systems use Bayes’ Theorem to determine whether an email is spam by updating the probability based on words used in the email.
• Medical Testing: It helps calculate the probability of a person having a disease given a positive test result, considering the test’s accuracy and disease prevalence.

Other Important Rules:

Total Probability Rule

The Total Probability Rule allows you to calculate the total probability of an event by considering all possible ways the event can happen, typically by partitioning the sample space into mutually exclusive events.

Example of Application:

Risk Management: In risk assessment, companies often use this rule to determine the overall probability of a system failure by analyzing the different ways in which it could fail.

Conclusion

These probability laws and rules form the backbone of probabilistic reasoning and modelling. From predicting outcomes in simple games to making informed decisions in medicine and business, these rules are essential tools for understanding uncertainty and making data-driven predictions. By mastering these principles, you can apply probability to various real-world scenarios effectively.