4.1. Uncertainty & Importance of Probability Theory#

We have seen that dealing with data means dealing with “some variables assuming some values”.

However, what can we say about what values such variables will assume?

In some cases, it is possible to predict the values of these variables with perfect accuracy, given a set of initial conditions. For instance, think about a system of equations describing the speed of objects according to Newtonian laws.

In other cases, modeling the relationship between variables in a deterministic way is not possible. We often say that this is due to uncertainty.

As reported here, uncertainty in a system can be due to different factors:

• Inherent stochasticity in the system being modeled. Some events such as drawing a card from a deck of cards, rolling a die, or the movements of subatomic particles can be seen as truly random events. The outcomes of such events cannot be predicted with perfect accuracy, so they are stochastic (random).

• Incomplete observability. Sometimes, even in a deterministic system, we cannot observe anything. For instance, in the Monty Hall problem, the participant to the game is asked to choose between three doors. One of the doors contains a prize, while the others lead to a goat. Even if the event is deterministic, the game contestant cannot observe everything, so from their point of view, the outcome is uncertain.

• Incomplete modeling. Sometimes a system has to discard some of the information needed to make a decision. For instance, imagine a robotic system aiming to pick objects from a table with a single RGB camera. The system cannot reconstruct the 3D position of the objects, but the RGB camera an allow to obtain an estimate of the 2D coordinates of each objects with respect to the robot’s point of view. All the information which cannot be observed is uncertain, despite the problem is a deterministic one.

4.2. Random Variables#

We have so far talked about “statistical variables”. When dealing with uncertain events, we need to use the concept of ‘random variables’. Informally (from wikipedia):

A random variable is a variable whose values depend on outcomes of a random phenomenon.

A random variable is characterized by a set of possible values often called sample space, probability space, or alphabet (this last term comes from information theory, where we often deal with sources emitting symbols from an alphabet, in which case the values of $X$ will be the symbols).

The definition of a random variable is very similar to that of a statistical variable. Formally, if $\mathrm{\Omega }$ is the sample space, we will define a random variable as a function:

Where E is a measurable space and often $E=\mathbb{R}$. This definition is similar to the one of statistical variable we have given before.

A random variable is generally denoted by a capital letter, such as $X$.

Similar to statistical variables, random variables can be discrete or continuous, scalar or multi-dimensional.

The following table lists some examples.

4.2.1. Urns and Marble Example#

We will now introduce an example that we will use in the rest of this lecture to reason on some basic properties of probabilities. Let’s assume we have two urns urns: one green and one blue.

Each box contains marbles of two colors: black and white.

We consider the experiment of randomly drawing a marble from one of the two urns. This happens in two stages:

• We first randomly pick one of the two urns;

• Then we randomly pick one of the marbles in the urn;

• After observing the type of the marble, we replace it in the same urn.

The outcome of the experiment can be characterized by two random variables:

• U represents the color of the urn and can take values g (green) and b (blue).

• M represents the color of the marble and can take values b (black) and w (white).

• If we pick a white marble from the blue urn, then the outcome of the experiment can be characterized by the values $M=w$, $U=b$;

4.3. Working Definition of Data#

We will define “data” as follows:

The values assumed by a random variable

4.4. Probability#

Since random variables are related to stochastic phenomena, we cannot say much about the outcome of a single phenomenon.

However, we expect to be able to characterize the class of experiments related to a random variable, to infer rules on what values the random variable is likely to take.

For instance, in the case of coin tossing, we can observe that, if I toss a coin for a large number of times, the number of heads will be roughly similar to the number of tails.

This kind of observations is useful, as it can give us a prior on what values we are likely to encounter and what are not.

To formally express such rules, we can define the concept of probability on a random variable.

Specifically, it is possible to assign a probability value to a given outcome. This is generally represented with a capital P:

• For instance, $P(U=b)$ represents the probability of picking a blue urn in the previous example;

• A probability $P(U=b)$ is a number comprised between 0 and 1 which quantifies how likely we believe the event to be;

  • 0 means impossible;

  • 1 means certain;

When it is clear from the context which variable we are referring to, the probability can also be expressed simply as:

4.4.3. Example of Probability#

Let’s consider the previous example of drawing marbles from the two urns.

Suppose we repeat this experiment for many times and observe that:

• We pick the green urn 40% of the times;

• We pick the blue urn 60% of the times;

• Once we selected a urn, we are equally likely to select any of the marble contained in it, but we know that colors are not distributed evenly in the urns (see figure).

Using a frequentist approach, we can define the probabilities:

• $P(U=b)=\frac{6}{10}$

• $P(U=r)=\frac{4}{10}$

This is done by using the formula:

• $P(X=x)=\frac{\mathrm{\#}oftimesX=x}{\mathrm{\#}trials}$

4.5. Joint probability#

We can define univariate (= with respect to only one variable) probabilities $P(U)$ and $P(M)$ as we have seen in the previous examples.

However, in some cases, it is useful to define probabilities on more than one variable at the time. For instance, we could be interested in studying the probability of picking a given fruit from a given box. In this case, we would be interested in the joint probability $P(B,F)$.

In general, we can have joint probabilities with arbitrary numbers of variable. For instance, $P(X_1,X_2,X_3,\dots ,X_n)$.

Joint probabilities are symmetric, i.e., $P(X,Y)=P(Y,X)$.

We should note that, when dealing with multiple unidimensional variables, we can always define a new multi-variate variable comprising all of them:

• $X=[X_1,X_2];$

• $P(X)=P(X_1,X_2)$.

4.5.1. Example#

We can see the concept of joint probability in the context of the examples of the two urns.

We have seen how to define the univariate probability $P(U)$ over the whole probability space of $U$.

However, we could be interested in the probability of both variables jointly: $P(U,M)$, i.e., the joint probability of U and M.

To ‘measure’ the joint probability, we could repeat the experiment for many times and observe the outcomes.

We can then build a table which keeps track of how many times we observed a given combination:

	Green Urn	Blue Urn	All
White	10	15	25
Black	30	45	75
All	40	60	100

A table like this is called a “contincency table”.

From the table above, we can easily derive the joint probability of a given pair of values using the frequentist approach. For instance:

$$P(U=b,M=b)=\frac{\mathrm{\#}times(b,a)occurs}{\mathrm{\#}trials}=\frac{15}{100}$$

Similarly, we can derive the other values:

• $F(U=b,M=b)=\frac{45}{100}$

• $F(U=g,M=w)=\frac{10}{100}$

• $F(U=g,M=b)=\frac{30}{100}$

Note that we can also obtain the univariate probabilities by using the values in the “sum” row and column. For instance:

$$P(U=b)=\frac{\mathrm{\#}timesboccurred}{\mathrm{\#}trials}=\frac{15+45}{100}=\frac{60}{100}$$

Similarly:

• $P(U=g)=\frac{40}{100}$

• $P(M=w)=\frac{25}{100}$

• $P(M=b)=\frac{30}{100}$

These univariate probabilities computed starting from joint probabilities are usually called “marginal probabilities” (we are using the sums in the margin of the table).

We can obtain a joint probability table by dividing the table by the total number of trials (100):

	Green Urn	Blue Urn	Sum
White	10/100	15/100	25/100
Black	30/100	45/100	75/100
Sum	40/100	60/100	100/100

4.6. Sum Rule (Marginal Probability)#

In the previous example, we have seen how we can compute marginal (univariate) probabilities from the contingency table. This is possible because the contingency table contains information on how the different possible outcomes distribute over the sample space.

In general, we can compute marginal probabilities form joint probabilities (i.e., we don’t need to have the non-normalized frequency counts of the contingency table). Let us consider the general contingency table:

We can compute the joint probability $P(X=x_i,Y=y_j)$ with a frequentist approach using the formula:

Note that these are the joint frequencies $f_{ij}$ we mentioned in the past.

Also, we note that we can define the marginal probabilities of X and Y as follows:

• $P(X=x_i)=\frac{n_{i+}}{n}$.

• $P(Y=y_j)=\frac{n_{+j}}{n}$.

Note that $n_{i+}$ can be seen as the sum of all occurrences in which $X=x_i$ (i.e., we are summing all values in row $i$):

We can write the marginal probability of $X$ as follows:

This result is known as the sum rule of probability, which allows to estimate marginal probabilities from joint probabilities. This can be seen in more general terms as:

The act of computing $P(X)$ from $P(X,Y)$ is also known as marginalization.

4.7. Conditional Probability#

In many cases, we are interested in the probability of some event, given that some other event happened.

This is called conditional probability and is denoted as$P(X=x|Y=y)$ and read as “P of X=y given that Y=y”. In this context, $Y=y$ is the condition, and we are interested in studying the probability of X only in the cases in which the condition is verified.

For instance, in the case of the two urns, we could be interested in $P(M=w|U=b)$, i.e., what is the probability of picking a white marble, given that we know that we are drawing from the blue urn?

Let’s consider our example contingency table of two variables again:

We can compute the conditional probability $P(X=x_i|Y=y_j)$ using the frequentist approach:

If we multiply the expression above by $1=\frac{n}{n}$, we obtain:

This leads us to the general definition of conditional probability:

The conditional probability is defined only when $P(Y=y)>0$, that is, we cannot define a probability conditioned on an event that never happens. It should be noted that, in general $P\left(X\right|Y)\ne P(X)$.

4.8. Product Rule (Factorization)#

We can see the definition of conditional probability:

As follows:

which is often referred to as the product rule.

The product rule allows to compute joint probabilities starting from conditional probabilities and marginal probabilities. This is useful because measuring joint probabilities generally involves creating large tables, whereas conditional and marginal probabilities might be easier to derive.

This operation of expressing a joint probability in terms of two factors is known as factorization.

4.9. The Chain Rule of Conditional Probabilities#

When dealing with multiple variables, the product rule can be applied in an iterative fashion, thus obtaining the ‘chain rule’ of conditional probabilities.

For instance:

Since:

We obtain:

Since joint probabilities are symmetric, we could equally obtain:

This rule can be formalized as follows:

4.10. Bayes’ Theorem#

Given two variables A and B, from the product rule, we obtain:

• $P(A,B)=P\left(A\right|B)P(B)$

• $P(B,A)=P\left(B\right|A)P(A)$

Since joint probabilities are symmetric, we have:

Which implies:

This last expression is known as Bayes’ Theorem (or Bayes’ rule).

Technically speaking, the Bayes’ rule can be used to “turn” probabilities of the kind $P(A|B)$ into probabilities of the kind $P(B|A)$.

More formally, the Bayes’ rule can be used to update our expectation that some event will happen (event A) when we observe some evidence (event B). This links to the Bayesian interpretation of probability, according to which probability can be seen as a reasonable expectation representing the quantification of a degree of belief (i.e., how much we believe some event will happen).

The different terms in the Bayes’ rule have specific names:

• $P(A)$ is called ‘the prior’ – this is our expectation that A happens when we do not have any other data to rely on

• $P(B|A)$ is called ‘the likelihood’ – this quantifies how likely it is to observe event B happening if we assume that event A has happened

• $P(B)$ is called ‘the evidence’ – this models the probability of observing event B

• $P(A|B)$ is called ‘the posterior’ – this is our updated probability, i.e., how likely it is A to happen once we have observed B happening

We’ll see an example in the section below.

4.11. Independence and Conditional Independence#

Two variables $X$ and $Y$ are independent if the outcome of one of the two does not influence the outcome of the other. Formally speaking, two variables are said to be independent if and only if:

We have seen why definition this makes sense in the context of the Pearson’s ${\chi }^2$ statistic. The same considerations apply here.

It should be noted that the expression above is generally not true as we cannot always assume that two variables are independent.

Independence can be denoted as:

Moreover, if two variables $X$ and $Y$ are independent, then:

This makes sense because it means that “the fact that $Y$ happens does not influence the fact that $X$ happens”, which is what we would expect of two independent variables $X$ and $Y$.

4.12. Example of Probability Manipulation#

Let’s get back to our example of the two boxes and let’s suppose that, by repeating several trials, we discovered the following probabilities (in a frequentist way):

• $P(U=g)=\frac{4}{10}$

• $P(U=b)=\frac{6}{10}$

We also focused on a given urn and performed different trials, observing the following proportions:

• $P(M=w|U=g)=\frac{1}{4}$

• $P(M=b|U=g)=\frac{3}{4}$

• $P(M=w|U=b)=\frac{3}{4}$

• $P(M=b|U=b)=\frac{1}{4}$

We shall note that these probabilities are normalized such that:

• $P(U=g)+P(U=b)=1$

• $P(M=w|U=g)+P(M=b|U=g)=1$

• $P(M=w|U=b)+P(M=b|U=b)=1$

We can now use the rules we have seen before to answer questions such as:

• What is the overall probability of choosing a white marble?

• What is the probability of picking a green urn, given that we have drawn a black marble?

What is the overall probability of choosing an white marble?

To answer this question, we need to find $P(M=w)$. We note that, by the sum rule:

We also observe that the joint probabilities can be recovered using the product rule:

Hence, our probability can be found using the formula:

From the definition of probability, we have:

Hence:

What is the probability of picking a green urn, given that we have drawn an black marble?

To answer this question, we need to find the conditional probability $P(U=g|M=b)$. To find this probability, we need to ‘invert’ our conditional probabilities using the Bayes’ rule:

4.13. References#

• Parts of chapter 1 of [1];

• Most of chapter 3 of [2];

• Parts of chapters 5-7 of [3].

[1] Bishop, Christopher M. Pattern recognition and machine learning. springer, 2006. https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf

[2] Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016. https://www.deeplearningbook.org/

[3] Heumann, Christian, and Michael Schomaker Shalabh. Introduction to statistics and data analysis. Springer International Publishing Switzerland, 2016.