Probability Theory and Statistics — lecture notes

Summary #

Quick-reference map of probability spaces, named laws, asymptotics, and statistics in these notes. Full derivations and worked examples stay in the linked chapters below.

Part I: Foundations #

Chapters 1–4.

I.0 Skill map #

I.1 Probability space #

When: modeling any random experiment.

Recipe

1. Choose outcomes $\Omega$.
2. Choose events $\mathcal{F}$ (usually $\mathcal{P}(\Omega)$ if finite/countable).
3. Assign $\mathbb{P}$ via axioms (or via masses $p_\omega$, or Laplace).

Key formula: $\boxed{\mathbb{P}[\Omega]=1,\quad \mathbb{P}\ge 0,\quad \text{countable additivity on disjoint unions}}$ → §1.3

I.2 Laplace and counting #

When: finite $\Omega$ with equal outcomes.

Recipe

1. Fix what an outcome is (ordered vs unordered).
2. Count $|\Omega|$ and $|A|$.
3. $\mathbb{P}[A]=|A|/|\Omega|$. Prefer complements for “at least one…”.

Key formula: $\boxed{\mathbb{P}[A]=\frac{|A|}{|\Omega|}}$ → §2.1

I.3 Conditional probability and Bayes #

When: information “given $B$”, medical tests, partitions.

Recipe

1. $\mathbb{P}[A\mid B]=\mathbb{P}[A\cap B]/\mathbb{P}[B]$ ($\mathbb{P}[B]>0$).
2. Total probability: $\mathbb{P}[A]=\sum_i\mathbb{P}[A\mid B_i]\mathbb{P}[B_i]$ for a partition $(B_i)$.
3. Bayes: flip to $\mathbb{P}[B_j\mid A]$.

Key formula: $\boxed{\mathbb{P}[B_j\mid A]=\frac{\mathbb{P}[A\mid B_j]\mathbb{P}[B_j]}{\sum_i\mathbb{P}[A\mid B_i]\mathbb{P}[B_i]}}$ → §3.3

I.4 Independence #

When: “no influence” between events or experiments.

Recipe

1. Events: $\mathbb{P}[A\cap B]=\mathbb{P}[A]\mathbb{P}[B]$ (pairwise is not enough for $n\ge 3$).
2. Equivalent: $\mathbb{P}[A\mid B]=\mathbb{P}[A]$ when $\mathbb{P}[B]>0$.
3. Construct via product experiments (independent coordinates / dice).

Key formula: $\boxed{\mathbb{P}[A\cap B]=\mathbb{P}[A]\,\mathbb{P}[B]}$ → §4.1

Part II: Discrete random variables #

Chapters 5–7.

II.0 Skill map #

II.1 Named discrete laws #

II.2 Expectation and variance recipes #

When: means, spreads, sums of indicators / Bernoullis.

Recipe

1. Direct: $\mathbb{E}[X]=\sum_x x\,\mathbb{P}[X=x]$; LOTUS for $g(X)$.
2. Nonnegative integer: tail sum $\mathbb{E}[T]=\sum_{n\ge 1}\mathbb{P}[T\ge n]$.
3. Linearity: $\mathbb{E}[\sum X_i]=\sum\mathbb{E}[X_i]$ (no independence needed).
4. Variance: compute $\mathbb{E}[X^2]$ or use Bernoulli-sum for Binomial.

Key formulas: $\boxed{\mathbb{E}[aX+bY]=a\mathbb{E}[X]+b\mathbb{E}[Y]}$ $\boxed{\mathrm{Var}(S)=np(1-p)\ \text{for }S\sim\mathrm{Bin}(n,p)}$ → §7.3–§7.6

Part III: Continuous laws and asymptotics #

Chapters 8–9.

III.0 Skill map #

III.1 Named continuous laws #

III.2 LLN and CLT #

When: long-run averages and Normal approximations for large $n$.

Recipe

1. LLN: i.i.d. with finite mean $m$ $\Rightarrow$ $\bar X_n\to m$.
2. CLT: also finite variance $\sigma^2$ $\Rightarrow$ $\sqrt{n}\,(\bar X_n-m)/\sigma$ has CDF $\to\Phi$.
3. For Binomial / Bernoulli frequencies, standardize with $\sigma=\sqrt{p(1-p)}$.

Key formula: $\boxed{\mathbb{P}\Bigl[\frac{\sqrt{n}\,(\bar X_n-m)}{\sigma}\le a\Bigr]\xrightarrow{n\to\infty}\Phi(a)}$ → §9.5

Part IV: Statistics #

Chapter 10.

IV.0 Skill map #

IV.1 Which statistical tool? #

Pipeline: data $\to$ MLE $\widehat{\theta}$ $\to$ CI for uncertainty $\to$ test to decide $H_0$ vs $H_1$.

Chapter 1 — Probability spaces & events (Wahrscheinlichkeitsräume) #

Build the triplet $(\Omega,\mathcal{F},\mathbb{P})$: sample space, events ($\sigma$-algebra), and a probability measure with its basic calculus.

1.1 Sample spaces #

1.1.1 Overview #

Definition (sample space / Grundraum) The set $\Omega$ of all possible outcomes of a random experiment is the sample space. An element $\omega\in\Omega$ is an outcome (Elementarereignis).

When to use

• First step of every model: name what an outcome is (a face, a tuple, a point, a sequence).
• Product spaces when several experiments are combined: dice, coins, bits.

Typical mistake

• Writing $\Omega=\{A,B\}$ for events instead of outcomes. Events are subsets of $\Omega$, not elements of $\Omega$ (unless you deliberately identify them).

1.2 Events and $\sigma$-algebras #

1.2.1 Events as subsets #

Definition (event / Ereignis) An event is a set $A\subset\Omega$. We write $\mathcal{F}$ for the collection of events we allow. The event $A$ occurs for $\omega$ if $\omega\in A$.

When to use

• Translate German/English verbal statements into subsets before computing $\mathbb{P}$.
• Use Venn pictures to track overlaps before applying formulas.

Typical mistake

• Confusing “or” (union) with “xor” (symmetric difference). In probability, “or” usually means inclusive union.

1.2.2 $\sigma$-algebras #

Definition ($\sigma$-algebra / $\sigma$-Algebra) A family $\mathcal{F}\subset\mathcal{P}(\Omega)$ is a $\sigma$-algebra if

1. $\Omega\in\mathcal{F}$,
2. $A\in\mathcal{F}\Rightarrow A^c\in\mathcal{F}$,
3. $A_1,A_2,\ldots\in\mathcal{F}\Rightarrow \bigcup_{i=1}^\infty A_i\in\mathcal{F}$.

Consequences (still in $\mathcal{F}$)

• $\emptyset=\Omega^c\in\mathcal{F}$,
• countable intersections (De Morgan),
• finite unions and intersections.

When to use

• Finite / countable $\Omega$: take $\mathcal{F}=\mathcal{P}(\Omega)$ without hesitation.
• Continuous models (area, length): think “Borel sets” — rectangles, open sets, and countable unions/intersections of those. You rarely construct Borel sets by hand in this course; you use measurable regions (intervals, polygons, discs).

Typical mistake

• Thinking H3 only allows finite unions. The countable union is essential for continuity of $\mathbb{P}$ and for series of events.

Brief pointer (indicators). For $A\in\mathcal{F}$, the indicator $1_A:\Omega\to\mathbb{R}$ with $1_A(\omega)=1$ if $\omega\in A$ and $0$ otherwise is a (discrete) random variable (RV). Full RV theory comes later; here, $1_A$ is just a convenient encoding of “did $A$ occur?”.

1.3 Probability measures #

1.3.1 Frequency motivation and axioms #

Definition (probability measure / Wahrscheinlichkeitsmaß) A map $\mathbb{P}:\mathcal{F}\to[0,1]$ is a probability measure if

• P1: $\mathbb{P}[\Omega]=1$,
• P2 (countable additivity): if $A_1,A_2,\ldots$ are pairwise disjoint and $A=\bigcup_{i=1}^\infty A_i$, then

Definition (probability space / Wahrscheinlichkeitsraum) The triplet $(\Omega,\mathcal{F},\mathbb{P})$ is a probability space.

When to use

• Building any model: specify $\Omega$, choose $\mathcal{F}$, define $\mathbb{P}$ on events (not only on singletons when $\Omega$ is uncountable).
• Checking a candidate formula is a valid probability (normalize, nonnegativity, additivity).

Typical mistake

• Defining $\mathbb{P}[A]=\sum_{\omega\in A}p_\omega$ on an uncountable $\Omega$ where every singleton has probability $0$ (droplet). That formula works for countable $\Omega$ only.

1.4 Calculus of probability #

1.4.1 Immediate consequences #

Proposition (basic calculus) Let $\mathbb{P}$ be a probability measure on $(\Omega,\mathcal{F})$.

1. $\mathbb{P}[\emptyset]=0$.
2. Finite additivity: for pairwise disjoint $A_1,\ldots,A_k$,

3. $\mathbb{P}[A^c]=1-\mathbb{P}[A]$.
4. For any events $A,B$,

When to use

• Complement trick when $A^c$ is easier than $A$.
• Inclusion–exclusion for two (or more) overlapping events.
• Split $\Omega$ into disjoint pieces and add.

Typical mistake

• Writing $\mathbb{P}[A\cup B]=\mathbb{P}[A]+\mathbb{P}[B]$ when $A\cap B\neq\emptyset$ (double-counting the intersection).

1.4.2 Monotonicity and the union bound #

Proposition (monotonicity) If $A\subset B$, then $\mathbb{P}[A]\le\mathbb{P}[B]$.

Proposition (union bound / Boole) For any sequence of events $A_1,A_2,\ldots$,

(The same holds for finite unions.)

When to use

• Monotonicity: sandwich an ugly event between simpler events.
• Union bound: many “bad” events; show the probability that at least one occurs is small.

Typical mistake

• Treating the union bound as equality. Equality needs disjointness (or almost-disjointness).

1.5 Continuity of probability #

1.5.1 Increasing and decreasing sequences of events #

Proposition (continuity)

• If $A_n\subset A_{n+1}$ for all $n$ (increasing), then

• If $B_n\supset B_{n+1}$ for all $n$ (decreasing), then

When to use

• Passing limits inside $\mathbb{P}$ for nested events.
• Later: proving properties of distribution functions $F_X(a)=\mathbb{P}[X\le a]$ (right-continuity, limits at $\pm\infty$).

Typical mistake

• Applying continuity to a non-monotone sequence without rewriting it as increasing/decreasing pieces.

1.6 Method map (quick reference) #

1.7 Exercises #

Exercise 1. For a fair die, let $A=\{1,2\}$, $B=\{2,3,4\}$. Compute $\mathbb{P}[A]$, $\mathbb{P}[B]$, $\mathbb{P}[A\cap B]$, $\mathbb{P}[A\cup B]$, $\mathbb{P}[A^c]$.

Solution. Goal: apply Laplace and the basic calculus rules on one die. Step 1: $\Omega=\{1,\ldots,6\}$, $|\Omega|=6$; $\mathbb{P}[A]=2/6=1/3$, $\mathbb{P}[B]=3/6=1/2$. Step 2: $A\cap B=\{2\}$, so $\mathbb{P}[A\cap B]=1/6$; $\mathbb{P}[A\cup B]=\mathbb{P}[A]+\mathbb{P}[B]-\mathbb{P}[A\cap B]=1/3+1/2-1/6=2/3$. Step 3: $\mathbb{P}[A^c]=1-\mathbb{P}[A]=2/3$. Check: $|A\cup B|=|\{1,2,3,4\}|=4$, so $4/6=2/3$.

Exercise 2. Fair die rolled twice (ordered pairs). Let $A=$ “sum is 7”, $B=$ “first die is 1”. Compute $\mathbb{P}[A]$, $\mathbb{P}[B]$, $\mathbb{P}[A\cap B]$, $\mathbb{P}[A\cup B]$.

Solution. Goal: ordered pair of fair dice; events $A=\{\text{sum }=7\}$, $B=\{\text{first }=1\}$. Step 1: Laplace on $\Omega=\{1,\ldots,6\}^2$, $|\Omega|=36$. Step 2: $A=\{(1,6),(2,5),(3,4),(4,3),(5,2),(6,1)\}$, so $|A|=6$, $\mathbb{P}[A]=6/36=1/6$. Step 3: $B=\{1\}\times\{1,\ldots,6\}$, $|B|=6$, $\mathbb{P}[B]=1/6$. Step 4: $A\cap B=\{(1,6)\}$, $\mathbb{P}[A\cap B]=1/36$. Step 5: $\mathbb{P}[A\cup B]=\mathbb{P}[A]+\mathbb{P}[B]-\mathbb{P}[A\cap B]=11/36$. Check: listing $A\cup B$ gives $6+6-1=11$ outcomes.

Chapter 2 — Laplace models & counting (Laplace-Modelle) #

Finite sample spaces where every outcome is equally likely: probability reduces to counting.

2.1 The Laplace model #

2.1.1 Definition #

Definition (Laplace model) Let $\Omega$ be finite. The Laplace model is $(\Omega,\mathcal{F},\mathbb{P})$ with $\mathcal{F}=\mathcal{P}(\Omega)$ and

Equivalently, $\mathbb{P}[\{\omega\}]=1/|\Omega|$ for every $\omega$.

When to use

• Fair coins, fair dice, well-shuffled cards, random sample of $k$ distinct items from $n$ when each $k$-subset (or each $k$-tuple) is equally likely.
• Any finite experiment where the problem states “uniformly at random” / “fair” / “zufällig mit gleicher Wahrscheinlichkeit”.

Typical mistake

• Using Laplace when outcomes are not equally likely (biased coin, waiting times with geometric masses).

2.2 Counting toolkit #

2.2.1 Product rule, permutations, combinations #

Formulas

Useful identities: $\binom{n}{0}=1$, $\binom{n}{k}=0$ if $k>n$ or $k<0$, and Pascal: $\binom{n}{k}=\binom{n-1}{k-1}+\binom{n-1}{k}$. Convention: $0!=1$.

When to use

• First decide: is an outcome an ordered tuple or an unordered set?
• Stick to one convention for both numerator and denominator.

Typical mistake

• Mixing conventions: counting $|A|$ with combinations but $|\Omega|$ with permutations (or vice versa), which systematically biases the answer.

2.3 Worked counting patterns #

2.3.1 Dice and cards #

2.4 Modelling pitfalls #

2.4.1 Ordered vs unordered, distinguishable vs not #

When to use

• Ask: “If I list two descriptions of the same physical draw, do they appear equally often in my $\Omega$?”
• Prefer ordered tuples with replacement / sequences of dice when independence is clear; use combinations when the experiment literally samples a set.

Typical mistake

• Bertrand-type paradoxes / boy-girl puzzles: ambiguous sample spaces dressed up as Laplace problems.

2.5 Method map (quick reference) #

2.6 Exercises #

Exercise 1. Fair coin three times (ordered). Probability of exactly two heads.

Solution. Goal: Laplace count on $\{H,T\}^3$. Step 1: $|\Omega|=8$; favorable outcomes $HHT,HTH,THH$, so $|A|=3$. Step 2: $\mathbb{P}[A]=\binom{3}{2}/8=3/8$. Check: binomial $\mathrm{Bin}(3,1/2)$ gives the same mass at $2$.

Exercise 2. From $\{1,\ldots,10\}$, pick a $3$-subset uniformly. Probability it contains $1$.

Solution. Goal: uniform $3$-subset of $\{1,\ldots,10\}$; probability it contains $1$. Step 1: $|\Omega|=\binom{10}{3}$. Step 2: if $1\in A$, choose $2$ more from $\{2,\ldots,10\}$: $|A|=\binom{9}{2}$. Step 3: $\mathbb{P}=\binom{9}{2}/\binom{10}{3}=(36)/(120)=3/10$. Check: alternatively, probability $1$ is among three draws without replacement in unordered sampling equals $3/10$ by symmetry among the ten labels.

Exercise 3. $n$ people seated randomly at a round table (rotations identified). Probability that persons $A$ and $B$ sit next to each other. (Hint: fix $A$’s seat.)

Solution. Goal: circular seating with rotations identified; neighbors of $A$ and $B$. Step 1: fix $A$’s seat (removes rotational symmetry). Step 2: $B$ has $n-1$ equally likely seats; exactly $2$ are adjacent to $A$. Step 3: $\mathbb{P}=2/(n-1)$. Check: for $n=3$, probability $1$; for $n=4$, probability $2/3$.

Chapter 3 — Conditional probability & Bayes (Bedingte Wahrscheinlichkeit) #

Update probabilities after learning that an event $B$ occurred; reverse conditioning with Bayes; split via the law of total probability.

3.1 Conditional probability #

3.1.1 Definition and interpretation #

Definition (conditional probability / bedingte Wahrscheinlichkeit) If $\mathbb{P}[B]>0$,

Immediate facts

• $\mathbb{P}[B\mid B]=1$.
• $\mathbb{P}[A\cap B]=\mathbb{P}[A\mid B]\,\mathbb{P}[B]$ (multiplication rule), and similarly $\mathbb{P}[A\cap B]=\mathbb{P}[B\mid A]\,\mathbb{P}[A]$ when $\mathbb{P}[A]>0$.
• The map $A\mapsto\mathbb{P}[A\mid B]$ is itself a probability measure on $(\Omega,\mathcal{F})$.

When to use

• The problem gives information (“given even”, “given positive test”, “given first die shows $1$”).
• You need a joint probability and know one conditional factor.

Typical mistake

• Writing $\mathbb{P}[A\mid B]=\mathbb{P}[A]/\mathbb{P}[B]$ (forgetting the intersection).
• Conditioning on a null event ($\mathbb{P}[B]=0$).

3.2 Law of total probability #

3.2.1 Partitions #

Definition (partition) Events $B_1,\ldots,B_n$ form a partition of $\Omega$ if they are pairwise disjoint and $\bigcup_{i=1}^n B_i=\Omega$.

Proposition (total probability / Satz von der totalen Wahrscheinlichkeit) If $B_1,\ldots,B_n$ partition $\Omega$ with $\mathbb{P}[B_i]>0$, then for every event $A$,

When to use

• Natural cases: disease / healthy; bit sent $0$ / $1$; machine type $1,\ldots,k$.
• Discrete RV taking values $x_1,\ldots,x_n$: condition on $\{X=x_i\}$.

Typical mistake

• Using overlapping $B_i$ (not a partition) and still summing as if disjoint.
• Forgetting a piece of the partition (probabilities of scenarios must sum to $1$).

3.3 Bayes formula #

3.3.1 From prior to posterior #

Proposition (Bayes) If $B_1,\ldots,B_n$ partition $\Omega$ with $\mathbb{P}[B_i]>0$, and $\mathbb{P}[A]>0$, then

When to use

• Wording: “given that the test is positive…”, “given that we received $1$…”, “posterior probability…”.
• You know $\mathbb{P}[A\mid B_i]$ (forward model) but need $\mathbb{P}[B_i\mid A]$.

Typical mistake

• Equating $\mathbb{P}[B\mid A]$ with $\mathbb{P}[A\mid B]$ (transposing the conditional).
• Ignoring base rates (rare disease): a high $\mathbb{P}[\text{positive}\mid\text{sick}]$ does not imply a high $\mathbb{P}[\text{sick}\mid\text{positive}]$.

3.4 Trees and classic applications #

3.4.1 Medical test #

When to use

• Any two-stage experiment: type then measurement; cause then symptom; class then feature.

Typical mistake

• Reporting the accuracy $0.99$ as the answer to “probability sick given positive”.

3.5 Method map (quick reference) #

3.6 Exercises #

Exercise 1. Fair die; $B=\{1,2,3,4\}$. Find $\mathbb{P}[\text{even}\mid B]$.

Solution. Goal: restrict to $B$ and renormalize. Step 1: even faces in $B$ are $\{2,4\}$, so $|A\cap B|=2$ with $A=\{\text{even}\}$. Step 2: $\mathbb{P}[A\mid B]=|A\cap B|/|B|=2/4=1/2$. Check: equivalently $\mathbb{P}[A\cap B]/\mathbb{P}[B]=(1/3)/(2/3)=1/2$.

Exercise 2. Urns: urn $1$ has $2$ red and $3$ blue; urn $2$ has $3$ red and $1$ blue. Pick urn uniformly, then one ball. Find $\mathbb{P}[\text{red}]$ and $\mathbb{P}[\text{urn }1\mid\text{red}]$.

Solution. Goal: random urn, then random ball; red probability and posterior of urn $1$. Step 1: partition $\{U_1,U_2\}$ with $\mathbb{P}[U_i]=1/2$. Step 2: $\mathbb{P}[R\mid U_1]=2/5$, $\mathbb{P}[R\mid U_2]=3/4$. Step 3 (total probability): $\mathbb{P}[R]=\tfrac12\cdot\tfrac25+\tfrac12\cdot\tfrac34=\tfrac{4}{20}+\tfrac{3}{8}=\tfrac{8}{40}+\tfrac{15}{40}=\tfrac{23}{40}$. Step 4 (Bayes): $\mathbb{P}[U_1\mid R]=\dfrac{\tfrac12\cdot\tfrac25}{\tfrac{23}{40}}=\dfrac{8}{23}$. Check: urn $2$ is redder, so posterior of urn $1$ given red is less than $1/2$.

Exercise 3. Fair die twice. Let $S$ be the sum. Compute $\mathbb{P}[\text{first die }=6\mid S=8]$.

Solution. Goal: conditional probability by listing equally likely outcomes on $\{S=8\}$. Step 1: outcomes with sum $8$: $(2,6),(3,5),(4,4),(5,3),(6,2)$ — five equally likely. Step 2: first die $=6$ only in $(6,2)$, so $\mathbb{P}=1/5$. Check: $\mathbb{P}[\text{first }=6,\,S=8]=\mathbb{P}[(6,2)]=1/36$ and $\mathbb{P}[S=8]=5/36$, ratio $1/5$.

Chapter 4 — Independence (Unabhängigkeit) #

Events (and later random variables) that do not inform each other: product formulas and the difference between pairwise and mutual independence.

4.1 Independence of two events #

4.1.1 Definition and equivalents #

Definition (independence of two events) Events $A,B$ are independent if

Proposition (equivalence with conditioning) If $\mathbb{P}[A]>0$ and $\mathbb{P}[B]>0$, then the following are equivalent:

1. $\mathbb{P}[A\cap B]=\mathbb{P}[A]\mathbb{P}[B]$,
2. $\mathbb{P}[A\mid B]=\mathbb{P}[A]$,
3. $\mathbb{P}[B\mid A]=\mathbb{P}[B]$.

Useful remarks

• If $\mathbb{P}[A]\in\{0,1\}$, then $A$ is independent of every event.
• If $A$ is independent of itself, then $\mathbb{P}[A]\in\{0,1\}$.
• $A$ independent of $B$ $\Leftrightarrow$ $A$ independent of $B^c$.

When to use

• Separate physical mechanisms (die 1 vs die 2; $x$- and $y$-coordinates of a uniform point in a square).
• Factoring probabilities in product spaces.

Typical mistake

• Confusing disjoint ($A\cap B=\emptyset$) with independent. Nontrivial disjoint events with positive probability are dependent (if $B$ occurred, $A$ cannot). Exception: if $\mathbb{P}[B]=0$ (or $\mathbb{P}[A]=0$), then $\mathbb{P}[A\cap B]=0=\mathbb{P}[A]\mathbb{P}[B]$, so they are independent.

4.2 Mutual independence of $n$ events #

4.2.1 All subcollections #

Definition (independence of $n$ events) Events $A_1,\ldots,A_n$ are independent (mutually independent) if for every $J\subset\{1,\ldots,n\}$ with $J\neq\emptyset$,

In particular, for three events $A,B,C$ you need all four identities

When to use

• Exam questions that explicitly say “unabhängig” for three events — verify the triple product, not only pairs.
• Families of events generated by different coordinates of an independent experiment.

Typical mistake

• Checking only $\mathbb{P}[A\cap B\cap C]=\mathbb{P}[A]\mathbb{P}[B]\mathbb{P}[C]$ and skipping pairwise (or the converse).

4.3 Preview: independent random variables #

4.3.1 CDF product formula (pointer only) #

Definition (independent RVs — preview) $X_1,\ldots,X_n$ are independent if for all $a_1,\ldots,a_n\in\mathbb{R}$,

An infinite sequence is independent if every finite initial segment is.

iid / i.i.d.: independent and identically distributed (same distribution function $F_{X_i}$).

When to use

• Modelling $n$ separate dice as coordinates on $\{1,\ldots,6\}^n$ with Laplace measure.
• Writing “let $X,Y$ be independent with distribution functions $F,G$” without constructing $\Omega$ explicitly (existence theorem in Acciaio).

Typical mistake

• Thinking $\mathrm{Cov}(X,Y)=0$ (later) is the same as independence — for general RVs it is weaker.

4.4 Building independent experiments #

4.4.1 Product thinking #

When to use

• “Independently” in a problem statement $\mapsto$ product construction.
• Monte Carlo / repeated trials: iid sequence.

Typical mistake

• Encoding two “independent” coins on a $3$-element $\Omega$ with uneven atoms so that the algebra cannot realize the product measure.

4.5 Method map (quick reference) #

4.6 Exercises #

Exercise 1. Fair die; $A=\{\le 2\}$, $B=\{\text{even}\}$. Independent?

Solution. Goal: fair die; $A=\{1,2\}$, $B=\{2,4,6\}$. Step 1: $\mathbb{P}[A]=2/6=1/3$, $\mathbb{P}[B]=1/2$. Step 2: $A\cap B=\{2\}$, $\mathbb{P}[A\cap B]=1/6$. Step 3: $\mathbb{P}[A]\mathbb{P}[B]=1/6$. Check: equal $\Rightarrow$ independent. (Intuition: among even faces, one of three is in $A$, and $\mathbb{P}[A]=1/3$, matching $\mathbb{P}[A\mid B]$.)

Exercise 2. True or false: if $A$ is independent of $B$ and $A$ is independent of $C$, then $A$ is independent of $B\cup C$. Prove or give a counterexample.

Solution. Goal: show pairwise independence with $B$ and $C$ need not imply independence of $B\cup C$. Step 1: take $\Omega=\{1,2,3,4\}$ uniform; $A=\{1,2\}$, $B=\{1,3\}$, $C=\{1,4\}$. Step 2: $\mathbb{P}[A\cap B]=1/4=\mathbb{P}[A]\mathbb{P}[B]$ and $\mathbb{P}[A\cap C]=1/4=\mathbb{P}[A]\mathbb{P}[C]$, so $A\perp B$ and $A\perp C$. Step 3: $B\cup C=\{1,3,4\}$ and $A\cap(B\cup C)=\{1\}$, so $\mathbb{P}[A\cap(B\cup C)]=1/4\neq(1/2)(3/4)=3/8$. Check: the claim is false; independence does not automatically pass to unions.

Chapter 5 — Random variables & CDFs (Zufallsvariablen) #

Numerical summaries of a probability space: maps $X\colon\Omega\to\mathbb{R}$, the CDF $F_X$, and the discrete / continuous split.

5.1 Random variables #

5.1.1 Overview #

Definition. Let $(\Omega,\mathcal{F},\mathbb{P})$ be a probability space. A random variable (r.v.) is a map $X\colon\Omega\to\mathbb{R}$ such that for every $a\in\mathbb{R}$,

Notation (omit $\omega$). Write

and

When to use

• You care about a numerical feature of a complicated experiment (profit, count, coordinate, waiting time).
• You want to push the model forward from $\Omega$ to $\mathbb{R}$ and work with one-dimensional events.

Typical mistake Treating $X$ as an event. Events are subsets of $\Omega$; random variables are functions $\Omega\to\mathbb{R}$. The event is $\{X\le a\}$, not “$X$”.

5.1.2 Further examples on geometric sample spaces #

Example: first coordinate. On $\Omega=[0,1]^2$ with area probability, $Y(\omega_1,\omega_2)=\omega_1$ is a random variable: for $0\le a\le 1$,

Example: product area. $Z(\omega_1,\omega_2)=\omega_1\omega_2$ is a random variable. For $0\le a\le 1$,

and $\mathbb{P}[Z\le a]=\mathrm{Area}(E_a)=a+a\log(1/a)$ (convention $0\log(1/0)=0$).

When to use

• Continuous sample spaces where “list the outcomes” is impossible.
• Building intuition that “distribution” lives on $\mathbb{R}$, not on the original $\Omega$.

Typical mistake Confusing the map $X$ with its law: two different models can induce the same $F_X$. Later we often say “let $X$ have CDF $F$” without fixing $\Omega$.

5.2 Distribution functions (CDFs) #

5.2.1 Definition and basic identity #

Definition. The distribution function (CDF) of $X$ is

Rule (basic identity). For $a<b$,

Proof sketch. $\{X\le b\}=\{X\le a\}\cup\{a<X\le b\}$ (disjoint), so $\mathbb{P}[X\le b]=\mathbb{P}[X\le a]+\mathbb{P}[a<X\le b]$.

When to use

• Any probability about $X$ that can be written with half-lines or finite intervals.
• Comparing two laws by comparing CDFs.

Typical mistake Writing $\mathbb{P}[a\le X\le b]=F(b)-F(a)$ without care. The basic identity uses the left-open interval $(a,b]$. At continuity points of $F$ the distinction vanishes; at atoms it matters (§5.2.3).

5.2.2 Properties that characterize CDFs #

Theorem (properties of CDFs). Let $F=F_X$. Then:

1. $F$ is nondecreasing.
2. $F$ is right-continuous: $F(a)=\lim_{h\downarrow 0}F(a+h)$.
3. $\lim_{a\to-\infty}F(a)=0$ and $\lim_{a\to+\infty}F(a)=1$.

Existence. If $F\colon\mathbb{R}\to[0,1]$ satisfies (1)–(3), there exist $(\Omega,\mathcal{F},\mathbb{P})$ and a r.v. $X$ with $F_X=F$. In practice one says: “let $X$ be a r.v. with distribution function $F$.”

When to use

• Sanity-check a candidate CDF.
• Define a law by writing down $F$ (no need to construct $\Omega$ by hand).

Typical mistake Expecting left-continuity. CDFs are right-continuous by convention (matching $\{X\le a\}$). Left limits encode atoms.

5.2.3 Jumps and point masses #

Definition. Write $F(a-)=\lim_{h\downarrow 0}F(a-h)$ for the left limit.

Proposition. For every $a\in\mathbb{R}$,

When to use

• Reading atoms off a plotted CDF (stairs).
• Distinguishing discrete laws (jumps) from continuous laws (no jumps).

Typical mistake Reading the height $F(a)$ as $\mathbb{P}[X=a]$. Height is cumulative; the jump is the atom.

5.3 Discrete vs continuous #

5.3.1 Two working classes #

Definition (discrete). $X$ is discrete if its image

is at most countable. Equivalently: there is a finite or countable $E\subset\mathbb{R}$ with $X(\omega)\in E$ for all $\omega$. Then the law is described by the pmf (probability mass function)

Definition (continuous). $X$ is continuous if there exists a nonnegative density $f$ (pdf — probability density function) with

Then $F_X$ is continuous, so $\mathbb{P}[X=x]=0$ for every fixed $x$. Informally, $f(x)\,dx\approx\mathbb{P}[X\in[x,x+dx]]$.

When to use

• Choose discrete tools (sums, pmfs) vs continuous tools (integrals, densities).
• Decide whether $\mathbb{P}[X=x]$ can be positive.

Typical mistake Calling every “non-discrete” r.v. continuous. There exist mixed laws (jumps plus an absolutely continuous part); this course focuses on the pure discrete and pure continuous classes.

5.4 Indicator random variables #

5.4.1 Definition and link to events #

Definition. For $A\in\mathcal{F}$,

Fact. $1_A$ is a random variable, and $1_A\sim\mathrm{Ber}(\mathbb{P}[A])$ in the sense of Ch 6.

When to use

• Counting how many of several events occur: $N=1_{A_1}+\cdots+1_{A_n}$.
• Translating probability statements into expectation statements (Ch 7).

Typical mistake Writing $1_A+1_B=1_{A\cup B}$ always. That fails when $A\cap B\neq\varnothing$; the correct identity is $1_{A\cup B}=1_A+1_B-1_{A\cap B}$.

5.5 Reading probabilities from the CDF #

5.5.1 Interval triage #

Formulas (discrete-friendly)

When to use

• Exam questions that give $F$ piecewise and ask for $\mathbb{P}[X\in\cdot]$.
• Checking consistency of a pmf against a staircase CDF.

Typical mistake Using $1-F(a)$ for $\mathbb{P}[X\ge a]$ when $a$ is an atom. Prefer $1-F(a-)$.

5.6 Method map (quick reference) #

5.7 Exercises #

Exercise 1. On a fair die, let $X(\omega)=\omega^2$. Compute $F_X(10)$ and $\mathbb{P}[X=16]$.

Solution. Goal: push a die face through $X=\omega^2$ and read CDF / atom. Step 1: $F_X(10)=\mathbb{P}[X\le 10]=\mathbb{P}[\omega\in\{1,2,3\}]=1/2$. Step 2: $\mathbb{P}[X=16]=\mathbb{P}[\omega=4]=1/6$. Check: values of $X$ are $1,4,9,16,25,36$; only $\omega\le 3$ give $X\le 10$.

Exercise 2. A r.v. $X$ has CDF

Find the support (atoms), the pmf, and $\mathbb{P}[0<X\le 3]$.

Solution. Goal: recover the discrete law from a staircase CDF. Step 1 (atoms): jumps at $0$, $1$, $3$:

Step 2 (pmf): support $\{0,1,3\}$ with probabilities $(1/4,1/2,1/4)$; sum $=1$. Step 3 (interval): $\{0<X\le 3\}=\{X=1\}\cup\{X=3\}$, probability $1/2+1/4=3/4$, matching $F(3)-F(0)$. Check: $\mathbb{P}[X\le 1]=3/4=F(1)$; right-continuity holds at each jump point.

Chapter 6 — Discrete distributions (Diskrete Verteilungen) #

Named discrete laws, joint pmfs, independence criteria, and the Poisson approximation of the binomial.

6.1 Discrete laws and pmfs #

6.1.1 Definition #

Definition. Let $X$ take values in a finite or countable set $E\subset\mathbb{R}$. The distribution (pmf) of $X$ is

Proposition. $\sum_{x\in E}p_x=1$.

Existence. Conversely, any $(p_x)_{x\in E}$ with $p_x\in[0,1]$ and $\sum_x p_x=1$ is the law of some discrete r.v. on some probability space. One may write: “let $X$ have distribution $(p_x)$.”

When to use

• Specifying a discrete law without building $\Omega$.
• Checking that a candidate formula is a valid pmf (nonnegative, sums to $1$).

Typical mistake Forgetting the support: writing a formula for $k=0,1,\ldots$ when the r.v. only lives on $\{1,2,\ldots\}$ (geometric) or $\{0,\ldots,n\}$ (binomial).

6.2 Bernoulli and Binomial #

6.2.1 Bernoulli #

Definition. $X\sim\mathrm{Ber}(p)$ for $0\le p\le 1$ if $X\in\{0,1\}$ and

When to use

• Single binary outcome; indicators $1_A$.
• Building block for Binomial via sums.

Typical mistake Parameter confusion: some texts write $\mathrm{Ber}(q)$ for failure probability. Stick to $p=\mathbb{P}[X=1]$.

6.2.2 Binomial #

Definition. $X\sim\mathrm{Bin}(n,p)$ if $X\in\{0,\ldots,n\}$ and

Sanity check. $\sum_{k=0}^n\binom{n}{k}p^k(1-p)^{n-k}=(p+(1-p))^n=1$.

Theorem (sum of i.i.d. Bernoullis). If $X_1,\ldots,X_n$ are i.i.d. $\mathrm{Ber}(p)$, then

Remark. $\mathrm{Bin}(1,p)=\mathrm{Ber}(p)$. If $X\sim\mathrm{Bin}(m,p)$, $Y\sim\mathrm{Bin}(n,p)$ are independent, then $X+Y\sim\mathrm{Bin}(m+n,p)$.

When to use

• Fixed number of independent identical trials; count successes.
• Sampling with replacement / with large population relative to sample.

Typical mistake Using Binomial for sampling without replacement from a finite population (use Hypergeometric — not in Acciaio Ch 2, but know the mismatch). Also: writing $\binom{n}{k}p^k$ and forgetting $(1-p)^{n-k}$.

6.3 Geometric and memorylessness #

6.3.1 Geometric as first success #

Definition. For $0<p\le 1$, $T\sim\mathrm{Geom}(p)$ if $T\in\{1,2,3,\ldots\}$ and

(Convention: for $p=1$, take $0^0=1$, so $\mathbb{P}[T=1]=1$.)

Construction. If $X_1,X_2,\ldots$ are i.i.d. $\mathrm{Ber}(p)$ and

then $T\sim\mathrm{Geom}(p)$ (with the harmless abuse that $\mathbb{P}[T=\infty]=0$).

Useful tail. $\mathbb{P}[T>n]=(1-p)^n$ (first $n$ trials all fail).

When to use

• Waiting time until first success in i.i.d. Bernoulli trials.
• Memoryless discrete waiting (next subsection).

Typical mistake Off-by-one support: some textbooks define Geometric as number of failures before first success (support $\{0,1,\ldots\}$). Acciaio / this recap uses trial of first success (support $\{1,2,\ldots\}$).

6.3.2 Memorylessness #

Proposition (absence of memory). Let $T\sim\mathrm{Geom}(p)$ with $0<p<1$. Then for all integers $n\ge 0$ and $k\ge 1$,

Proof sketch. From $\mathbb{P}[T>n]=(1-p)^n$,

When to use

• Conditional waiting problems (“already waited 10 minutes / 10 trials…”).
• Recognizing Geometric among positive integer-valued laws (memoryless characterizes Geometric on $\{1,2,\ldots\}$ among discrete laws on that support).

Typical mistake Applying memorylessness to Binomial or Poisson — those are not memoryless.

6.4 Poisson and Poisson–Binomial approximation #

6.4.1 Poisson law #

Definition. For $\lambda>0$, $N\sim\mathrm{Poisson}(\lambda)$ if $N\in\{0,1,2,\ldots\}$ and

Sanity check. $\sum_{k\ge 0}e^{-\lambda}\lambda^k/k!=e^{-\lambda}e^{\lambda}=1$.

When to use

• Counts of rare independent incidents (typos, arrivals, defects) when $n$ is large and $p$ small with $np\approx\lambda$.
• Limit object for $\mathrm{Bin}(n,\lambda/n)$.

Typical mistake Using Poisson when trials are strongly dependent or when $p$ is not small. Also: confusing parameter $\lambda$ with a probability (it is a mean, can exceed $1$).

6.4.2 Poisson approximation of the binomial #

Proposition. Fix $\lambda>0$. Let $X_n\sim\mathrm{Bin}(n,\lambda/n)$ and let $N\sim\mathrm{Poisson}(\lambda)$. Then for every fixed $k\in\mathbb{N}$,

Proof idea. Write

Application (newspaper typos). A page has $n=10^4$ characters; each mistyped with probability $p=10^{-3}$ (so $\lambda=np=10$). Then e.g.

When to use

• Approximate $\mathrm{Bin}(n,p)$ when $n$ large, $p$ small, $\lambda=np$ moderate.
• Quick rare-event probabilities without summing binomial coefficients.

Typical mistake Using the approximation when $p$ is not small (e.g. $n=10$, $p=0.4$). Always check $p\to 0$ with $np$ stable.

6.5 Joint distributions and independence #

6.5.1 Joint pmf and images #

Definition. For discrete $X_i$ with values in countable $E_i$, the joint distribution is

Independence $\Rightarrow$ product. If $X_1,\ldots,X_n$ are independent,

Proposition (image). For $\varphi\colon\mathbb{R}^n\to\mathbb{R}$ and $Z=\varphi(X_1,\ldots,X_n)$,

When to use

• Laws of sums, maxima, products of discrete r.v.s.
• Building a joint table from a story (two dice, two Poissons, etc.).

Typical mistake Assuming the joint is the product of margins without checking independence. Margins never determine the joint when dependence is possible.

6.5.2 Independence criteria for discrete r.v.s #

Theorem (pair). For discrete $X,Y$, the following are equivalent:

1. $X,Y$ are independent (in the CDF sense of the framework chapter).
2. $\mathbb{P}[X=a,Y=b]=\mathbb{P}[X=a]\mathbb{P}[Y=b]$ for all $a,b$.
3. $\mathbb{E}[f(X)g(Y)]=\mathbb{E}[f(X)]\mathbb{E}[g(Y)]$ whenever the expectations exist.

Theorem ($n$-tuple). Analogous factorization for atoms and for $\mathbb{E}[f_1(X_1)\cdots f_n(X_n)]$.

When to use

• Prove independence from a joint table (check all atoms, not just a few).
• Pull expectations of products apart (Ch 7).

Typical mistake Checking only $\mathbb{P}[X=a,Y=b]=\mathbb{P}[X=a]\mathbb{P}[Y=b]$ for “typical” $(a,b)$ and skipping the zeros / edges of the table.

6.5.3 Almost sure events (parenthesis) #

Definition. An event $A$ occurs almost surely (a.s.) if $\mathbb{P}[A]=1$. Write $X\le Y$ a.s. if $\mathbb{P}[X\le Y]=1$.

When to use

• Hypotheses of expectation theorems (“$X\ge 0$ a.s.”).
• Ignoring null sets (e.g. geometric $T<\infty$ a.s.).

Typical mistake Confusing “a.s.” with “surely for every $\omega$”. On continuous spaces they differ; for finite discrete $\Omega$ with full support they coincide.

6.6 Method map (quick reference) #

6.7 Exercises #

Exercise 1. If $X\sim\mathrm{Ber}(0.2)$, compute $\mathbb{P}[X=0]$ and $F_X(0.5)$.

Solution. Goal: Bernoulli masses and CDF at a point between $0$ and $1$. Step 1: $\mathbb{P}[X=0]=1-0.2=0.8$. Step 2: $F_X(0.5)=\mathbb{P}[X\le 0.5]=\mathbb{P}[X=0]=0.8$. Check: $F_X$ jumps by $0.2$ at $1$, so for $a\in[0,1)$ one has $F_X(a)=0.8$.

Exercise 2. The joint pmf of $(X,Y)$ on $\{0,1\}^2$ is

Are $X$ and $Y$ independent? Compute $\mathbb{P}[X+Y=1]$.

Solution. Goal: independence via the atom test; law of a sum via the image formula. Step 1: compute margins by summing rows/columns:

Step 2: compare one atom: $\mathbb{P}[X=0]\mathbb{P}[Y=0]=0.12\neq 0.1$. Stop — dependent. Step 3: $\{X+Y=1\}=\{(0,1),(1,0)\}$, disjoint, probabilities $0.3+0.2=0.5$. Check: image formula with $\varphi(x,y)=x+y$ sums exactly those two joint masses.

Chapter 7 — Expectation & variance — discrete (Erwartungswert und Varianz) #

Averages and spreads for discrete random variables: image formula, linearity, tail sum, Jensen, and variance under independence.

7.1 Expectation: definition #

7.1.1 Nonnegative and integrable cases #

Definition (nonnegative). If $X$ is discrete and $X\ge 0$ a.s., with values in countable $E$,

Definition (integrable). $X$ is integrable if $\mathbb{E}[|X|]<\infty$. Then

When to use

• Any “average value” question for a discrete r.v.
• As a building block for variance ($\mathbb{E}[X^2]$) and for $\mathbb{E}[\varphi(X)]$.

Typical mistake Writing $\sum x\,p_x$ for a signed r.v. without checking absolute convergence — rearrangements can fail if $\sum |x|p_x=\infty$.

7.2 Tail-sum formula #

7.2.1 Nonnegative integer-valued r.v.s #

Proposition (tail sum). If $X$ is discrete with values in $\mathbb{N}=\{0,1,2,\ldots\}$, then

Proof sketch. Write $\mathbb{P}[X\ge n]=\sum_{k\ge n}\mathbb{P}[X=k]$ and switch sums by Fubini for nonnegative terms: each $k$ contributes to exactly $k$ of the inner indicators $1_{k\ge n}$.

Application (Geometric). For $T\sim\mathrm{Geom}(p)$, $\mathbb{P}[T\ge n]=(1-p)^{n-1}$ for $n\ge 1$, so

When to use

• Geometric and other waiting times with simple tails.
• Integer-valued nonnegative $X$ when survival is easier than the pmf.

Typical mistake Starting the sum at $n=0$ (that term is $1$ and spoils the formula) or applying the formula to signed / non-integer $X$.

7.3 Image formula and linearity #

7.3.1 Expectation of $\varphi(X_1,\ldots,X_n)$ #

Theorem (image / LOTUS — law of the unconscious statistician). Let $X_1,\ldots,X_n$ be discrete on countable $E$, and $\varphi\colon\mathbb{R}^n\to\mathbb{R}$. If

then $Z=\varphi(X_1,\ldots,X_n)$ is integrable and

If $\varphi\ge 0$, the formula holds in $[0,\infty]$ without the integrability check.

Special case $n=1$. $\mathbb{E}[\varphi(X)]=\sum_x\varphi(x)\mathbb{P}[X=x]$ whenever $\sum|\varphi(x)|p_x<\infty$ (or $\varphi\ge 0$).

When to use

• $\mathbb{E}[X^2]$, $\mathbb{E}[e^{tX}]$, $\mathbb{E}[|X-a|]$, etc., directly from the pmf.
• Avoiding an intermediate computation of the law of $\varphi(X)$.

Typical mistake Writing $\mathbb{E}[\varphi(X)]=\varphi(\mathbb{E}[X])$ — false in general (only affine $\varphi$, or Jensen inequalities for convex $\varphi$).

7.3.2 Linearity #

Theorem (linearity). If $X,Y$ are integrable discrete r.v.s and $\alpha\in\mathbb{R}$, then $\alpha X$ and $X+Y$ are integrable and

Hence for integrable $X_1,\ldots,X_n$ and scalars $\alpha_i$,

Application (Binomial mean). If $S\sim\mathrm{Bin}(n,p)$, couple $S$ with $S_n=X_1+\cdots+X_n$ for i.i.d. $\mathrm{Ber}(p)$. Then

(The direct sum $\sum_k k\binom{n}{k}p^k(1-p)^{n-k}$ is unnecessary.)

When to use

• Means of sums, counts via indicators, Binomial/ Poisson additivity stories.
• Any time a r.v. is written as a sum — try linearity before expanding the pmf.

Typical mistake Believing linearity needs independence. Independence is needed for $\mathbb{E}[XY]=\mathbb{E}[X]\mathbb{E}[Y]$ and for adding variances, not for adding expectations.

7.4 Jensen's inequality #

7.4.1 Convex maps and consequences #

Theorem (Jensen). Let $X$ be discrete and $\varphi\colon\mathbb{R}\to\mathbb{R}$ convex. If $\mathbb{E}[\varphi(X)]$ and $\mathbb{E}[X]$ are well-defined, then

Consequences.

1. $|\mathbb{E}[X]|\le\mathbb{E}[|X|]$ for integrable $X$ (take $\varphi(x)=|x|$).
2. If $\mathbb{E}[X^2]<\infty$, then $\mathbb{E}[|X|]\le\sqrt{\mathbb{E}[X^2]}$ (apply Jensen to $t\mapsto t^2$ on $|X|$, or note $\varphi(x)=x^2$ gives $(\mathbb{E}[X])^2\le\mathbb{E}[X^2]$ and combine with absolute values as in the lecture). In particular $\mathbb{E}[X^2]<\infty$ implies integrability of $X$.

When to use

• Justify that finite second moment $\Rightarrow$ finite first moment.
• Quick inequalities relating $\mathbb{E}[X]$, $\mathbb{E}[|X|]$, $\mathbb{E}[X^2]$.

Typical mistake Reversing Jensen for concave $\varphi$ (the inequality flips). Also applying Jensen to non-convex maps like $x\mapsto\sin x$.

7.5 Expectation and independence #

7.5.1 Product rule #

Reminder. For discrete $X,Y$, independence $\Leftrightarrow$ $\mathbb{E}[f(X)g(Y)]=\mathbb{E}[f(X)]\mathbb{E}[g(Y)]$ whenever defined (§6.5.2).

When to use

• Compute $\mathbb{E}[XY]$ under independence: $\mathbb{E}[XY]=\mathbb{E}[X]\mathbb{E}[Y]$.
• Kill cross terms in variance expansions (§7.6).

Typical mistake Using $\mathbb{E}[XY]=\mathbb{E}[X]\mathbb{E}[Y]$ without independence (counterexample: $Y=X$ with $\mathbb{P}[X=\pm 1]=1/2$ gives $\mathbb{E}[XY]=1\neq 0=\mathbb{E}[X]\mathbb{E}[Y]$). Independence also does not give $\mathbb{E}[X/Y]=\mathbb{E}[X]/\mathbb{E}[Y]$.

7.6 Variance #

7.6.1 Definition and computational identity #

Definition. If $\mathbb{E}[X^2]<\infty$, the variance is

The standard deviation is $\sqrt{\mathrm{Var}(X)}$.

Proposition (computational form).

Examples.

• If $X\equiv a$ a.s., then $\mathrm{Var}(X)=0$.
• If $\mathbb{P}[X=a]=\mathbb{P}[X=b]=1/2$, then $\mathrm{Var}(X)=\bigl(\frac{b-a}{2}\bigr)^2$.
• If $X\sim\mathrm{Ber}(p)$, then $\mathbb{E}[X^2]=p$, so $\mathrm{Var}(X)=p-p^2=p(1-p)$.

When to use

• Quantify spread; compare fluctuations of different laws with the same mean.
• Prepare for concentration / LLN (law of large numbers) heuristics: sums of i.i.d. have std. dev. growing like $\sqrt{n}$.

Typical mistake Forgetting to subtract $(\mathbb{E}[X])^2$, or computing $\mathbb{E}[(X-m)]$ (that is always $0$) instead of $\mathbb{E}[(X-m)^2]$.

7.6.2 Variance of sums under independence #

Proposition. If $X_1,\ldots,X_n$ are pairwise independent with finite second moments and $S=X_1+\cdots+X_n$, then

Application (Binomial variance). For $S\sim\mathrm{Bin}(n,p)$, couple with i.i.d. Bernoullis:

Hence $\mathbb{E}[S]=np$ while $\sqrt{\mathrm{Var}(S)}=\sqrt{np(1-p)}$ grows like $\sqrt{n}$.

When to use

• Variance of sums of independent trials / indicators with weak dependence replaced by independence assumptions.
• Comparing signal ($np$) to noise ($\sqrt{np(1-p)}$).

Typical mistake Adding variances for dependent summands (e.g. without-replacement indicators). Also writing $\mathrm{Var}(aX)=a\mathrm{Var}(X)$ instead of $a^2\mathrm{Var}(X)$.

7.6.3 Binomial applications checklist #

Pattern

When to use

• Exam triage for “$n$ independent trials” stories.
• Quickly discarding impossible approaches (expanding $\sum k\binom{n}{k}\cdots$ by hand).

Typical mistake Computing $\mathbb{E}[S]$ by differentiating the probability generating function when linearity is available in one line.

7.7 Method map (quick reference) #

7.8 Exercises #

Exercise 1. $X\sim\mathrm{Ber}(0.3)$. Compute $\mathbb{E}[X]$, $\mathbb{E}[X^2]$, $\mathrm{Var}(X)$.

Solution. Goal: Bernoulli mean and variance from the two-point support. Step 1: $\mathbb{E}[X]=0\cdot 0.7+1\cdot 0.3=0.3$. Step 2: $\mathbb{E}[X^2]=0^2\cdot 0.7+1^2\cdot 0.3=0.3$. Step 3: $\mathrm{Var}(X)=\mathbb{E}[X^2]-(\mathbb{E}[X])^2=0.3-0.09=0.21=0.3\cdot 0.7$. Check: general formula $\mathrm{Var}(\mathrm{Ber}(p))=p(1-p)$.

Exercise 2. A device fails each day independently with probability $p\in(0,1)$. Let $T$ be the day of first failure (so $T\sim\mathrm{Geom}(p)$). Let $A=\{T>10\}$.

• Compute $\mathbb{E}[1_A]$ and $\mathbb{E}[T\mid T>10]$ using memorylessness (interpret the conditional expectation as the mean of $10+\widetilde T$ with $\widetilde T\sim\mathrm{Geom}(p)$).
• If one instead runs $n$ independent devices and lets $S$ be the number still working after $10$ days, identify the law of $S$ and give $\mathbb{E}[S]$, $\mathrm{Var}(S)$.

Solution. Goal: combine indicators, Geometric memoryless, and Binomial mean/variance. Step 1 (single device). $A=\{T>10\}$ means the first $10$ days have no failure, probability $(1-p)^{10}$. Hence $\mathbb{E}[1_A]=\mathbb{P}[A]=(1-p)^{10}$. Step 2 (conditional mean). On $\{T>10\}$, write $T=10+T'$ where $T'$ is the additional wait until failure. Memorylessness (Ch 6) gives $T'\sim\mathrm{Geom}(p)$ independent of the past in the conditional law, so

Step 3 (many devices). Device $i$ still works after $10$ days iff its failure time exceeds $10$, an event of probability $q=(1-p)^{10}$. Independence across devices $\Rightarrow$ $S\sim\mathrm{Bin}(n,q)$. Step 4: $\mathbb{E}[S]=nq$, $\mathrm{Var}(S)=nq(1-q)$ by §7.6.2. Check: for $p=1$, $q=0$, so $S=0$ a.s., mean and variance $0$. For $p\to 0$ with $n$ fixed, $q\to 1$ and $\mathbb{E}[S]\to n$.

Chapter 8 — Continuous distributions (Stetige Verteilungen) #

Densities replace probability mass functions: Uniform, Exponential, Normal; transforms; expectation, variance, Jensen; Cauchy as a non-example.

8.1 Continuous random variables and densities #

8.1.1 Overview #

Definition (Continuous random variable). A random variable $X\colon\Omega\to\mathbb{R}$ is continuous if its CDF $F_X$ can be written as

for all $a\in\mathbb{R}$, for some nonnegative function $f\colon\mathbb{R}\to\mathbb{R}_+$ called a density of $X$ (often written $f_X$).

Immediate consequences

• $F_X$ is continuous, and $\mathbb{P}[X=a]=F_X(a)-F_X(a-)=0$ for every $a$.
• For $a\le b$, all interval variants agree:

• A density must integrate to one: $\int_{-\infty}^{\infty} f(x)\,dx=1$. Values of $f$ at finitely many points can be changed arbitrarily without changing the law.

When to use

• Outcomes are lengths, times, or measurement values with a continuum of possibilities.
• The CDF looks “smooth” (no jumps) and you need interval probabilities.

Typical mistake

• Reading $f(x)$ as a probability. Densities can exceed $1$; only integrals $\int_A f$ are probabilities.
• Writing $\mathbb{P}[X=a]=f(a)$ (that is discrete thinking). Continuously, $\mathbb{P}[X=a]=0$.

8.1.2 Recognizing a continuous law from the CDF #

Theorem (Density from $F_X$). Let $X$ be a random variable. Assume $F_X$ is continuous and piecewise $C^1$: there exist $-\infty=x_0<x_1<\cdots<x_n=+\infty$ such that $F_X$ is $C^1$ on every $(x_i,x_{i+1})$. Then $X$ is continuous, and a density is given by

with arbitrary values at the finitely many breakpoints $x_1,\ldots,x_{n-1}$.

Sketch of reason: On each open interval where $F$ is $C^1$, the fundamental theorem of calculus gives $F(b)-F(a)=\int_a^b F'$. Continuity of $F$ glues the pieces into $F(x)=\int_{-\infty}^x f$.

When to use

• You have computed $F_Y$ for $Y=g(X)$ and want $f_Y$.
• A problem states $F_X$ explicitly and asks for a density.

Typical mistake

• Differentiating a CDF that still has jumps (mixed / discrete component) and claiming a density exists on all of $\mathbb{R}$.
• Forgetting to treat intervals separately when $F$ is only piecewise smooth (e.g. at $0$ for $Y=X^2$).

8.2 Uniform distribution #

8.2.1 Overview #

Definition. $X\sim\mathcal{U}([a,b])$ (with $a<b$) if it has density

CDF

Key property. For any subinterval $[c,c+\ell]\subset[a,b]$,

When to use

• No preferred location inside a known bounded interval (random angle, random seed on $[0,1]$, …).
• Building block for transforms (e.g. $X\sim\mathcal{U}([0,1])$, then $Y=X^n$).

Typical mistake

• Using density $1$ on $[a,b]$ when $b-a\neq 1$ (must be $1/(b-a)$).
• Confusing continuous Uniform with discrete Uniform on $\{1,\ldots,n\}$.

8.3 Exponential distribution #

8.3.1 Overview #

Definition. $T\sim\mathrm{Exp}(\lambda)$ with rate $\lambda>0$ if

CDF and tail

Memorylessness. For all $t,s\ge 0$,

Reason: $\mathbb{P}[T>t+s]/\mathbb{P}[T>t]=e^{-\lambda(t+s)}/e^{-\lambda t}=e^{-\lambda s}$.

When to use

• Waiting times / interarrival times with no aging (memoryless).
• Continuous analogue of Geometric (discrete memoryless law).

Typical mistake

• Confusing rate $\lambda$ with mean: $\mathbb{E}[T]=1/\lambda$, not $\lambda$.
• Writing density $\lambda e^{-\lambda x}$ for $x\in\mathbb{R}$ (support is $[0,\infty)$).
• Using $\mathbb{P}[T\ge t]=\lambda e^{-\lambda t}$ (that is the density, not the tail).

8.4 Normal distribution #

8.4.1 Overview #

Definition. $X\sim\mathcal{N}(m,\sigma^2)$ with $m\in\mathbb{R}$ and $\sigma^2>0$ if

Standard Normal. $Z\sim\mathcal{N}(0,1)$ has density $\varphi$ and CDF $\Phi$:

Affine stability. If $X_1,\ldots,X_n$ are independent, $X_i\sim\mathcal{N}(m_i,\sigma_i^2)$, and $a_0,a_1,\ldots,a_n\in\mathbb{R}$, then

In particular, if $X\sim\mathcal{N}(0,1)$, then $m+\sigma X\sim\mathcal{N}(m,\sigma^2)$. Conversely, if $X\sim\mathcal{N}(m,\sigma^2)$, then

Three-sigma rule of thumb. Most mass sits in $[m-3\sigma,m+3\sigma]$:

which is independent of $(m,\sigma)$ after standardization.

When to use

• Measurement error / additive noise models.
• Any time you standardize to use a $\Phi$-table / software.
• Linear combinations of independent Normals (exact Normal, not only asymptotic). $\mathbb{P}[S<T]$ is $\mathbb{P}[S-T<0]$.

Typical mistake

• Writing $\mathcal{N}(m,\sigma)$ when the second parameter is variance $\sigma^2$ (check the course convention: Acciaio uses $\mathcal{N}(m,\sigma^2)$).
• Forgetting to square the coefficients in $\mathrm{Var}(\sum a_i X_i)=\sum a_i^2\sigma_i^2$.
• Treating $\Phi$ values as densities.

8.5 Density of a transformed random variable #

8.5.1 Overview #

Method (density of $Y=g(X)$)

1. Compute $F_Y(y)=\mathbb{P}[g(X)\le y]$ (express the event in terms of $X$).
2. If $F_Y$ is continuous and piecewise $C^1$, set $f_Y(y)=F_Y'(y)$ on each smooth piece (Theorem in §8.1.2).

Caveat. $g(X)$ need not be continuous. A constant map yields a discrete (Dirac) law with no density. The method needs $X$ and $g$ “nice enough.”

When to use

• Squaring, absolute values, powers, monotone maps of a continuous RV.
• Linear images of Normal (often faster via affine stability in §8.4).

Typical mistake

• Differentiating under the wrong branch when $g$ is not one-to-one (e.g. $Y=X^2$: both $-\sqrt{y}$ and $+\sqrt{y}$ contribute).
• Claiming $f_Y(y)=f_X(g^{-1}(y))$ without the Jacobian / chain-rule factor from differentiating $F_Y$.

8.6 Expectation and variance (continuous) #

8.6.1 Expectation via the density #

Definition (Nonnegative case). If $X$ is continuous with density $f$ and $X\ge 0$ a.s., set

Definition (Integrable case). $X$ is integrable if $\mathbb{E}[|X|]<\infty$. Then

Law of the unconscious statistician (LOTUS). If $\int |g(x)|f(x)\,dx<\infty$,

This is consistent with computing $\mathbb{E}[Y]$ from a density of $Y=g(X)$ when that density exists, and with the discrete formula when $Y$ is discrete.

Classical means

Uniform: $\int_a^b x\cdot\frac{1}{b-a}\,dx=(a+b)/2$.

Exponential: integrate by parts $\int_0^\infty x\lambda e^{-\lambda x}\,dx=1/\lambda$.

Normal: write $X\stackrel{d}{=}m+\sigma Y$ with $Y$ standard; $\mathbb{E}[Y]=0$ because $y\varphi(y)$ is odd.

When to use

• Any continuous mean / moment computation.
• Prefer LOTUS over finding $f_{g(X)}$ when only $\mathbb{E}[g(X)]$ is needed.

Typical mistake

• Computing $\int g(y)f_X(y)\,dy$ with the wrong variable after a transform (mix $X$ and $Y$).
• Declaring $\mathbb{E}[X]$ for a non-integrable law (Cauchy — next block).

8.6.2 Cauchy: expectation need not exist #

Definition (Cauchy). Density

Non-integrability.

Pathological limits. Different symmetric / asymmetric truncations give different finite limits for $\int_{-A}^{B} x/(\pi(1+x^2))\,dx$, so there is no natural value for “$\mathbb{E}[X]$”.

When to use

• Counterexample whenever someone claims “every continuous RV has a mean.”
• Warning that $\int x f(x)\,dx$ may fail to exist even if $f$ is even and nice-looking.

Typical mistake

• Claiming $\mathbb{E}[X]=0$ for Cauchy “by symmetry.” Symmetry alone does not define the integral when $\mathbb{E}[|X|]=\infty$.

8.6.3 Jensen’s inequality #

Theorem (Jensen). Let $g\colon\mathbb{R}\to\mathbb{R}$ be convex. If $X$ is continuous and $\mathbb{E}[X]$, $\mathbb{E}[g(X)]$ are well defined, then

When to use

• Comparing $\mathbb{E}[X^2]$ and $(\mathbb{E}[X])^2$ (variance nonnegative).
• Bounding $\mathbb{E}[e^{tX}]$, $\mathbb{E}[|X|]$, etc.

Typical mistake

• Reversing the inequality for concave $g$ (then $\ge$ becomes $\le$ after a sign flip / use $-g$).
• Applying Jensen when $\mathbb{E}[|X|]=\infty$.

8.6.4 Variance #

Definition. If $\mathbb{E}[X^2]<\infty$ and $m=\mathbb{E}[X]$,

The standard deviation is $\sigma_X=\sqrt{\mathrm{Var}(X)}$.

Proposition (basic properties). Assume $\mathbb{E}[X^2]<\infty$.

1. $\mathrm{Var}(X)=\mathbb{E}[X^2]-(\mathbb{E}[X])^2$.
2. $\mathrm{Var}(aX+\mu)=a^2\,\mathrm{Var}(X)$ for $a,\mu\in\mathbb{R}$.
3. If $X_1,\ldots,X_n$ are pairwise independent with finite second moments and $S=\sum_i a_i X_i$, then $\mathrm{Var}(S)=\sum_i a_i^2\,\mathrm{Var}(X_i)$.

Classical variances

Uniform sketch: $Y\sim\mathcal{U}([0,1])$ has $\mathbb{E}[Y^2]=\int_0^1 x^2\,dx=1/3$, mean $1/2$, so $\mathrm{Var}(Y)=1/3-1/4=1/12$. Then $X=a+(b-a)Y$ gives factor $(b-a)^2$.

Exponential: $\mathbb{E}[X^2]=2/\lambda^2$ by parts twice; subtract $(1/\lambda)^2$.

Normal: reduce to standard $Y$; integrate by parts to get $\mathbb{E}[Y^2]=1$, then scale by $\sigma^2$.

When to use

• Spread / standardization / CLT assumptions ($\mathrm{Var}(X_1)<\infty$).
• Scaling rules under affine maps.

Typical mistake

• Using $\mathrm{Var}(aX+\mu)=a\,\mathrm{Var}(X)$ (missing the square).
• Adding variances without an independence (or pairwise uncorrelation) hypothesis.

8.7 Method map (quick reference) #

8.8 Exercises #

Exercise 1. $X\sim\mathcal{U}([1,5])$. Compute $\mathbb{P}[2\le X\le 4]$ and $F_X(3)$.

Solution. Goal: Uniform probabilities via length ratios and the Uniform CDF. Step 1: length of $[2,4]$ over length of $[1,5]$ gives $\mathbb{P}=2/4=1/2$. Step 2: $F_X(3)=(3-1)/(5-1)=1/2$. Check: density $1/4$ on $[1,5]$, so $\int_2^4\frac14\,dx=1/2$.

Exercise 2. $X\sim\mathcal{U}([0,1])$. Find the density of $Y=-\log X$ (for $X\in(0,1]$).

Solution. Goal: $X\sim\mathcal{U}([0,1])$, $Y=-\log X$. Identify the law of $Y$. Step 1 (support). $X\in(0,1]$ a.s. $\Rightarrow Y=-\log X\in[0,\infty)$. Step 2 (CDF). For $y>0$,

because $\mathbb{P}[X\ge u]=1-u$ for $u\in[0,1]$. Step 3 (density). $F_Y$ is continuous, $C^1$ on $(0,\infty)$, and $F_Y'(y)=e^{-y}$. Hence $Y\sim\mathrm{Exp}(1)$. Check: $\mathbb{E}[Y]=\int_0^1 (-\log x)\,dx=1$, matching $\mathrm{Exp}(1)$.

Exercise 3. Let $X$ have density $f(x)=c\,x$ on $[0,2]$ (and $0$ elsewhere). Find $c$, then $\mathbb{E}[X]$ and $\mathrm{Var}(X)$.

Solution. Goal: normalize a triangular-type density and compute moments. Step 1: $\int_0^2 c x\,dx=2c=1\Rightarrow c=1/2$. Step 2: $\mathbb{E}[X]=\int_0^2 x\cdot\frac{x}{2}\,dx=\frac{4}{3}$. Step 3: $\mathbb{E}[X^2]=\int_0^2 x^2\cdot\frac{x}{2}\,dx=2$, so $\mathrm{Var}(X)=2-(4/3)^2=2/9$. Check: $f\ge 0$ and integrates to $1$; $\mathrm{Var}>0$.

Chapter 9 — Joints, marginals & asymptotics (Gemeinsame Verteilungen, GGZ, ZGWS) #

Joint densities, marginals, independence; then the LLN / GGZ (Gesetz der großen Zahlen) and the CLT / ZGWS (zentraler Grenzwertsatz).

9.1 Joint densities #

9.1.1 Overview #

Definition. Random variables $X,Y\colon\Omega\to\mathbb{R}$ have a continuous joint distribution if there exists $f\colon\mathbb{R}^2\to\mathbb{R}_+$ such that for all $-\infty\le a\le a'\le\infty$ and $-\infty\le b\le b'\le\infty$,

Such an $f$ is a joint density of $(X,Y)$ (written $f_{X,Y}$).

Normalization. Necessarily

Conversely, any nonnegative integrable $f$ with integral $1$ is a joint density for some pair $(X,Y)$ on a suitable space.

When to use

• Two continuous measurements at once (position in the plane, two waiting times, …).
• Computing $\mathbb{P}[(X,Y)\in R]$ for a rectangle — or, by approximation / integration, for nicer regions.

Typical mistake

• Treating $f(x,y)$ as a probability.
• Integrating only once when the event constrains both coordinates.

9.2 Marginal densities #

9.2.1 Overview #

Formulas. If $f_{X,Y}$ is a joint density, then $X$ and $Y$ are continuous with

Reason: $F_X(a)=\mathbb{P}[X\le a]=\int_{-\infty}^a\bigl(\int_{-\infty}^{\infty} f(x,y)\,dy\bigr)dx$.

When to use

• “What is the law of $X$ alone?” given a joint density.
• Checking independence via $f_{X,Y}\stackrel{?}{=}f_X f_Y$.

Typical mistake

• Using the joint formula restricted to a curve without integrating (confusing with conditional densities later).
• Forgetting the domain constraints inside the indicator (disk example).

9.3 Independence (continuous) #

9.3.1 Overview #

Theorem (Independence criteria). Let $X,Y$ be continuous with densities $f_X,f_Y$. The following are equivalent:

1. $X,Y$ are independent;
2. $(X,Y)$ is jointly continuous with $f_{X,Y}(x,y)=f_X(x)\,f_Y(y)$;
3. For all (suitable) $g,h\colon\mathbb{R}\to\mathbb{R}$,

whenever the expectations exist.

Consequence. Independent continuous RVs are automatically jointly continuous (product density).

Expectation of a function of the pair. If $f_{X,Y}$ exists and the integral is well defined,

Linearity (jointly continuous case). Applying this to $g(x,y)=ax+\mu y$ and using marginals recovers

When to use

• Prove / disprove independence by comparing $f_{X,Y}$ to $f_X f_Y$.
• Factor double integrals under independence.
• Compute $\mathbb{E}[g(X,Y)]$ when only the joint density is known.

Typical mistake

• Concluding independence from “both marginals look Uniform” (disk: marginals are not Uniform, and even if they were nice, the joint may not factor).
• Checking equality $f_{X,Y}=f_X f_Y$ only on part of the support.

9.4 Law of large numbers (GGZ) #

9.4.1 Overview #

Setup. Fix i.i.d. $X_1,X_2,\ldots$ with common CDF $F$, and write

Theorem (Strong law of large numbers). If $\mathbb{E}[|X_1|]<\infty$ and $m=\mathbb{E}[X_1]$, then

Meaning of “a.s.”. Define

Then $E$ is an event and $\mathbb{P}[E]=1$: the exceptional outcomes where the average fails have probability zero.

Remark. Only $X_1$ appears in the hypothesis because identical distribution passes finite mean to every $X_i$.

When to use

• Justifying “sample mean $\approx$ expectation” for large $n$.
• Bernoulli frequency $\to p$; Exponential sample mean $\to 1/\lambda$.

Typical mistake

• Thinking LLN gives the rate of fluctuations (that is CLT: order $1/\sqrt{n}$).
• Applying LLN without an integrability assumption (Cauchy averages do not settle to a mean).

9.5 Central limit theorem (ZGWS) #

9.5.1 Overview #

Gaussian warm-up. If $X_i$ are i.i.d. $\mathcal{N}(m,\sigma^2)$, then exactly

so typical size is $\sigma/\sqrt{n}$. The standardized sum

is exactly $\mathcal{N}(0,1)$ for every $n$.

Theorem (Central limit theorem). Assume $\mathbb{E}[X_1^2]<\infty$, write $m=\mathbb{E}[X_1]$, $\sigma^2=\mathrm{Var}(X_1)>0$, and $S_n=X_1+\cdots+X_n$. Then for every $a\in\mathbb{R}$,

Equivalent average form.

When to use

• Approximate $\mathbb{P}[S_n\le t]$ or $\mathbb{P}[\bar X_n\in I]$ for large $n$ by standardizing and reading $\Phi$.
• Binomial tails at $n\approx 100$: write $X$ as a sum of i.i.d. Bernoullis, then CLT (exam pattern in §10.5.5).
• Build confidence intervals / justify Normal error models later in statistics.

Typical mistake

• Forgetting to standardize (using $S_n-nm$ without dividing by $\sigma\sqrt{n}$).
• Using CLT when $\mathrm{Var}(X_1)=\infty$.
• Treating the CLT statement as almost-sure convergence of the standardized sum (it is convergence in distribution / CDF convergence to $\Phi$).

9.6 Method map (quick reference) #

9.7 Exercises #

Exercise 1. $f_{X,Y}(x,y)=1$ on $[0,1]^2$. Compute $\mathbb{P}[X\le\tfrac12,\,Y\le\tfrac12]$.

Solution. Goal: rectangle probability under the uniform joint density on the unit square. Step 1: the event is $[0,1/2]\times[0,1/2]$, area $1/4$. Step 2: $\mathbb{P}=\iint_{[0,1/2]^2} 1\,dx\,dy=1/4$. Check: length $\times$ length under independence of coordinates.

Exercise 2. $f_{X,Y}(x,y)=c\,xy$ on $[0,1]^2$ (zero elsewhere). Find $c$, the marginal $f_X$, and decide independence.

Solution. Goal: normalize, marginalize, and test factorization. Step 1: $\int_0^1\int_0^1 c xy\,dx\,dy=c/4=1\Rightarrow c=4$. Step 2: $f_X(x)=\int_0^1 4xy\,dy=2x$ on $[0,1]$. Similarly $f_Y(y)=2y$. Step 3: product $f_X f_Y=4xy=f_{X,Y}$ on the square $\Rightarrow$ independent. Check: both margins integrate to $1$; factorization holds on the whole support.

Exercise 3. $X_i$ i.i.d. with $\mathbb{E}[X_1]=2$, $\mathrm{Var}(X_1)=4$. Approximate $\mathbb{P}[\bar X_{64}>2.25]$.

Solution. Goal: Approximate $\mathbb{P}[\bar X_{64}>2.25]$ when $m=2$, $\sigma^2=4$. Step 1: CLT uses $Z_n=\sqrt{n}\,(\bar X_n-m)/\sigma$. Step 2: Here $n=64$, $\sigma=2$, so

Step 3: $\bar X_{64}>2.25\Leftrightarrow Z_{64}>1$. Step 4: $\mathbb{P}[Z_{64}>1]\approx 1-\Phi(1)$. Check: Same as $\mathbb{P}[S_{64}>64\cdot 2.25]=\mathbb{P}[S_{64}>144]$, since $S_{64}-128$ standardized by $\sigma\sqrt{n}=2\cdot 8=16$ gives threshold $(144-128)/16=1$.

Chapter 10 — Statistics (Statistik) #

From a parametric probabilistic model and observed data, estimate unknown parameters, quantify uncertainty with confidence intervals, and decide between competing hypotheses.

10.1 Parameter estimation overview #

10.1.1 Overview #

Definition (parametric model, parametric model / parametrisches Modell) We study a model given by i.i.d. random variables $X_1,\ldots,X_n$ whose common distribution depends on a real parameter $\theta$ (or several parameters $\theta_1,\theta_2,\ldots$). Typical discrete families:

• Bernoulli($p$), $p\in[0,1]$
• Geometric($p$), $p\in[0,1]$
• Poisson($\lambda$), $\lambda>0$

Typical continuous families:

• Uniform$[0,\theta]$, $\theta\ge 0$
• Exponential($\lambda$), $\lambda>0$
• Normal($m,\sigma^2$), $m\in\mathbb{R}$, $\sigma^2>0$

Definition (realization / Realisierung) A realization is a vector $x=(x_1,\ldots,x_n)\in\mathbb{R}^n$ of observed values for $(X_1,\ldots,X_n)$. The PVK phrase: “mögliche Werte, die bei einem Zufallsexperiment herauskommen.”

Main question Given $x$, can we estimate the underlying parameter(s)? Write an estimator as a function of the data, $ \widehat{\theta}=\widehat{\theta}(x_1,\ldots,x_n). $ When the same formula is applied to the random sample, one also writes $\widehat{\theta}(X_1,\ldots,X_n)$ — that random variable has a sampling distribution used later for intervals and tests.

When to use

• The exam states “i.i.d. with unknown parameter $\theta$” and asks for an estimator, CI (confidence interval), or test.
• You are told the family (Bernoulli, Normal, …) but not the numerical value of $\theta$.

Typical mistake

• Treating $\theta$ itself as random. In the classical setup of this course, $\theta$ is fixed but unknown; randomness sits in the sample (and therefore in $\widehat{\theta}$, interval endpoints, and the decision $d$).

10.1.2 Estimator vs estimate #

Definition

• An estimator is a measurable function $T=T(X_1,\ldots,X_n)$ of the sample (e.g. $\overline{X}$, $\max_i X_i$).
• An estimate is the realized number $T(x)$ for observed data $x$.

In this course the default construction for $T$ is the MLE (maximum likelihood estimator) $\widehat{\theta}(X)$. Later, confidence intervals and tests are built from the sampling distribution of $T$ under the true $\theta$.

When to use

• Any exam wording that mixes “Schätzer” (random) and a numerical “Schätzwert”.

Typical mistake

• Writing $\mathbb{P}[\widehat{\theta}=5/8]$ after $\widehat{\theta}(x)=5/8$ has already been computed from fixed data — that probability is about the procedure before seeing $x$, not about a constant.

10.2 MLE discrete #

10.2.1 Likelihood and MLE (discrete) #

Definition (likelihood / Likelihood-Funktion) Let $X_1,\ldots,X_n$ be i.i.d. discrete on a finite or countable set $E$, with $ \mathbb{P}[X_i=x]=p_\theta(x),\qquad x\in E. $ For a realization $x=(x_1,\ldots,x_n)\in E^n$, the likelihood is $ L(\theta)=L_x(\theta)=\mathbb{P}[X_1=x_1,\ldots,X_n=x_n]. $ Under i.i.d., this factors as $ L_x(\theta)=p_\theta(x_1)\cdots p_\theta(x_n). $

Definition (maximum likelihood estimator / Maximum-Likelihood-Schätzer) The MLE $\widehat{\theta}=\widehat{\theta}(x)$ is a parameter value maximizing the likelihood: $ L_x(\widehat{\theta})=\max_\theta L_x(\theta). $

Rule (log-likelihood trick) Products become sums under $\log$. Define $\ell(\theta)=\log L_x(\theta)$ (any base; natural log is fine). Critical points of $\ell$ are the same as those of $L$ where $L>0$. The PVK checklist:

1. Write the joint pmf → $L_x(\theta)$.
2. Pass to $\log L$.
3. Set $\partial\log L/\partial\theta=0$.
4. Solve for $\theta$; call the solution $\widehat{\theta}$.
5. Evaluate numerically if data are given.
6. (Sanity) check boundaries / second derivative if the critical point is not obviously a max.

When to use

• Discrete i.i.d. family with explicit $p_\theta(x)$ (Bernoulli, Binomial counts via sum, Geometric, Poisson).
• Exam asks for “Maximum-Likelihood-Schätzer” or “MLE”.

Typical mistake

• Maximizing $L$ with respect to the data instead of $\theta$. The sample is fixed; $\theta$ is the variable.
• Forgetting support constraints (e.g. $p\in[0,1]$, $\lambda>0$) and reporting a critical point outside the parameter space.
• Writing $L$ as a sum instead of a product for independent observations.

10.2.2 Bernoulli ↔ Binomial reduction #

Formula $ \mathbb{P}[S=s]=\binom{n}{s}p^s(1-p)^{n-s}. $ The binomial coefficient does not depend on $p$, so $ \widehat{p}=\frac{s}{n} $ whether you start from $(x_1,\ldots,x_n)$ or from the single count $s$.

When to use

• Problem gives only “$s$ successes in $n$ trials”.

Typical mistake

• Treating $\binom{n}{s}$ as part of the parameter-dependent piece and trying to “maximize” it.

10.3 MLE continuous #

10.3.1 Likelihood via densities #

Definition (continuous likelihood) Let $X_1,\ldots,X_n$ be i.i.d. continuous with density $f_\theta$. For a realization $x\in\mathbb{R}^n$, $ L(\theta)=L_x(\theta)=f_\theta(x_1)\cdots f_\theta(x_n). $ Intuitively $L_x(\theta)$ scales the probability of a small box around $x$; the constant volume factor does not affect the maximizing $\theta$.

Definition (MLE, possibly multiparameter) $ L_x(\widehat{\theta})=\max_\theta L_x(\theta). $ If the model has several parameters $(\theta_1,\theta_2,\ldots)$, maximize jointly: $ L_x(\widehat{\theta}_1,\widehat{\theta}_2,\ldots)=\max_{\theta_1,\theta_2,\ldots} L_x(\theta_1,\theta_2,\ldots). $

When to use

• Continuous i.i.d. family with an explicit density (Normal, Exponential, Uniform$[0,\theta]$, …).
• Same log-trick as in the discrete case; use partial derivatives if there are several parameters.

Typical mistake

• Using the CDF instead of the density in $L$.
• For Uniform$[0,\theta]$, differentiating as if the support did not depend on $\theta$ (the max is typically on the boundary $\widehat{\theta}=\max_i x_i$, not from $\ell'=0$).
• Confusing the MLE variance $\widehat{\sigma}^2=\frac1n\sum(x_i-\widehat{m})^2$ with the unbiased sample variance that divides by $n-1$.

Remark (bias of $\widehat{\sigma}^2$) Under $\mathcal{N}(m,\sigma^2)$, $ \mathbb{E}[\widehat{\sigma}^2]=\frac{n-1}{n}\sigma^2\neq\sigma^2 $ for finite $n$. The MLE is still the maximizer of $L$; unbiasedness is a different criterion. The usual unbiased estimator divides by $n-1$. Exams in this course typically want the MLE formula with $1/n$ unless they ask for unbiasedness explicitly.

10.4 Confidence intervals #

10.4.1 What a CI means #

Definition ($z\%$-confidence interval / Konfidenzintervall) For a model with parameter $\theta$ and $0\le z\le 100$, a $z\%$-confidence interval associates to each realization $x$ an interval $I(x)=[a(x),b(x)]$ such that $ \forall\theta\qquad \mathbb{P}\bigl[a(X_1,\ldots,X_n)\le\theta\le b(X_1,\ldots,X_n)\bigr]\ge\frac{z}{100}. $ Equivalently one often writes level $1-\alpha$ with $\alpha=1-z/100$ (e.g. $z=95$ ↔ $\alpha=0.05$).

Remark (what is random?) The parameter $\theta$ is not random. The endpoints $a(X)$, $b(X)$ are. After observing $x$, the statement “$\theta\in I(x)$” is either true or false; the $z\%$ refers to the procedure, not to a posterior probability on $\theta$.

When to use

• You have an estimator with a known (or approximate) sampling distribution and need a quantitative uncertainty statement.
• Exam: “Konfidenzintervall zum Niveau …” / “95%-CI for $m$”.

Typical mistake

• Saying “$\theta$ lies in $I(x)$ with probability 95%” after $x$ is fixed, as if $\theta$ were random.
• Using $c=1.96$ with the wrong standardization (wrong $\sigma/\sqrt{n}$ factor).
• Mixing up one-sided and two-sided quantiles.

10.4.2 Exact CI for the normal mean (known $\sigma^2=1$) #

Formula (Acciaio + PVK table) Let $X_1,\ldots,X_n$ be i.i.d. $\mathcal{N}(m,1)$, and $\widehat{m}=\overline{X}$. For $c>0$, $ \mathbb{P}\Bigl[\widehat{m}-\frac{c}{\sqrt{n}}\le m\le\widehat{m}+\frac{c}{\sqrt{n}}\Bigr] =\mathbb{P}[-c\le Z\le c]=2\Phi(c)-1, $ where $ Z=\sqrt{n}\,(\widehat{m}-m)\sim\mathcal{N}(0,1),\qquad \Phi(c)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{c}e^{-t^2/2}\,dt. $ Choose $c$ so that $2\Phi(c)-1\ge z/100$, i.e. $ c=\Phi^{-1}\Bigl(1-\frac{\alpha}{2}\Bigr),\qquad\alpha=1-\frac{z}{100}. $ Then $ I(x)=\Bigl[\widehat{m}(x)-\frac{c}{\sqrt{n}},\ \widehat{m}(x)+\frac{c}{\sqrt{n}}\Bigr] $ is a $z\%$-CI for $m$.

PVK / standard normal critical values (two-sided)

In particular, $c=1.96$ gives (at least) a 95%-CI: $ I(x)=\Bigl[\widehat{m}(x)-\frac{1.96}{\sqrt{n}},\ \widehat{m}(x)+\frac{1.96}{\sqrt{n}}\Bigr]. $

When to use

• Normal model with known variance $\sigma^2=1$ as in the lecture (or rescale: if $\sigma^2=\sigma_0^2$ known, replace $c/\sqrt{n}$ by $c\,\sigma_0/\sqrt{n}$).
• You need an exact finite-sample guarantee via $\Phi$, not a large-$n$ approximation.

Typical mistake

• Forgetting the $\sqrt{n}$ (using $c$ instead of $c/\sqrt{n}$ as half-width).
• Using $\Phi(c)$ instead of $2\Phi(c)-1$ for a two-sided interval.

The shaded central band is the probability mass $2\Phi(1.96)-1\approx 0.95$ that defines the half-width $1.96/\sqrt{n}$ of the CI for $m$.

10.4.3 Coverage intuition (same formula, many samples) #

Larger $n$ shrinks the interval like $1/\sqrt{n}$; the coverage probability stays $\approx 95\%$ if you keep $c=1.96$. To get a narrower interval at the same coverage you need more data (or smaller $\sigma$).

10.4.4 Known variance $\sigma_0^2\neq 1$ (same idea) #

Formula $ I(x)=\Bigl[\overline{x}-\frac{c\,\sigma_0}{\sqrt{n}},\ \overline{x}+\frac{c\,\sigma_0}{\sqrt{n}}\Bigr], \qquad c=\Phi^{-1}(1-\alpha/2). $ Acciaio’s lecture works the case $\sigma_0=1$; exams sometimes leave $\sigma_0$ symbolic.

When to use

• Normal mean, variance known from instrument precision / problem statement.

Typical mistake

• Estimating $\sigma$ from the same sample and still quoting exact $N(0,1)$ quantiles without naming that you changed the pivot (Student-$t$ is beyond this chapter’s Acciaio treatment).

10.5 Hypothesis tests #

10.5.1 Setup: $H_0$, $H_1$, decision, errors #

Framework (simple hypotheses) $X_1,\ldots,X_n$ i.i.d. with parameter $\theta$. Consider $ H_0\colon\theta=\theta_0,\qquad H_1\colon\theta=\theta_1 $ (simple = each hypothesis pins down the law completely). German: Nullhypothese $H_0$, Alternativhypothese $H_1$.

Definition (test / decision function) A test is a function $d(x)\in\{0,1\}$:

• $d(x)=0$: accept $H_0$,
• $d(x)=1$: reject $H_0$ (decide for $H_1$).

The set $\{x:d(x)=1\}$ is the critical region (Ablehnungsbereich).

Definition (Type I / Type II, level, power)

Notation $\mathbb{P}_{H_0}$ means the law under $\theta=\theta_0$ (similarly $\mathbb{P}_{H_1}$).

Principles

• Primary goal: keep Type I error $\alpha$ small (choose level first).
• Secondary goal: among level-$\alpha$ tests, maximize power (Neyman–Pearson).

When to use

• Exam states two concrete parameter values and asks to accept/reject, or to compute $\alpha$ and power for a given critical region.
• You must decide which hypothesis is $H_0$: put the “dangerous false rejection” under Type I. Equivalently: put what you want to prove in $H_1$.
• Composite one-sided Bernoulli/Binomial at large $n$ with the CLT: §10.5.5.

Typical mistake

• Swapping $H_0$ and $H_1$, which swaps which error is called Type I.
• Confusing power $1-\beta$ with level $\alpha$.
• Thinking “accept $H_0$” means $H_0$ is proved — it only means the data were not extreme enough under the chosen level.

Decision vs truth (PVK-style table):

10.5.2 Likelihood-ratio tests #

Definition (likelihood ratio) For discrete models (continuous: replace probabilities by densities), $ r(x)=\frac{\mathbb{P}_{H_0}[X=x]}{\mathbb{P}_{H_1}[X=x]}=\frac{L_x(\theta_0)}{L_x(\theta_1)}. $ Fix a threshold $c>0$ and set $ d(x)=\begin{cases} 0 & \text{if }r(x)>c\quad\text{(accept }H_0\text{)},\\ 1 & \text{if }r(x)\le c\quad\text{(reject }H_0\text{)}. \end{cases} $ Then $ \alpha=\mathbb{P}_{H_0}[r(X)\le c],\qquad 1-\beta=\mathbb{P}_{H_1}[r(X)\le c]. $ In discrete problems one often chooses the largest critical region of the form $\{r\le c\}$ (equivalently a threshold on a sufficient statistic such as $\|x\|$) so that $\alpha$ does not exceed a target.

Theorem (Neyman–Pearson lemma — statement) For two simple hypotheses, among all tests with Type I error at most $\alpha$, a likelihood-ratio test of that level maximizes the power $1-\beta$.

(Proof idea in Acciaio: compare any competitor $d^*$ to the LR test $d$ by checking the pointwise inequality involving $(c\,L_1-L_0)$ and taking expectations under $H_0$ and $H_1$.)

When to use

• Both hypotheses are simple and fully specify $L_x(\theta_0)$ and $L_x(\theta_1)$.
• Exam asks for Likelihood-Quotienten-Test / to tabulate $r$ vs a count statistic.

Typical mistake

• Writing $r=L_1/L_0$ and then using the wrong reject region (must match your definition — Acciaio rejects when $r=L_0/L_1$ is small).
• Computing $\alpha$ under $H_1$ or power under $H_0$.

10.5.3 Likelihood table (Eddy’s numerical ratios) #

This is the table used in the example above: each column is one $k$, and you sum a row over the critical $k$ to get $\alpha$ (top row) or power (middle row). Acciaio lists likelihoods as functions of $k=\|x\|$ (heads in 10 tosses). Under $H_0$ ($p=0.7$) and $H_1$ ($p=0.5$):

Reading the table: $r$ crosses $1$ between $k=6$ and $k=7$, and crosses $0.1$ between $k=3$ and $k=4$. That is why the reject regions were $\{\|X\|\le 6\}$ and $\{\|X\|\le 3\}$.

PVK operational tip: build exactly this three-row table (pmf under $H_0$, under $H_1$, ratio), then pick the largest $c$ (or the corresponding $k$-cutoff) compatible with a target $\alpha$.

10.5.4 Neyman–Pearson — proof sketch (optional depth) #

Score. Write $L_0(x)=\mathbb{P}_{H_0}[X=x]$ and $L_1(x)=\mathbb{P}_{H_1}[X=x]$. Reject $H_0$ on $x$ when the alternative “pays” at least $c$ times the Type I cost:

So $d(x)=1$ exactly on the nonnegative-score outcomes — the same rule as §10.5.2.

Toy picture (three outcomes). $L_0$ and $L_1$ are two pmfs. LR with threshold $c=1$ rejects only the smallest-$r$ point $C$:

LR spends $\alpha=L_0(C)=0.10$ and gets power $L_1(C)=0.60$. A competitor that rejects $A$ instead spends $\alpha=0.50$ for power $0.10$; one that rejects $B$ spends $0.40$ for power $0.30$. Both pay more Type I for less power — they spent the budget on worse $r$.

Why $d$ beats any competitor $d^*$ (Acciaio). Let $d$ be the LR test of threshold $c$, with level $\alpha=\mathbb{E}_{H_0}[d(X)]$ (this is $\mathbb{P}_{H_0}[\text{reject}]$). Let $d^*$ be any other $0$–$1$ test with $\mathbb{E}_{H_0}[d^*(X)]\le\alpha$. Write $s(x)=c\,L_1(x)-L_0(x)$.

• If $s(x)\ge 0$, LR already set $d(x)=1$, and $d^*(x)\le 1$, so $d^*(x)\,s(x)\le d(x)\,s(x)$.
• If $s(x)<0$, LR already set $d(x)=0$, and $d^*(x)\ge 0$, so $d^*(x)\,s(x)\le 0=d(x)\,s(x)$.

That is the pointwise inequality $ d^*(x)\bigl(c\,L_1(x)-L_0(x)\bigr) \le d(x)\bigl(c\,L_1(x)-L_0(x)\bigr). $ Sum over $x$: $c\,\mathbb{E}_{H_1}[d^*]-\mathbb{E}_{H_0}[d^*]\le c\,\mathbb{E}_{H_1}[d]-\mathbb{E}_{H_0}[d]$. Rearrange, using $\mathbb{E}_{H_0}[d]=\alpha\ge\mathbb{E}_{H_0}[d^*]$ and $\mathbb{E}_{H_1}[d]=\text{power}$: $ \underbrace{\mathbb{E}_{H_1}[d(X)]-\mathbb{E}_{H_1}[d^*(X)]}_{\text{power}(d)-\text{power}(d^*)} \ge \frac{1}{c}\bigl(\underbrace{\mathbb{E}_{H_0}[d(X)]-\mathbb{E}_{H_0}[d^*(X)]}_{\alpha-\text{level}(d^*)}\bigr)\ge 0. $ Hence the LR test has at least as much power as $d^*$.

When to use

• Exam asks to state Neyman–Pearson or to justify why an LR critical region is optimal.
• Full reproduction of the inequality chain is rare; knowing the statement + “LR tests are MP at fixed $\alpha$” is the practical takeaway.

Typical mistake

• Claiming NP applies unchanged to composite hypotheses ($H_0\colon\theta\le\theta_0$, etc.) without extra structure — this chapter’s lemma is for two simple hypotheses. For one-sided composite Bernoulli tests at large $n$, use the CLT at the boundary instead (§10.5.5).

10.5.5 Composite one-sided tests via CLT (exam pattern) #

Rule (what goes in $H_1$) The statement you are trying to establish is $H_1$. $H_0$ is the opposite (the status you will keep unless the data are extreme). Type I = wrongly celebrating $H_1$.

Rule (composite $\Rightarrow$ boundary) If $H_0\colon p<0.9$ and you reject for large success counts, the Type I probability is largest at $p=0.9$. It is enough to enforce $\mathbb{P}_{p=0.9}[\text{reject}]\le\alpha$; smaller $p$ makes large counts even less likely.

Rule (one-sided quantile) For a two-sided $95\%$ CI one uses $c=\Phi^{-1}(0.975)\approx 1.96$. For a one-sided $5\%$ test one uses $\Phi^{-1}(0.95)\approx 1.65$ (same number as the $90\%$ two-sided $c$ in §10.4).

When to use

• “Behauptung statistisch beweisen”, Signifikanzniveau $5\%$, $n$ large, Bernoulli/Binomial, “verwende den ZGWS”.
• You must choose $H_0$/$H_1$ and then give a decision rule in the count $k$.

Typical mistake

• Putting the claim you want to prove in $H_0$ (then rejecting $H_0$ “proves” the wrong thing).
• Using $c=1.96$ on a one-sided $5\%$ test.
• Computing the Binomial tail at some $p$ inside $H_1$ instead of at the $H_0$ boundary.

10.6 Method map #

10.7 Exercises #

Exercise 1. A sensor reports success/fail. You observe $x=(1,1,0,1,0,0)$ and model the trials as i.i.d. Bernoulli($p$).

(a) Write $L_x(p)$ and the MLE $\widehat{p}$.

(b) Someone suggests reporting the 95% interval $[\overline{x}\pm 1.96/\sqrt{n}]$ as an exact CI for $p$ from this chapter. Is that the right tool? What would be the right tool if the exam instead asked you to decide between $p=1/2$ and $p=3/4$?

Solution. Goal: Compute the Bernoulli MLE, then choose the matching later tool (CI vs test). Step 1: $n=6$, $\|x\|=3$, so $L_x(p)=p^3(1-p)^3$ and $\widehat{p}=\overline{x}=1/2$. Step 2 (b, CI): No. The exact CI in §10.4 is for i.i.d. $\mathcal{N}(m,\sigma_0^2)$ with $\sigma_0$ known. A Bernoulli $p$ is not a Normal mean. For large $n$ one may approximate via the CLT; $n=6$ is not that, and this chapter does not give an exact $N(0,1)$ CI for $p$. Step 3 (b, test): Deciding between two concrete values $p=1/2$ and $p=3/4$ is a hypothesis test: write $r=L_x(1/2)/L_x(3/4)$ (or the other way around, matching your $H_0$) and reject when $r$ is small — §10.5. Check: Same $L_x$ for (a) and (b); the job (maximize vs wrap an interval vs compare two points) picks the tool.

Exercise 2. Normal model with known $\sigma^2=1$. How large must $n$ be so that the half-width of the 95%-CI is at most $0.1$? (Use $c=1.96$.)

Solution. Goal: Minimal $n$ with 95%-CI half-width $\le 0.1$ when $\sigma^2=1$. Step 1: the interval is $[\overline{x}\pm 1.96/\sqrt{n}]$. Step 2: require

Step 3: since $n$ is an integer, take $n=385$. Check: $1.96/\sqrt{385}\approx 0.0999\le 0.1$, while $n=384$ is still slightly above $0.1$. This is the same tradeoff as in the 90% vs 99% example: to keep coverage and shrink the interval you buy more data, not a different $c$.

Exercise 3. Let $X_1,X_2,X_3$ be i.i.d. with density $f_\theta(x)=\theta x^{\theta-1}$ on $(0,1)$, $\theta>0$. Observed $x=(e^{-1},e^{-1},e^{-2})$. Find the MLE $\widehat{\theta}$.

Solution. Goal: MLE for $f_\theta(x)=\theta x^{\theta-1}$ on $(0,1)$, then a number from the sample. Step 1: joint density / likelihood:

Step 2: $\ell(\theta)=3\log\theta+(\theta-1)\sum_{i=1}^{3}\log x_i$. Then $\ell'(\theta)=3/\theta+\sum\log x_i=0$ gives the general formula

Step 3: $\log x=(-1,-1,-2)$, so $-\sum\log x_i=4$ and

Check: $\ell''(\theta)=-3/\theta^2<0$, so a maximum. Each $x_i\in(0,1)$ makes $-\log x_i>0$, so $\widehat{\theta}>0$ automatically.

End of Chapter 10 draft. Sources: Acciaio lecture Ch. 4 (estimation, CI, tests / Neyman–Pearson / Eddy); PVK §5 Statistik (MLE checklist, CI table, error table).