Course Notebooks

Simple 2-component Gaussian mixture model showcasing identifiability and label-switching problems without ordering, then solved with ordered transform. Uses NormalMixture…

2D Gaussian mixtures with custom logp, Metropolis sampling, and mini-batch ADVI.

Gaussian Mixture Models with ADVI

Detailed 2D Gaussian mixture model with marginalization and ADVI. Covers custom logp via DensityDist, MCMC sampling, and ADVI with mini-batching using multivariate normals…

The log-sum-exp trick for marginalizing out cluster assignments in mixture models.

Marginalizing Over Discrete Variables

Demonstrates how to marginalize over discrete cluster assignments in Gaussian mixture models using the log-sum-exp trick. Covers NormalMixture, custom DensityDist with…

Gaussian mixtures, label-switching, and identifiability in MCMC sampling.

Mixture Models and MCMC

Explores Gaussian mixture models with MCMC sampling using Old Faithful data and synthetic 3-component mixtures. Demonstrates label-switching problems, identifiability…

From CAVI to scalable black-box VI with automatic differentiation.

Automatic Differentiation Variational Inference

Covers the progression from CAVI to stochastic CAVI to ADVI, including transformation to unconstrained space, automatic differentiation for gradient computation, and…

Generative modeling with VAEs on MNIST using PyTorch.

Variational Autoencoder

Implements a variational autoencoder (VAE) in PyTorch for the MNIST dataset, covering the encoder-decoder architecture, the reparameterization trick, and the ELBO loss…

Bayesian neural network classification using ADVI in PyMC.

Variational Inference with Neural Networks

classification

Builds a Bayesian neural network for binary classification on the moons dataset using variational inference (ADVI) in PyMC, demonstrating uncertainty quantification in…

Approximating posteriors through optimization instead of sampling.

Variational Inference

Introduction to variational inference: the core idea of turning Bayesian inference into optimization, KL divergence, the ELBO, and coordinate ascent variational inference…

Prior selection, simulation-based calibration, and posterior predictive checks for GLMs.

Bayesian Workflow: Zero-Inflated Poisson

From multivariate normals to kernel-based function priors and predictive distributions.

A complete Bayesian workflow applied to a zero-inflated Poisson model of monks producing manuscripts. Covers prior predictive checks, simulation-based calibration…

Jun 11, 2025

The Idea Behind the Gaussian Process

Starting from multivariate Gaussian marginals and conditionals, we build intuition for Gaussian Processes by constructing covariance matrices from kernels, drawing prior…

From kernel trick to infinite basis functions — the theory behind GP regression.

Gaussian Processes and Non-parametric Bayes

gaussian-processes

We develop the theoretical foundations of Gaussian Processes: the dual representation of Bayesian linear regression, kernel tricks, Mercer’s theorem, infinite basis function…

Hyperparameter learning via empirical Bayes and MCMC with PyMC.

Inference for Gaussian Processes

We tackle GP hyperparameter inference using the marginal likelihood: both Type-2 MLE (empirical Bayes) via optimization and full Bayesian MCMC via PyMC. The sockeye salmon…

Gaussian process covariance captures spatial correlation between island societies.

Geographic Correlation and Oceanic Tools

We extend the oceanic tools Poisson regression with a custom GP covariance matrix that models geographic correlation between societies. The inverse-square distance kernel…

A condensed recap of GP theory applied to sockeye salmon recruitment.

GP Recap and Salmon Example

gaussian-processes

A concise review of Gaussian Process fundamentals — MVN conditionals, kernel covariance, and GP regression — applied to the sockeye salmon stock-recruitment dataset with…

WAIC model comparison, ensemble averaging, and varying-intercepts for overdispersed counts.

Poisson Regression — Model Comparison and Hierarchical Overdispersion

We return to the oceanic tools dataset to illustrate model comparison using WAIC, model averaging with Akaike weights, counterfactual posterior predictives, and fighting…

Centering, correlations, and counterfactual posteriors in Oceanic tool kit evolution.

Poisson Regression: Modeling Tool Diversity Across Islands

Analyzes technological diversity across Oceanic island societies using Poisson regression with interaction terms. Shows how parameter centering reduces posterior correlation…

May 21, 2025

Choosing Priors: From Flat to Weakly Informative

Uninformative, Jeffreys, and regularizing priors for Bayesian inference.

Explores how to choose priors that are robust to transformation and prevent overfitting without strongly biasing inference. Covers flat, improper, Jeffreys, and weakly…

May 21, 2025

Bayesian Regression with Normal Models

From heights to height-weight relationships using conjugate priors and posterior predictives.

Demonstrates Bayesian regression by modeling height using normal distributions and PyMC. Covers posterior inference, the identifiability problem in regression parameters…

May 21, 2025

Identifiability in Bayesian Models

How parameter redundancy breaks MCMC convergence — and how priors can help.

Explores how unidentifiable parameters cause MCMC samplers to fail, even when derived quantities converge perfectly. Demonstrates convergence diagnostics that reveal the…

May 14, 2025

Gelman Schools and Hierarchical Pathology

Diagnosing and fixing funnel pathology in hierarchical models with reparameterization.

hierarchical

Learning to pool information across groups with non-centered parametrization.

Explores the 8-schools hierarchical model to reveal how centered parametrizations create pathological funnel structures that trap HMC samplers. Shows how non-centered…

May 7, 2025

Hierarchical Bayesian Modeling: The 8 Schools Example

Explores the Gelman Schools hierarchical normal-normal model, a foundational example in Bayesian statistics for estimating treatment effects across multiple groups.…

May 7, 2025

HMC/NUTS Tuning and Diagnostics

Why centered models diverge and non-centered models don’t — a hands-on HMC exploration.

Explores HMC/NUTS sampling tuning through hierarchical Bayesian models, demonstrating how parametrization, step-size adaptation, and leapfrog length affect sampling…

May 7, 2025

Data Augmentation

Iterative sampling via hidden variables using the Tanner-Wong algorithm and Gibbs.

From typical sets to leapfrog integration: a detailed exploration of HMC and NUTS.

Introduces data augmentation as iterative sampling via hidden variables. Contrasts the stochastic Tanner-Wong version with Gibbs sampling, showing how augmenting with latent…

Apr 30, 2025

Exploring Hamiltonian Monte Carlo

Gibbs sampling on a bimodal, highly-correlated posterior with lots of autocorrelation.

Detailed exploration of Hamiltonian Monte Carlo following Neal and Betancourt. Covers typical set geometry, gradient-based smooth exploration, leapfrog integration, and the…

Apr 30, 2025

A Tetchy Gibbs Sampler

Exploiting conjugate prior pairs for efficient Gibbs sampling.

Demonstrates Gibbs sampling on a bimodal, highly-correlated posterior distribution. Shows that conditional posteriors are Gaussian despite an ugly joint form, and…

Apr 30, 2025

Gibbs Sampling with Conjugate Conditionals

Demonstrates Gibbs sampling that exploits conjugate prior pairs like Beta-Binomial. Shows how conditional posteriors simplify to known distributions, making each Gibbs step…

Sampling from complex distributions with asymmetric proposals and limited support.

The Metropolis-Hastings Algorithm

Extends the basic Metropolis algorithm to handle asymmetric proposal distributions and limited support domains. Demonstrates MH with a Weibull-like distribution using a…

Why combining rejection with Metropolis sampling wastes efficiency.

Metropolis and Support Mismatch

Demonstrates the efficiency loss when combining rejection sampling with Metropolis by using a proposal distribution with larger support than the target. Shows that a…

A coal-mine disaster switchpoint model for convergence testing and posterior predictive checks.

Imputation and Convergence

Uses a coal-mine disaster switchpoint model to illustrate MCMC convergence testing and posterior predictive imputation. Detects a regulatory change around 1900 with a…

Metropolis sampling from discrete distributions using proposal matrices.

Discrete MCMC

Demonstrates how to use Metropolis MCMC to sample from discrete distributions. Shows that MCMC handles discrete state spaces using proposal matrices that satisfy detailed…

MCMC via alternating conditional sampling from a complex joint distribution.

Introduction to Gibbs Sampling

Gibbs sampling enables sampling from complex joint distributions by alternately sampling from univariate conditionals. Demonstrates the method through a 2D example, deriving…

Stationarity, reversibility, and the mathematical foundation for Metropolis sampling.

Markov Chains and MCMC

Introduces Markov chains as the mathematical foundation for MCMC sampling. Covers stationarity, ergodicity, detailed balance, and the Metropolis algorithm, with a practical…

Understanding detailed balance and the Metropolis algorithm for sampling probability distributions.

From Annealing to Metropolis

Explores the Metropolis algorithm as a method for sampling from probability distributions. Connects detailed balance from simulated annealing to the Metropolis-Hastings…

From partial pooling to shrinkage estimation with the rat tumors example.

Hierarchical Bayesian Models

hierarchical

Inferring parameters from incomplete data through iterative expectation and maximization.

Introduces hierarchical Bayesian models as a framework for estimating parameters across related units by borrowing statistical strength from collective data. Demonstrates…

Apr 9, 2025

The EM Algorithm

Understanding discriminative boundaries vs. generative class models through logistic regression and LDA.

Introduces the Expectation-Maximization algorithm for maximum likelihood estimation with missing or latent variables. Starts with estimating bivariate normal parameters from…

Apr 2, 2025

Generative vs Discriminative Models

classification

From generative classifiers to clustering: supervised, unsupervised, and semi-supervised approaches.

Explores Empirical Risk Minimization versus Bayesian learning, comparing discriminative and generative models. Uses a height/weight classification task to contrast logistic…

Mar 26, 2025

Mixture Models and Types of Learning

classification

From utility functions to Bayes actions: applying decision theory to regression and model comparison.

Covers the three fundamental types of machine learning through the lens of Gaussian Mixture Models. Demonstrates how GDA works in supervised settings, contrasts with…

Mar 26, 2025

Utility, Risk, and Decision Theory

decision-theory

Building and training multi-layer perceptrons as universal function approximators.

Covers decision theory fundamentals, explaining how to formulate and solve optimal decision problems using Bayesian methods. Demonstrates through a stock market prediction…

Mar 26, 2025

Regression in PyTorch

Training neural networks to classify synthetic data with decision boundaries.

Demonstrates the Universal Approximation Theorem by implementing MLPs in PyTorch to fit complex 1D functions. Trains neural networks with ReLU and tanh activations using…

Mar 19, 2025

Multi-Layer Perceptron for Classification

Visualizing neural network function approximation through hidden layer activations.

Implements multi-layer perceptrons for classification tasks using PyTorch. Generates synthetic datasets and trains networks to learn non-linear decision boundaries…

Mar 19, 2025

How Sigmoids Combine

Optimization by following the slope downhill.

visualization

Explores how sigmoid activation functions combine in MLPs to create increasingly complex function approximations. Trains neural networks with varying architectures —…

Mar 19, 2025

Gradient Descent and SGD

From MLE classification to computing gradients through layers.

Introduces gradient descent for optimization, starting with linear regression as a worked example. Covers batch gradient descent, stochastic gradient descent, mini-batch…

Mar 12, 2025

Logistic Regression and Backpropagation

Putting priors on regression coefficients and updating with data.

Develops logistic regression as a maximum likelihood classification model, then introduces backpropagation as the algorithm for computing gradients through layered…

Mar 12, 2025

Bayesian Regression

Building a Bayesian regression from Kung San census data.

Formulates linear regression in the Bayesian framework with conjugate normal priors on the coefficients. Derives the posterior distribution over weights, shows how the…

Mar 5, 2025

From the Normal Model to Regression

Walks through the progression from a normal model to Bayesian linear regression using Howell’s Kung San height and weight data. Implements grid approximation and PyMC3 MCMC…

Mar 5, 2025

Lab: The Beta-Binomial Globe Model

Grid approximation, quadratic approximation, and MCMC in practice.

Hands-on lab working through Bayesian inference on the globe-tossing model. Implements grid approximation, Laplace (quadratic) approximation, conjugate prior updates, and…

Sample where it matters most to compute integrals efficiently.

Importance Sampling

montecarlo

integration

Introduces importance sampling as a method for computing expectations and integrals by reweighting samples from a proposal distribution. Shows how choosing a good proposal…

Turning uniform random numbers into any distribution you want.

The Inverse Transform

Introduces the inverse transform method for generating samples from arbitrary distributions using uniform random variables and the inverse CDF. Includes the Box-Muller…

Conjugate priors for the workhorse of statistics.

The Normal Model

Derives the normal-normal conjugate model for Bayesian inference on the mean of a Gaussian with known variance. Shows how the posterior interpolates between prior belief and…

Accept or reject: a simple algorithm for hard distributions.

Rejection Sampling

montecarlo

Presents von Neumann’s rejection sampling algorithm for drawing samples from distributions with known functional form. Covers the basic method and an enhanced version that…

Treating parameters as random variables changes everything.

Bayesian Statistics

Introduces Bayesian inference by contrasting it with the frequentist approach. Develops the normal-normal conjugate model, derives the MAP estimate, and shows how MCMC…

Feb 19, 2025

The Beta-Binomial Globe Model

Conjugate priors, Bayesian updating, and decision theory on a globe toss.

When less data tells you everything, and order doesn’t matter.

Develops the beta-binomial conjugate model for estimating the fraction of water on a globe. Covers prior selection, sequential Bayesian updating, loss functions, decision…

Feb 19, 2025

Sufficient Statistics and Exchangeability

Quantifying uncertainty and the distributions it favors.

Connects the exponential family to conjugate priors through sufficient statistics. Introduces exchangeability and de Finetti’s theorem, then develops the Poisson-Gamma…

Feb 19, 2025

Entropy and Maximum Entropy

information-theory

Measuring how far your model is from the truth.

Introduces information entropy as a measure of uncertainty and derives maximum entropy distributions. Shows how constraints on moments lead naturally to the exponential…

Feb 12, 2025

Divergence and Deviance

information-theory

Develops KL divergence as a distance between distributions, connects it to cross-entropy and log-likelihood, and introduces deviance as a practical measure for model…

When a drug works, how much does it matter?

The Significance and Size of Effects

Explores significance testing and effect size using a drug vs. placebo experiment. Demonstrates permutation tests, bootstrap confidence intervals, and Cohen’s d as…

Why the average of a function isn’t the function of the average.

Convexity and Jensen’s Inequality

Introduces convex functions and proves Jensen’s inequality. Shows how convexity underpins key results in information theory and probabilistic modeling.

An information-theoretic shortcut for model selection.

Understanding AIC

information-theory

Explains the Akaike Information Criterion from first principles: how deviance, effective parameters, and information theory combine to estimate out-of-sample predictive…

What happens when you fit a model to perfect data.

Learning Without Noise

Introduces the learning problem through a noiseless example: fitting polynomial models to data, measuring in-sample vs out-of-sample error, and understanding how hypothesis…

Bias, variance, and the tradeoff that haunts every model.

Learning With Noise

Extends the learning framework to noisy data, where stochastic noise forces a tradeoff between model complexity and generalization. Derives the bias-variance decomposition…

Taming complexity by penalizing parameters.

Regularization

Introduces regularization as an alternative to model selection for controlling overfitting. Compares Ridge and Lasso regression, showing how a penalty term on coefficients…

How to honestly evaluate what your model has learned.

Learning Bounds and the Test Set

Formalizes out-of-sample error estimation using Hoeffding’s inequality and the train/test split. Walks through scikit-learn’s data pipeline for polynomial regression…

Why one split is never enough.

Validation and Cross-Validation

Explains why a separate validation set is needed to choose model complexity, and how cross-validation averages over multiple splits to give a more robust estimate of…

Find the parameters that make your data most probable.

Maximum Likelihood Estimation

Let randomness do the heavy lifting.

Develops maximum likelihood estimation from first principles, deriving MLEs for the exponential, linear regression, and logistic regression models. Shows how MLE connects…

Jan 22, 2025

Basic Monte Carlo

montecarlo

Introduces Monte Carlo methods by estimating the area of a unit circle with random points. Covers the hit-or-miss method and rejection sampling as building blocks for Monte…

What you expect is what you get — eventually.

Expectations and the Law of Large Numbers

Defines expected values, the Law of the Unconscious Statistician (LOTUS), and variance for discrete and continuous distributions. Demonstrates the Law of Large Numbers and…

When calculus is hard, sample instead.

Monte Carlo Integration

Formalizes Monte Carlo integration using uniform random samples and LOTUS, then extends to multidimensional integrals. Uses the Central Limit Theorem to estimate integration…

Why everything looks normal in the limit.

Sampling and the Central Limit Theorem

Builds sampling distributions from repeated coin-flip experiments, showing how the sample mean and its variance behave as sample size grows. Derives the Central Limit…

Simulating a presidential election with coin flips.

Distributions Example: Elections

From coin flips to Bayes’ theorem.

elections

Simulates the 2012 US presidential election using Bernoulli coin flips for each state based on PredictWise probabilities. Introduces the Binomial distribution, the CLT’s…

Jan 8, 2025

Probability