Part B Foundations of Statistical Inference

This is a course on advanced statistical inference for 3rd year students studying Mathematics and Statistics at Oxford.

Course aims 

Understanding how data can be interpreted in the context of a statistical model. Working knowledge and understanding of key-elements of model-based statistical inference, including awareness of similarities, relationships and differences between Bayesian and frequentist approaches.

Synopsis

Exponential families: Curved and linear exponential families; canonical parametrization; likelihood equations. Sufficiency: Factorization theorem; sufficiency in exponential families.

Frequentist estimation: unbiasedness; method of moments; the Cramer-Rao information inequality; Rao-Blackwell theorem, Lehmann-Scheffe Theorem and Rao-Blackwellization.
Statement of complete sufficiency for Exponential families.

The Bayesian paradigm: likelihood principal; subjective probability; prior to posterior analysis; asymptotic normality; conjugacy; examples from exponential families. Choice of prior distribution: proper and improper priors; Jeffreys and maximum entropy priors. Hierarchical Bayes models.

Computational techniques: Markov chain Monte Carlo methods; The Metropolis-
Hastings algorithm. Gibbs Sampling. Variational Bayesian methods. The EM
algorithm. Approximations to marginal likelihood : Laplace approximation and
BIC.

Decision theory: risk function; Minimax rules, Bayes rules. Point estimators and admissability of Bayes rules. The James-Stein estimator, shrinkage estimators and Empirical Bayes. Hypothesis testing as decision problem.

Lecture slides

PDF of all 16 lectures in 4up format : bs2a_4up.pdf

Problem sheets 

Class
Sheet
Week 3
ex1.pdf
Week 4
ex2.pdf
Week 5
ex3.pdf
Week 6
ex4.pdf
Week 7
ex5.pdf
Week 8
ex6.pdf

Reading

  1. P. H. Garthwaite, I. T. Jolliffe and Byron Jones, Statistical Inference, Second ed. Oxford University Press, 2002
  2. G.A.Young and R.L. Smith,  Essentials of Statistical Inference, Cambridge University  Press, 2005. 
  3. T. Leonard and J.S.J. Hsu, Bayesian Methods, Cambridge University Press, 2005.

Further reading

  1. D. R. Cox, Principles of Statistical Inference, Cambridge University Press, 2006
  2. H. Liero and S Zwanzig, Introduction to the Theory of Statistical Inference, CRC Press,  2012
  3. D. Barber, Bayes Reasoning and Machine Learning, Cambridge University Press,
    2012