12  Conjugate Pairs: Conclusion

In Bayesian stats, we update our belief about a parameter based on observed data using the Bayes theorem. This is done by multiplying the prior distribution (initial belief) by the likelihood function (observation) to get the posterior distribution (updated belief). The math for this can sometimes be complex due to the denominator in Bayes theorem that requires us to look at all possible outcomes.

To overcome this we have conjugate priors. A conjugate prior is an analytical shortcut that allows us to directly find the posterior distribution without having to do any complicated math. A conjugate prior is a special type of prior distribution that when combined with a specific likelihood function, results in a posterior distribution that belongs to the same family of distributions as the prior.

We saw two conjugate pairs in our previous examples and saw how the posterior can be directly found using the shortcut:

Model parameter Conjugate prior Likelihood Prior hyperparameters Posterior hyperparameters
p (probability) Beta Bernoulli \(\alpha , \beta\) \(\alpha + k\), \(\beta + n-k\)
p (probability) Beta Binomial \(\alpha , \beta\) \(\alpha + k\), \(\beta + n-k\)
\(\lambda\) (rate) Gamma Poisson \(\alpha , \beta\) \(\alpha + \sum x_i\), \(\beta + n\)
\(\sigma^2 (variance)\) Inverse Gamma Normal \(\alpha , \beta\) \(\alpha + n/2\), \(\beta\) + Σ(xᵢ - μ)²/2
\(\mu (mean)\) Normal Normal \(\mu_0 , \tau_0\) \(\tau_n = \tau_0 + n*\tau, \mu_n = \frac{\tau_0\mu_0+\tau\sum x_i}{\tau_0 + n*\tau}\)

To see more conjugate pairs I highly recommend Wikipedia which has a nice article on conjugate distributions with a big list of conjugate pairs that’s worth checking out!