[ { "path": ".gitignore", "content": "*.log\n*.toc\n*.aux\n*.idx\n*.out\n*.synctex.gz" }, { "path": "README.md", "content": "Machine learning cheat sheet\n============================\n\nThis cheat sheet contains many classical equations and diagrams on machine learning, which will help you quickly recall knowledge and ideas on machine learning.\n\nThe cheat sheet will also appeal to someone who is preparing for a job interview related to machine learning.\n\n\n## Download PDF\n[machine-learning-cheat-sheet.pdf](https://github.com/soulmachine/machine-learning-cheat-sheet/raw/master/machine-learning-cheat-sheet.pdf) \n\n\n## How to compile\n\n docker pull soulmachine/texlive\n docker run -it --rm -v $(pwd):/data -w /data soulmachine/texlive xelatex -synctex=1 --enable-write18 -interaction=nonstopmode machine-learning-cheat-sheet.tex\n\n\n## LaTeX template\nThis open-source book adopts the [Springer latex template](http://www.springer.com/authors/book+authors?SGWID=0-154102-12-970131-0).\n\n\n## How to compile on Windows\n1. Install [Tex Live 2014](http://www.tug.org/texlive/), then add its `bin` path for example `D:\\texlive\\2012\\bin\\win32` to he PATH environment variable.\n2. Install [TeXstudio](http://texstudio.sourceforge.net/).\n3. Configure TeXstudio. \n Run TeXstudio, click `Options-->Configure Texstudio-->Commands`, set `XeLaTex` to `xelatex -synctex=1 -interaction=nonstopmode %.tex`.\n \n Click `Options-->Configure Texstudio-->Build`, \n set `Build & View` to `Compile & View`, \n set `Default Compiler` to `XeLaTex`, \n set `PDF Viewer` to `Internal PDF Viewer(windowed)`, so that when previewing it will pop up a standalone window, which will be convenient.\n4. Compile. Use Open `machine-learning-cheat-sheet.tex` with TeXstudio，click the green arrow on the menu bar, then it will start to compile. \n In the messages window below we can see the compilation command that TeXstudio is using is `xelatex -synctex=1 --enable-write18 -interaction=nonstopmode \"machine-learning-cheat-sheet\".tex`\n" }, { "path": "acknow.tex", "content": "%%%%%%%%%%%%%%%%%%%%%%acknow.tex%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n% sample acknowledgement chapter\n%\n% Use this file as a template for your own input.\n%\n%%%%%%%%%%%%%%%%%%%%%%%% Springer %%%%%%%%%%%%%%%%%%%%%%%%%%\n\n\\extrachap{Acknowledgements}\n\nUse the template \\emph{acknow.tex} together with the Springer document class SVMono (monograph-type books) or SVMult (edited books) if you prefer to set your acknowledgement section as a separate chapter instead of including it as last part of your preface.\n\n" }, { "path": "acronym.tex", "content": "%%%%%%%%%%%%%%%%%%%%%%acronym.tex%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n% sample list of acronyms\n%\n% Use this file as a template for your own input.\n%\n%%%%%%%%%%%%%%%%%%%%%%%% Springer %%%%%%%%%%%%%%%%%%%%%%%%%%\n\n\\extrachap{Acronyms}\n\nUse the template \\emph{acronym.tex} together with the Springer document class SVMono (monograph-type books) or SVMult (edited books) to style your list(s) of abbreviations or symbols in the Springer layout.\n\nLists of abbreviations\\index{acronyms, list of}, symbols\\index{symbols, list of} and the like are easily formatted with the help of the Springer-enhanced \\verb|description| environment.\n\n\\begin{description}[CABR]\n\\item[ABC]{Spelled-out abbreviation and definition}\n\\item[BABI]{Spelled-out abbreviation and definition}\n\\item[CABR]{Spelled-out abbreviation and definition}\n\\end{description}" }, { "path": "cblist.tex", "content": "%%%%%%%%%%%%%%%%%%%%clist.tex %%%%%%%%%%%%%%%%%%%%%%%%\n% \n% sample list of contributors and their addresses \n% \n% Use this file as a template for your own input. \n% \n%%%%%%%%%%%%%%%%%%%%%%%% Springer %%%%%%%%%%%%%%%%%%%%\n\\contributors\n\n\\begin{thecontriblist}\nWei Zhang\n\\at PhD candidate at the Institute of Software, Chinese Academy of Sciences (ISCAS), Beijing, P.R.CHINA, \\email{zh3feng@gmail.com}, has written chapters of Naive Bayes and SVM.\n\\and\nFei Pan\n\\at Master at Beijing University of Technology, Beijing, P.R.CHINA, \\email{example@gmail.com}, has written chapters of KMeans, AdaBoost.\n\\and\nYong Li\n\\at PhD candidate at the Institute of Automation of the Chinese Academy of Sciences (CASIA), Beijing, P.R.CHINA, \\email{liyong3forever@gmail.com}, has written chapters of Logistic Regression.\n\\and\nJiankou Li\n\\at PhD candidate at the Institute of Software, Chinese Academy of Sciences (ISCAS), Beijing, P.R.CHINA, \\email{lijiankoucoco@163.com}, has written chapters of BayesNet.\n\\end{thecontriblist}" }, { "path": "chapterABM.tex", "content": "\\chapter{Adaptive basis function models}\n\n\n\\section{AdaBoost}\n\n\n\\subsection{Representation}\n\\begin{equation}\ny=\\text{sign}(f(\\vec{x}))=\\text{sign}\\left(\\sum\\limits_{i=1}^m \\alpha_mG_m(\\vec{x})\\right)\n\\end{equation}\nwhere $G_m(\\vec{x})$ are sub classifiers.\n\n\n\\subsection{Evaluation}\n\\begin{equation} \\nonumber\nL(y,f(\\vec{x}))=\\exp[-yf(\\vec{x})] \\text{ i.e., exponential loss function}\n\\end{equation}\n\n\\begin{equation}\n(\\alpha_m,G_m(x))= \\arg\\min_{\\alpha,G} \\sum_{i=1}^N \\exp{[-y_i(f_{m-1}(\\vec{x}_i)+\\alpha G(\\vec{x}_i))]}\n\\end{equation}\n\nDefine $\\bar{w}_{mi}=\\exp{[-y_i(f_{m-1}(\\vec{x}_i)]}$, which is constant w.r.t. $\\alpha, G$\n\\begin{equation}\n(\\alpha_m,G_m(x))= \\arg\\min_{\\alpha,G} \\sum_{i=1}^N {\\bar{w}}_{mi} \\exp{(-y_i \\alpha G(x_i))}\n\\end{equation}\n\n\n\\subsection{Optimization}\n\n\n\\subsubsection{Input}\n\\begin{eqnarray*}\n& \\mathcal{D}=\\{(\\vec{x}_1,y_1),(\\vec{x}_2,y2),\\dots,(\\vec{x}_N,y_N)\\} \\\\\n& \\quad \\text{where } \\vec{x}_i \\in \\mathbb{R}^D,\\ y_i \\in \\{-1,+1\\} \\\\\n& \\text{ Weak classifiers } \\{G_1,G_2,\\dots,G_m\\}\n\\end{eqnarray*}\n\n\n\\subsubsection{Output}\nFinal classifier: $G(x)$\n\n\n\\subsubsection{Algorithm}\n\\begin{enumerate}\n\\item Initialize the weights' distribution of training data(when $m=1$)\n\\begin{equation}\\nonumber\n\\mathcal{D}_0=(w_{11},w_{12},\\cdots,w_{1n})=(\\frac{1}{N},\\frac{1}{N},\\cdots,\\frac{1}{N})\n\\end{equation}\n\\item Iterate over $m=1,2,\\dotsc,M$\n\\subitem (a) Use training data with current weights' distribution $\\mathcal{D}_m$ to get a classifier $G_m(\\vec{x})$\n\\subitem (b) Compute the error rate of $G_m(\\vec{x})$ over the training data \\\\\n\\begin{equation}\ne_m=P(G_m(\\vec{x}_i)\\neq y_i)=\\sum_{i=1}^N {w_{mi}\\mathbb{I}(G_m(\\vec{x}_i) \\neq y_i)}\n\\end{equation}\n\n\\subitem (c) Compute the coefficient of classifier $G_m(x)$\n\\begin{equation}\n\\alpha_m = \\frac{1}{2}\\log{\\frac{1-e_m}{e_m}}\n\\end{equation}\n\\subitem (d) Update the weights' distribution of training data\n\\begin{equation}\nw_{m+1,i}=\\frac{w_{mi}}{Z_m}\\exp(-\\alpha_m y_i G_m(\\vec{x}_i))\n\\end{equation}\nwhere $Z_m$ is the normalizing constant\n\\begin{equation}\nZ_m=\\sum_{i=1}^N w_{mi}\\exp(-\\alpha_m y_i G_m(\\vec{x}_i))\n\\end{equation}\n\n\\item Ensemble $M$ weak classifiers\n\\begin{equation}\nG(x)=\\text{sign}f(\\vec{x})=\\text{sign}\\left[\\sum_{m=1}^M \\alpha_m G_m(\\vec{x})\\right]\n\\end{equation}\n\\end{enumerate} \n\n\\subsection{The upper bound of the training error of AdaBoost}\n\\begin{theorem}\nThe upper bound of the training error of AdaBoost is \n\\begin{equation}\n\\frac{1}{N} \\sum_{i=1}^N \\mathbb{I}(G(\\vec{x}_i)\\neq y_i) \\leq \\frac{1}{N} \\sum_{i=1}^N \\exp(-y_i f(\\vec{x}_i))=\\prod_{m=1}^M Z_m\n\\end{equation}\n\nNote: the following equation would help proof this theorem\n\\begin{equation}\nw_{mi}\\exp(-\\alpha_m y_i G_m(\\vec{x}_i))=Z_m w_{m+1,i}\n\\end{equation}\n\\end{theorem}\n" }, { "path": "chapterBayesianStatistics.tex", "content": "\\chapter{Bayesian statistics}\n\n\n\\section{Introduction}\nUsing the posterior distribution to summarize everything we know about a set of unknown variables is at the core of Bayesian statistics. In this chapter, we discuss this approach to statistics in more detail.\n\n\n\\section{Summarizing posterior distributions}\nThe posterior $p(\\vec{\\theta}|\\mathcal{D})$ summarizes everything we know about the unknown quantities $\\vec{\\theta}$. In this section, we discuss some simple quantities that can be derived from a probability distribution, such as a posterior. These summary statistics are often easier to understand and visualize than the full joint.\n\n\n\\subsection{MAP estimation}\nWe can easily compute a \\textbf{point estimate} of an unknown quantity by computing the posterior mean, median or mode. In Section \\ref{sec:Bayesian-decision-theory}, we discuss how to use decision theory to choose between these methods. Typically the posterior mean or median is the most appropriate choice for a realvalued quantity, and the vector of posterior marginals is the best choice for a discrete quantity. However, the posterior mode, aka the MAP estimate, is the most popular choice because it reduces to an optimization problem, for which efficient algorithms often exist. Futhermore, MAP estimation can be interpreted in non-Bayesian terms, by thinking of the log prior as a regularizer (see Section TODO for more details).\n\nAlthough this approach is computationally appealing, it is important to point out that there are various drawbacks to MAP estimation, which we briefly discuss below. This will provide motivation for the more thoroughly Bayesian approach which we will study later in this chapter(and elsewhere in this book).\n\n\n\\subsubsection{No measure of uncertainty}\nThe most obvious drawback of MAP estimation, and indeed of any other \\emph{point estimate} such as the posterior mean or median, is that it does not provide any measure of uncertainty. In many applications, it is important to know how much one can trust a given estimate. We can derive such confidence measures from the posterior, as we discuss in Section \\ref{sec:Credible-intervals}.\n\n\\subsubsection{Plugging in the MAP estimate can result in overfitting}\nIf we don’t model the uncertainty in our parameters, then our predictive distribution will be overconfident. Overconfidence in predictions is particularly problematic in situations where we may be risk averse; see Section \\ref{sec:Bayesian-decision-theory} for details.\n\n\\subsubsection{The mode is an untypical point}\nChoosing the mode as a summary of a posterior distribution is often a very poor choice, since the mode is usually quite untypical of the distribution, unlike the mean or median. The basic problem is that the mode is a point of measure zero, whereas the mean and median take the volume of the space into account. See Figure \\ref{fig:untypical-point}.\n\n\\begin{figure}[hbtp]\n\\centering\n\\subfloat[]{\\includegraphics[scale=.50]{untypical-point-a.png}} \\\\\n\\subfloat[]{\\includegraphics[scale=.50]{untypical-point-b.png}}\n\\caption{(a) A bimodal distribution in which the mode is very untypical of the distribution. The thin blue vertical line is the mean, which is arguably a better summary of the distribution, since it is near the majority of the probability mass. (b) A skewed distribution in which the mode is quite different from the mean.}\n\\label{fig:untypical-point} \n\\end{figure}\n\nHow should we summarize a posterior if the mode is not a good choice? The answer is to use decision theory, which we discuss in Section \\ref{sec:Bayesian-decision-theory}. The basic idea is to specify a loss function, where $L(\\theta,\\hat{\\theta})$ is the loss you incur if the truth is $\\theta$ and your estimate is $\\hat{\\theta}$. If we use 0-1 loss $L(\\theta,\\hat{\\theta})=\\mathbb{I}(\\theta \\neq \\hat{\\theta})$(see section \\ref{sec:Loss-function-and-risk-function}), then the optimal estimate is the posterior mode. 0-1 loss means you only get “points” if you make no errors, otherwise you get nothing: there is no “partial credit” under this loss function! For continuous-valued quantities, we often prefer to use squared error loss, $L(\\theta,\\hat{\\theta})=(\\theta-\\hat{\\theta})^2$ ; the corresponding optimal estimator is then the posterior mean, as we show in Section \\ref{sec:Bayesian-decision-theory}. Or we can use a more robust loss function, $L(\\theta,\\hat{\\theta})=|\\theta-\\hat{\\theta}|$, which gives rise to the posterior median.\n\n\\subsubsection{MAP estimation is not invariant to reparameterization *}\nA more subtle problem with MAP estimation is that the result we get depends on how we parameterize the probability distribution. Changing from one representation to another equivalent representation changes the result, which is not very desirable, since the units of measurement are arbitrary (e.g., when measuring distance, we can use centimetres or inches).\n\nTo understand the problem, suppose we compute the posterior forx. If we define y=f(x), the distribution for yis given by Equation \\ref{eqn:General-transformations}. The $\\frac{\\mathrm{d}x}{\\mathrm{d}y}$ term is called the Jacobian, and it measures the change in size of a unit volume passed\nthrough $f$. Let $\\hat{x}=\\arg\\max_x p_x(x)$ be the MAP estimate for $x$. In general it is not the case that $\\hat{x}=\\arg\\max_x p_x(x)$ is given by $f(\\hat{x})$. For example, let $X \\sim \\mathcal{N}(6,1)$ and $y=f(x)$, where $f(x)=1/(1+\\exp(-x+5))$. \n\n\\begin{figure}[hbtp]\n\\centering\n \\includegraphics[scale=.70]{mode-reparameterization.png}\n\\caption{Example of the transformation of a density under a nonlinear transform. Note how the mode of the transformed distribution is not the transform of the original mode. Based on Exercise 1.4 of (Bishop 2006b).}\n\\label{fig:mode-reparameterization} \n\\end{figure}\n\nWe can derive the distribution of $y$ using Monte Carlo simulation (see Section \\ref{sec:Monte-Carlo-approximation}). The result is shown in Figure \\ref{sec:mode-reparameterization}. We see that the original Gaussian has become “squashed” by the sigmoid nonlinearity. In particular, we see that the mode of the transformed distribution is not equal to the transform of the original mode.\n\nThe MLE does not suffer from this since the likelihood is a function, not a probability density. Bayesian inference does not suffer from this problem either, since the change of measure is taken into account when integrating over the parameter space.\n\n\n\\subsection{Credible intervals}\n\\label{sec:Credible-intervals}\nIn addition to point estimates, we often want a measure of confidence. A standard measure of confidence in some (scalar) quantity $\\theta$ is the “width” of its posterior distribution. This can be measured using a $100(1−\\alpha)\\%$ credible interval, which is a (contiguous) region $C=(\\ell,u)$(standing for lower and upper) which contains $1−\\alpha$ of the posterior probability mass, i.e.,\n\\begin{equation}\nC_{\\alpha}(\\mathcal{D}) \\quad \\text{where } P(\\ell \\leq \\theta \\leq u)=1-\\alpha\n\\end{equation}\n\nThere may be many such intervals, so we choose one such that there is $(1−\\alpha)/2$ mass in each tail; this is called a \\textbf{central interval}.\n\nIf the posterior has a known functional form, we can compute the posterior central interval using $\\ell=F^{-1}(\\alpha/2)$ and $u=F^{-1}(1-\\alpha/2)$, where $F$ is the cdf of the posterior. \n\nIf we don’t know the functional form, but we can draw samples from the posterior, then we can use a Monte Carlo approximation to the posterior quantiles: we simply sort the $\\mathcal{S}$ samples, and find the one that occurs at location $\\alpha/\\mathcal{S}$ along the sorted list. As $\\mathcal{S} \\rightarrow \\infty$, this converges to the true quantile. \n\nPeople often confuse Bayesian credible intervals with frequentist confidence intervals. However, they are not the same thing, as we discuss in Section TODO. In general, credible intervals are usually what people want to compute, but confidence intervals are usually what they actually compute, because most people are taught frequentist statistics but not Bayesian statistics. Fortunately, the mechanics of computing a credible interval is just as easy as computing a confidence interval. \n\n\n\\subsection{Inference for a difference in proportions}\nSometimes we have multiple parameters, and we are interested in computing the posterior distribution of some function of these parameters. For example, suppose you are about to buy something from Amazon.com, and there are two sellers offering it for the same price. Seller 1 has 90 positive reviews and 10 negative reviews. Seller 2 has 2 positive reviews and 0 negative reviews. Who should you buy from?\\footnote{This example is from \\url{http://www.johndcook.com/blog/2011/09/27/bayesian-amazon/}}.\n\nOn the face of it, you should pick seller 2, but we cannot be very confident that seller 2 is better since it has had so few reviews. In this section, we sketch a Bayesian analysis of this problem. Similar methodology can be used to compare rates or proportions across groups for a variety of other settings.\n\nLet $\\theta_1$ and $\\theta_2$ be the unknown reliabilities of the two sellers. Since we don’t know much about them, we’ll endow them both with uniform priors, $\\theta_i \\sim \\text{Beta}(1,1)$. The posteriors are $p(\\theta_1|\\mathcal{D}_1)=\\text{Beta}(91,11)$ and $p(\\theta_2|\\mathcal{D}_2)=\\text{Beta}(3,1)$.\n\nWe want to compute $p(\\theta_1 >\\theta_2|\\mathcal{D})$. For convenience, let us define $\\delta=\\theta_1-\\theta_2$ as the difference in the rates. (Alternatively we might want to work in terms of the log-odds ratio.) We can compute the desired quantity using numerical integration\n\\begin{equation}\n\\begin{split}\np(\\delta>0|\\mathcal{D}) & = \\int_0^1\\int_0^1 \\mathbb{I}(\\theta_1>\\theta_2)\\text{Beta}(\\theta_1|91,11) \\\\\n & \\quad \\text{Beta}(\\theta_2|3,1)\\mathrm{d}\\theta_1\\mathrm{d}\\theta_2\n\\end{split}\n\\end{equation}\n\nWe find $p(\\delta>0|\\mathcal{D})=0.710$, which means you are better off buying from seller 1! \n\n\n\\section{Bayesian model selection}\n\\label{sec:Bayesian-model-selection}\n\nIn general, when faced with a set of models (i.e., families of parametric distributions) of different complexity, how should we choose the best one? This is called the \\textbf{model selection} problem.\n\nOne approach is to use cross-validation to estimate the generalization error of all the candidate models, and then to pick the model that seems the best. However, this requires fitting each model $K$ times, where $K$ is the number of CV folds. A more efficient approach is to compute the posterior over models,\n\\begin{equation}\np(m|\\mathcal{D})=\\dfrac{p(\\mathcal{D}|m)p(m)}{\\sum_{m'}p(\\mathcal{D}|m')p(m')}\n\\end{equation}\n\nFrom this, we can easily compute the MAP model, $\\hat{m}=\\arg\\max_m{p(m|\\mathcal{D})}$. This is called \\textbf{Bayesian model selection}.\n\nIf we use a uniform prior over models, this amounts to picking the model which maximizes\n\\begin{equation}\\label{eqn:marginal-likelihood}\np(\\mathcal{D}|m)=\\int{p(\\mathcal{D}|\\vec{\\theta})p(\\vec{\\theta}|m)}\\mathrm{d}\\vec{\\theta}\n\\end{equation}\n\nThis quantity is called the \\textbf{marginal likelihood}, the \\textbf{integrated likelihood}, or the \\textbf{evidence} for model $m$. The details on how to perform this integral will be discussed in Section \\ref{sec:Computing-the-marginal-likelihood}. But first we give an intuitive interpretation of what this quantity means.\n\n\n\\subsection{Bayesian Occam's razor}\nOne might think that using $p(\\mathcal{D}|m)$ to select models would always favour the model with the most parameters. This is true if we use $p(\\mathcal{D}|\\hat{\\vec{\\theta}}_m)$ to select models, where $\\hat{\\vec{\\theta}}_m)$ is the MLE or MAP estimate of the parameters for model $m$, because models with more parameters will fit the data better, and hence achieve higher likelihood. However, if we integrate out the parameters, rather than maximizing them, we are automatically protected from overfitting: models with more parameters do not necessarily have higher \\emph{marginal likelihood}. This is called the \\textbf{Bayesian Occam’s razor} effect (MacKay 1995b; Murray and Ghahramani 2005), named after the principle known as \\textbf{Occam’s razor}, which says one should pick the simplest model that adequately explains the data.\n\nOne way to understand the Bayesian Occam’s razor is to notice that the marginal likelihood can be rewritten as follows, based on the chain rule of probability (Equation \\ref{eqn:product-rule}):\n\\begin{equation}\\begin{split}\np(D) & =p((\\vec{x}_1,y_1))p((\\vec{x}_2,y_2)|(\\vec{x}_1,y_1)) \\\\\n & \\quad p((\\vec{x}_3,y_3)|(\\vec{x}_1,y_1):(\\vec{x}_2,y_2))\\cdots \\\\\n\t & \\quad p((\\vec{x}_N,y_N)|(\\vec{x}_1,y_1):(\\vec{x}_{N-1},y_{N-1}))\n\\end{split}\\end{equation}\n\nThis is similar to a leave-one-out cross-validation estimate (Section \\ref{sec:Cross-validation}) of the likelihood, since we predict each future point given all the previous ones. (Of course, the order of the data does not matter in the above expression.) If a model is too complex, it will overfit the “early” examples and will then predict the remaining ones poorly.\n\nAnother way to understand the Bayesian Occam’s razor effect is to note that probabilities must sum to one. Hence $\\sum_{p(\\mathcal{D}')} p(m|\\mathcal{D}')=1$, where the sum is over all possible data sets. Complex models, which can predict many things, must spread their probability mass thinly, and hence will not obtain as large a probability for any given data set as simpler models. This is sometimes called the \\textbf{conservation of probability mass} principle, and is illustrated in Figure \\ref{fig:Bayesian-Occams-razor}.\n\n\\begin{figure}[hbtp]\n\\centering\n \\includegraphics[scale=.80]{Bayesian-Occams-razor.png}\n\\caption{A schematic illustration of the Bayesian Occam’s razor. The broad (green) curve corresponds to a complex model, the narrow (blue) curve to a simple model, and the middle (red) curve is just right. Based on Figure 3.13 of (Bishop 2006a). }\n\\label{fig:Bayesian-Occams-razor} \n\\end{figure}\n\nWhen using the Bayesian approach, we are not restricted to evaluating the evidence at a finite grid of values. Instead, we can use numerical optimization to find $\\lambda^*=\\arg\\max_{\\lambda}p(\\mathcal{D}|\\lambda)$. This technique is called \\textbf{empirical Bayes} or \\textbf{type II maximum likelihood} (see Section \\ref{sec:Empirical-Bayes} for details). An example is shown in Figure TODO(b): we see that the curve has a similar shape to the CV estimate, but it can be computed more efficiently.\n\n\n\\subsection{Computing the marginal likelihood (evidence)}\n\\label{sec:Computing-the-marginal-likelihood}\nWhen discussing parameter inference for a fixed model, we often wrote\n\\begin{equation}\np(\\vec{\\theta}|\\mathcal{D},m) \\propto p(\\vec{\\theta}|m)p(\\mathcal{D}|\\vec{\\theta},m)\n\\end{equation}\nthus ignoring the normalization constant $p(\\mathcal{D}|m)$. This is valid since $p(\\mathcal{D}|m)$is constant wrt $\\vec{\\theta}$. However, when comparing models, we need to know how to compute the marginal likelihood, $p(\\mathcal{D}|m)$. In general, this can be quite hard, since we have to integrate over all possible parameter values, but when we have a conjugate prior, it is easy to compute, as we now show.\n\nLet $p(\\vec{\\theta})=q(\\vec{\\theta})/Z_0$ be our prior, where $q(\\vec{\\theta})$ is an unnormalized distribution, and $Z_0$ is the normalization constant of the prior. Let $p(\\mathcal{D}|\\vec{\\theta})=q(\\mathcal{D}|\\vec{\\theta})/Z_{\\ell}$ be the likelihood, where $Z_{\\ell}$ contains any constant factors in the likelihood. Finally let $p(\\vec{\\theta}|\\mathcal{D})=q(\\vec{\\theta}|\\mathcal{D})/Z_N$ be our posterior , where $q(\\vec{\\theta}|\\mathcal{D})=q(\\mathcal{D}|\\vec{\\theta})q(\\vec{\\theta})$ is the unnormalized posterior, and $Z_N$ is the normalization constant of the posterior. We have\n\\begin{align}\np(\\vec{\\theta}|\\mathcal{D})& =\\dfrac{p(\\mathcal{D}|\\vec{\\theta})p(\\vec{\\theta})}{p(\\mathcal{D})} \\\\\n\\dfrac{q(\\vec{\\theta}|\\mathcal{D})}{Z_N}& =\\dfrac{q(\\mathcal{D}|\\vec{\\theta})q(\\vec{\\theta})}{Z_{\\ell}Z_0p(\\mathcal{D})} \\\\\np(\\mathcal{D})& = \\dfrac{Z_N}{Z_0Z_{\\ell}}\n\\end{align}\n\nSo assuming the relevant normalization constants are tractable, we have an easy way to compute the marginal likelihood. We give some examples below.\n\n\\subsubsection{Beta-binomial model}\nLet us apply the above result to the Beta-binomial model. Since we know $p(\\vec{\\theta}|\\mathcal{D})=\\mathrm{Beta}(\\vec{\\theta}|a',b')$, where $a'=a+N_1$, $b'=b+N_0$, we know the normalization constant of the posterior is $B(a',b')$. Hence\n\\begin{align}\np(\\theta|\\mathcal{D})& =\\dfrac{p(\\mathcal{D}|\\theta)p(\\theta)}{p(\\mathcal{D})} \\\\\n & =\\dfrac{1}{p(\\mathcal{D})}\\left[\\dfrac{1}{B(a,b)}\\theta^{a-1}(1-\\theta)^{b-1}\\right] \\nonumber \\\\\n\t& \\quad \\left[\\dbinom{N}{N_1}\\theta^{N_1}(1-\\theta)^{N_0}\\right] \\\\\n\t& =\\dbinom{N}{N_1}\\dfrac{1}{p(\\mathcal{D})}\\dfrac{1}{B(a,b)}\\left[\\theta^{a+N_1-1}(1-\\theta)^{b+N_0-1}\\right]\n\\end{align}\n\nSo\n\\begin{align}\n\\dfrac{1}{B(a+N_1,b+N_0)} & = \\dbinom{N}{N_1}\\dfrac{1}{p(\\mathcal{D})}\\dfrac{1}{B(a,b)} \\\\\np(\\mathcal{D}) & = \\dbinom{N}{N_1}\\dfrac{B(a+N_1,b+N_0)}{B(a,b)}\n\\end{align}\n\nThe marginal likelihood for the Beta-Bernoulli model is the same as above, except it is missingthe $\\binom{N}{N_1}$ term.\n\n\\subsubsection{Dirichlet-multinoulli model}\nBy the same reasoning as the Beta-Bernoulli case, one can show that the marginal likelihood for the Dirichlet-multinoulli model is given by\n\\begin{align}\np(\\mathcal{D}) & =\\dfrac{B(\\vec{N}+\\vec{\\alpha})}{B(\\vec{\\alpha})} \\\\\n & = \\dfrac{\\Gamma(\\sum_k \\alpha_k)}{\\Gamma(N+\\sum_k \\alpha_k)}\\prod\\limits_k \\dfrac{\\Gamma(N_k+\\alpha_k)}{\\Gamma(\\alpha_k)}\n\\end{align}\n\n\\subsubsection{Gaussian-Gaussian-Wishart model}\nConsider the case of an MVN with a conjugate NIW prior. Let $Z_0$ be the normalizer for the prior, $Z_N$ be normalizer for the posterior, and let $Z_{\\ell}(2\\pi)^{ND/2}=$ be the normalizer for the likelihood. Then it is easy to see that\n\\begin{align}\np(\\mathcal{D})& =\\dfrac{Z_N}{Z_0Z_{\\ell}} \\\\\n & = \\dfrac{1}{(2\\pi)^{ND/2}}\\dfrac{\\left(\\frac{2\\pi}{\\kappa_N}\\right)^{D/2}|\\vec{S}_N|^{-\\nu_N/2}2^{(\\nu_0+N)D/2}\\Gamma_D(\\nu_N/2)}{\\left(\\frac{2\\pi}{\\kappa_0}\\right)^{D/2}|\\vec{S}_0|^{-\\nu_0/2}2^{\\nu_0D/2}\\Gamma_D(\\nu_0/2)} \\\\\n & = \\dfrac{1}{\\pi^{ND/2}}\\left(\\dfrac{\\kappa_0}{\\kappa_N}\\right)^{D/2}\\dfrac{|\\vec{S}_0|^{\\nu_0/2}}{|\\vec{S}_N|^{\\nu_N/2}}\\dfrac{\\Gamma_D(\\nu_N/2)}{\\Gamma_D(\\nu_0/2)}\n\\end{align}\n\n\\subsubsection{BIC approximation to log marginal likelihood}\nIn general, computing the integral in Equation \\ref{eqn:marginal-likelihood} can be quite difficult. One simple but popular approximation is known as the \\textbf{Bayesian information criterion} or \\textbf{BIC}, which has the following form (Schwarz 1978):\n\\begin{equation}\n\\mathrm{BIC} \\triangleq \\log p(\\mathcal{D}|\\hat{\\vec{\\theta}})-\\dfrac{\\mathrm{dof}(\\hat{\\vec{\\theta}})}{2}\\log{N}\n\\end{equation}\nwhere $\\mathrm{dof}(\\hat{\\vec{\\theta}})$ is the number of \\textbf{degrees of freedom} in the model, and $\\hat{\\vec{\\theta}}$ is the MLE for the model. We see that this has the form of a \\textbf{penalized log likelihood}, where the penalty term depends on the model’s complexity. See Section TODO for the derivation of the BIC score.\n\nAs an example, consider linear regression. As we show in Section TODO, the MLE is given by $\\hat{\\vec{w}}=(\\vec{X}^T\\vec{X})^{-1}\\vec{X}^T\\vec{y}$ and $\\sigma^2=\\frac{1}{N}\\sum_{i=1}^N (y_i-\\hat{\\vec{w}}^T\\vec{x}_i)$. The corresponding log likelihood is given by\n\\begin{equation}\n\\log p(\\mathcal{D}|\\hat{\\vec{\\theta}}) = -\\dfrac{N}{2}\\log(2\\pi\\hat{\\sigma}^2)-\\dfrac{N}{2}\n\\end{equation}\n\nHence the BIC score is as follows (dropping constant terms)\n\\begin{equation}\n\\mathrm{BIC}=-\\dfrac{N}{2}\\log(\\hat{\\sigma}^2)-\\dfrac{D}{2}\\log{N}\n\\end{equation}\nwhere $D$ is the number of variables in the model. In the statistics literature, it is common to use an alternative definition of BIC, which we call the BIC \\emph{cost}(since we want to minimize it):\n\\begin{equation}\n\\mathrm{BIC\\text{-}cost} \\triangleq -2\\log p(\\mathcal{D}|\\hat{\\vec{\\theta}})-\\mathrm{dof}(\\hat{\\vec{\\theta}})\\log{N} \\approx -2\\log{p(\\mathcal{D})}\n\\end{equation}\n\nIn the context of linear regression, this becomes\n\\begin{equation}\n\\mathrm{BIC\\text{-}cost} = N\\log(\\hat{\\sigma}^2)+D\\log{N}\n\\end{equation}\n\nThe BIC method is very closely related to the \\textbf{minimum description length} or \\textbf{MDL} principle, which characterizes the score for a model in terms of how well it fits the data, minus how complex the model is to define. See (Hansen and Yu 2001) for details.\n\nThere is a very similar expression to BIC/ MDL called the \\textbf{Akaike information criterion} or \\textbf{AIC}, defined as\n\\begin{equation}\n\\mathrm{AIC}(m,\\mathcal{D}) = \\log{p(\\mathcal{D}|\\hat{\\vec{\\theta}}_{MLE})}-\\mathrm{dof}(m)\n\\end{equation}\n\nThis is derived from a frequentist framework, and cannot be interpreted as an approximation to the marginal likelihood. Nevertheless, the form of this expression is very similar to BIC. We see that the penalty for AIC is less than for BIC. This causes AIC to pick more complex models. However, this can result in better predictive accuracy. See e.g., (Clarke et al. 2009, sec 10.2) for further discussion on such information criteria.\n\n\\subsubsection{Effect of the prior}\nSometimes it is not clear how to set the prior. When we are performing posterior inference, the details of the prior may not matter too much, since the likelihood often overwhelms the prior anyway. But when computing the marginal likelihood, the prior plays a much more important role, since we are averaging the likelihood over all possible parameter settings, as weighted by the prior.\n\nIf the prior is unknown, the correct Bayesian procedure is to put a prior on the prior. If the prior is unknown, the correct Bayesian procedure is to put a prior on the prior. \n\n\n\\subsection{Bayes factors}\nSuppose our prior on models is uniform, $p(m) \\propto 1$. Then model selection is equivalent to picking the model with the highest marginal likelihood. Now suppose we just have two models we are considering, call them the \\textbf{null hypothesis}, $M_0$, and the \\textbf{alternative hypothesis}, $M_1$. Define the \\textbf{Bayes factor} as the ratio of marginal likelihoods:\n\\begin{equation}\n\\mathrm{BF}_{1,0} \\triangleq \\dfrac{p(\\mathcal{D}|M_1)}{p(\\mathcal{D}|M_0)}=\\dfrac{p(M_1|\\mathcal{D})}{p(M_2|\\mathcal{D})}/\\dfrac{p(M_1)}{p(M_0)}\n\\end{equation}\n\n\n\\section{Priors}\nThe most controversial aspect of Bayesian statistics is its reliance on priors. Bayesians argue this is unavoidable, since nobody is a \\textbf{tabula rasa} or \\textbf{blank slate}: all inference must be done conditional on certain assumptions about the world. Nevertheless, one might be interested in minimizing the impact of one’s prior assumptions. We briefly discuss some ways to do this below.\n\n\n\\subsection{Uninformative priors}\nIf we don’t have strong beliefs about what $\\theta$ should be, it is common to use an \\textbf{uninformative} or \\textbf{non-informative} prior, and to “let the data speak for itself”.\n\n\n\\subsection{Robust priors}\nIn many cases, we are not very confident in our prior, so we want to make sure it does not have an undue influence on the result. This can be done by using \\textbf{robust priors}(Insua and Ruggeri 2000), which typically have heavy tails, which avoids forcing things to be too close to the prior mean.\n\n\n\\subsection{Mixtures of conjugate priors}\nRobust priors are useful, but can be computationally expensive to use. Conjugate priors simplify the computation, but are often not robust, and not flexible enough to encode our prior knowledge. However, it turns out that a \\textbf{mixture of conjugate priors} is also conjugate, and can approximate any kind of prior (Dallal and Hall 1983; Diaconis and Ylvisaker 1985). Thus such priors provide a good compromise between computational convenience and flexibility.\n\n\n\\section{Hierarchical Bayes}\nA key requirement for computing the posterior $p(\\vec{\\theta}|\\mathcal{D})$ is the specification of a prior $p(\\vec{\\theta}|\\vec{\\eta})$, where $\\vec{\\eta}$ are the hyper-parameters. What if we don’t know how to set $\\vec{\\eta}$? In some cases, we can use uninformative priors, we we discussed above. A more Bayesian approach is to put a prior on our priors! In terms of graphical models (Chapter TODO), we can represent the situation as follows:\n\\begin{equation}\n\\vec{\\eta} \\rightarrow \\vec{\\theta} \\rightarrow \\mathcal{D}\n\\end{equation}\n\nThis is an example of a \\textbf{hierarchical Bayesian model}, also called a \\textbf{multi-level} model, since there are multiple levels of unknown quantities. \n\n\n\\section{Empirical Bayes}\n\\label{sec:Empirical-Bayes}\n\n\\begin{table}\n\\centering\n\\begin{tabular}{ll}\n\\hline\\noalign{\\smallskip}\n\\textbf{Method} & \\textbf{Definition} \\\\\n\\noalign{\\smallskip}\\svhline\\noalign{\\smallskip}\nMaximum likelihood & $\\hat{\\vec{\\theta}}=\\arg\\max_{\\vec{\\theta}} p(\\mathcal{D}|\\vec{\\theta})$ \\\\\nMAP estimation & $\\hat{\\vec{\\theta}}=\\arg\\max_{\\vec{\\theta}} p(\\mathcal{D}|\\vec{\\theta})p(\\vec{\\theta}|\\vec{\\eta})$ \\\\\n\\multirow{2}{*}{ML-II (\\textbf{Empirical Bayes})} & $\\hat{\\vec{\\eta}}=\\arg\\max_{\\vec{\\eta}} \\int p(\\mathcal{D}|\\vec{\\theta})p(\\vec{\\theta}|\\vec{\\eta})\\mathrm{d}\\vec{\\theta}$ \\\\\n & $\\quad =\\arg\\max_{\\vec{\\eta}}p(\\mathcal{D}|\\vec{\\eta})$ \\\\\n\\multirow{2}{*}{MAP-II} & $\\hat{\\vec{\\eta}}=\\arg\\max_{\\vec{\\eta}} \\int p(\\mathcal{D}|\\vec{\\theta})p(\\vec{\\theta}|\\vec{\\eta})p(\\vec{\\eta})\\mathrm{d}\\vec{\\theta}$ \\\\\n & $\\quad =\\arg\\max_{\\vec{\\eta}}p(\\mathcal{D}|\\vec{\\eta})p(\\vec{\\eta})$ \\\\\nFull Bayes & $p(\\vec{\\theta},\\vec{\\eta}|\\mathcal{D}) \\propto p(\\mathcal{D}|\\vec{\\theta})p(\\vec{\\theta}|\\vec{\\eta})$ \\\\\n\\noalign{\\smallskip}\\hline\n\\end{tabular}\n\\end{table}\n\n\n\\section{Bayesian decision theory}\n\\label{sec:Bayesian-decision-theory}\nWe have seen how probability theory can be used to represent and updates our beliefs about the state of the world. However, ultimately our goal is to convert our beliefs into actions. In this section, we discuss the optimal way to do this.\n\nOur goal is to devise a \\textbf{decision procedure} or \\textbf{policy}, $f(\\vec{x}) : \\mathcal{X} \\rightarrow \\mathcal{Y}$, which minimizes the \\textbf{expected loss} $R_{\\mathrm{exp}}(f)$(see Equation \\ref{eqn:expected-loss}).\n\nIn the Bayesian approach to decision theory, the optimal output, having observed $\\vec{x}$, is defined as the output $a$ that minimizes the \\textbf{posterior expected loss}:\n\\begin{equation}\n\\rho(f)=\\mathbb{E}_{p(y|\\vec{x})}[L(y,f(\\vec{x}))]=\\begin{cases}\n\\sum\\limits_y L[y,f(\\vec{x})]p(y|\\vec{x}) \\\\\n\\int\\limits_y L[y,f(\\vec{x})]p(y|\\vec{x})\\mathrm{d}y\n\\end{cases}\n\\end{equation}\n\nHence the \\textbf{Bayes estimator}, also called the \\textbf{Bayes decision rule}, is given by\n\\begin{equation}\n\\delta(\\vec{x})=\\arg\\min\\limits_{f \\in \\mathcal{H}} \\rho(f)\n\\end{equation}\n\n\n\\subsection{Bayes estimators for common loss functions}\n\n\n\\subsubsection{MAP estimate minimizes 0-1 loss}\nWhen $L(y,f(x))$ is \\textbf{0-1 loss}(Section \\ref{sec:Loss-function-and-risk-function}), we can proof that MAP estimate minimizes 0-1 loss, \n\\begin{align*}\n\\arg\\min\\limits_{f \\in \\mathcal{H}} \\rho(f)& =\\arg\\min\\limits_{f \\in \\mathcal{H}} \\sum\\limits_{i=1}^K{L[C_k,f(\\vec{x})]p(C_k|\\vec{x})} \\\\\n & =\\arg\\min\\limits_{f \\in \\mathcal{H}} \\sum\\limits_{i=1}^K{\\mathbb{I}(f(\\vec{x}) \\neq C_k)p(C_k|\\vec{x})} \\\\\n\t\t & =\\arg\\min\\limits_{f \\in \\mathcal{H}} \\sum\\limits_{i=1}^K{p(f(\\vec{x}) \\neq C_k|\\vec{x})} \\\\\n\t\t & =\\arg\\min\\limits_{f \\in \\mathcal{H}} \\left[1-{p(f(\\vec{x}) = C_k|\\vec{x})}\\right] \\\\\n\t\t & =\\arg\\max\\limits_{f \\in \\mathcal{H}} p(f(\\vec{x}) = C_k|\\vec{x})\n\\end{align*}\n\n\n\\subsubsection{Posterior mean minimizes\t$\\ell_2$(quadratic) loss}\nFor continuous parameters, a more appropriate loss function is \\textbf{squared error}, \\textbf{$\\ell_2$ loss}, or \\textbf{quadratic loss}, defined as $L(y,f(\\vec{x}))=\\left[y-f(\\vec{x})\\right]^2$.\n\nThe posterior expected loss is given by\n\\begin{equation}\\begin{split}\n\\rho(f) & =\\int\\limits_y L[y,f(\\vec{x})]p(y|\\vec{x})\\mathrm{d}y \\\\\n & =\\int\\limits_y \\left[y-f(\\vec{x})\\right]^2p(y|\\vec{x})\\mathrm{d}y \\\\\n & =\\int\\limits_y \\left[y^2-2yf(\\vec{x})+f(\\vec{x})^2\\right]p(y|\\vec{x})\\mathrm{d}y\n\\end{split}\\end{equation}\n\nHence the optimal estimate is the posterior mean:\n\\begin{align}\n& \\dfrac{\\partial \\rho}{\\partial f} =\\int\\limits_y [-2y+2f(\\vec{x})]p(y|\\vec{x})\\mathrm{d}y=0 \\Rightarrow \\nonumber \\\\\n& \\int\\limits_y f(\\vec{x})p(y|\\vec{x})\\mathrm{d}y = \\int\\limits_y yp(y|\\vec{x})\\mathrm{d}y \\nonumber \\\\\n& f(\\vec{x}) \\int\\limits_y p(y|\\vec{x})\\mathrm{d}y = \\mathbb{E}_{p(y|\\vec{x})}[y] \\nonumber \\\\\n& f(\\vec{x}) = \\mathbb{E}_{p(y|\\vec{x})}[y]\n\\end{align}\n\nThis is often called the \\textbf{minimum mean squared error} estimate or \\textbf{MMSE} estimate.\n\n\n\\subsubsection{Posterior median minimizes $\\ell_1$(absolute) loss}\nThe $\\ell_2$ loss penalizes deviations from the truth quadratically, and thus is sensitive to outliers. A more robust alternative is the absolute or $\\ell_1$ loss. The optimal estimate is the posterior median, i.e., a value $a$ such that $P(y