[
  {
    "path": ".gitignore",
    "content": "*.aux\n*.log\n*.out\n*.synctex.gz\n*.toc\n*.txss\n.DS_Store"
  },
  {
    "path": "Computer_Vision_1/cv_appendix.tex",
    "content": "\\section{Practicals}\nGathering some interesting/important questions from the practicals and old exams.\n\\subsection{Color spaces}\n\\subsubsection{General parameters in color spaces}\n\\begin{itemize}\n\t\\item \\textbf{Chromaticity}: the color component regardless of its luminance/intensity. For example, the $xy$-diagram in Figure~\\ref{fig:rgb_color_wavelength_distribution_XYZ_diagram} visualizes the chromaticity (includes saturation and hue)\n\t\\item \\textbf{Saturation}: defined as ``colorfulness of a stimulus relative to its own brightness''. In the normalized $rgb$ space, it is the distance to the point $(1/3,1/3,1/3)$ (ratio to the maximum distance). In case of the wavelength distribution, a color is saturated if it is very peaked.\n\t\\item \\textbf{Intensity}: the energy of the light. It is the integral of the wavelength distribution.\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.25\\textwidth]{figures/cv_image_formation_rg_chromaticity.png}\n\t\t\\caption{\\textit{rg}-chromaticity diagram. A point in this space symbolizes the chromaticity (color without intensity), and the distance to the point $(1/3,1/3)$ (if considered white light source as reference) with ratio to distance to border the saturation.}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{XYZ color space}\nCalculate saturation, hue, intensity, plotting in the diagram, using reference lights, etc.\n\nInterpolate between colors. We can perceive color (e.g. white) although it is not as we would define \n\\subsubsection{Color invariance}\nHow to determine whether formula is color invariant or not. \n\\begin{itemize}\n\t\\item Color invariance is trying to remove transformations that do not directly affect the color, but let the sensor perceive it differently. \n\t\\item Hence, color invariant models are more or less insensitive to varying imaging conditions such as variations in illumination (light source) and object pose (shading, highlighting cues)\n\t\\item For example, if we assume a Lambertian world where we only have body reflection and a white light source (equal for all wavelengths), we get for the $rgb$ space (note that $R=cos\\theta \\cdot e\\cdot \\int_{\\lambda} p(\\lambda) f_R(\\lambda)d\\lambda$):\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tr & = \\frac{R}{R + G + B} = \\frac{\\cancel{cos\\theta} \\cdot \\cancel{e}\\cdot \\int_{\\lambda} p(\\lambda) f_R(\\lambda)d\\lambda}{\\cancel{cos\\theta} \\cdot \\cancel{e}\\cdot \\int_{\\lambda} p(\\lambda) \\left(f_R(\\lambda) + f_G(\\lambda) + f_B(\\lambda)\\right)d\\lambda}\n\t\t\\end{split}\n\t\\end{equation*}\n\tThus, the \\textit{rgb} color space is color invariant when assuming a Lambertian reflection model.\n\\end{itemize}\n\\subsection{Convolution operator}\n\\subsubsection{Difference between convolution and correlation}\nFormally, correlation is a measurement of similarity between two signals whilst convolution is a measures the effect of one signal on the other. In practice however, correlation simply moves the filter over the image and computes the sum of the box at each pixel. Convolution is practically the same however before moving over the image, the filter is rotated 180 degrees. The formulas are:\n\\begin{equation*}\n\t\\begin{split}\n\t\t\\text{Correlation:} & I_{out} = I \\otimes h,\\hspace{1mm} I_{out}(i,j) = \\sum\\limits_{k,l} I(i+k, j+l) \\cdot h(k,l)\\\\\n\t\t\\text{Convolution:} &  I_{out} = I \\ast h,\\hspace{1mm} I_{out}(i,j) = \\sum\\limits_{k,l} I(i-k, j-l) \\cdot h(k,l)\n\t\\end{split}\n\\end{equation*}\nNote that for both methods there is no difference in the result if we take the center pixel or a corner pixel as the start point for a filter. \n\\subsubsection{Convolving two filters}\nTwo consecutive filters applied to an image can be summarized into one by convolving two filters. There are two ways to calculate the convolution of two filters. The more intuitive way to calculate the effect of every element of the second filter based on the first one.\nExample:\n\\begin{equation*}\n\t\\begin{split}\n\t\tf &=\\left[\\begin{array}{ccc}3 & 7 & 6\\end{array}\\right], \\hspace{2mm}g=\\left[\\begin{array}{ccc}-1 & 5 & 8\\end{array}\\right] \\Rightarrow f\\ast g \\\\[5pt]\n\t\t& \\implies \\begin{array}{cccccc}\n\t\t\t& [-1\\cdot 3 & 5\\cdot 3 & 8\\cdot 3] & & \\\\\n\t\t +\t& & [-1\\cdot 7 & 5\\cdot 7 & 8\\cdot 7] & \\\\\n\t\t +\t& & & [-1\\cdot 6 & 5\\cdot 6 & 8\\cdot 6] \\\\[5pt]\n\t\t \\hline\n\t\t & [ -3 & 8 & 53 & 86 & 48 ]\n\t\t\\end{array}\n\t\\end{split}\n\\end{equation*}\nThe second option is to apply convolution right away with extended zero padding. We can imagine to use infinite zero padding but remove the zero elements in the convolved filter again. Note that we perform convolution, and therefore have to flip the second filter.\n\\begin{equation*}\n\t\\begin{split}\n\t\tf\\ast g & = \\left[\\begin{array}{ccccccc}0 & 0 & 3 & 7 & 6 & 0 & 0\\end{array}\\right] \\otimes \\left[\\begin{array}{ccc}8 & 5 & -1\\end{array}\\right]\\\\\n\t\t& = \\left[\\begin{array}{ccccccc}-1\\cdot 3 & (5\\cdot 3 - 1\\cdot 7) & (8\\cdot 3 + 5\\cdot 7 - 1\\cdot 6) & (8\\cdot 7 + 5\\cdot 6) & 8\\cdot 6\\end{array}\\right]\\\\\n\t\t& = \\left[\\begin{array}{ccccc}-3 & 8 & 53 & 86 & 48\\end{array}\\right]\n\t\\end{split}\n\\end{equation*}\n\\subsubsection{Linearly Separable Filters}\nSome 2D filters are separable in their $x$ and $y$ dimension. We can test it by comparing the convolution of separated $x$ and $y$ filters with the 2D version.\n\\begin{itemize}\n\t\\item \\textit{What is the benefit of separable filters?}\n\t\n\t\\underline{Answer}: The computational cost is reduced form $k^2$ to $2\\cdot k$.\n\t\n\t\\item \\textit{Prove that a 2D Gaussian filter is linearly separable.}\n\t\n\t\\underline{Answer}: We can show this holds for the continuous case, and thus also for the discrete. Note that we can neglect a constant factor $c$ for normalization as this does not introduce any significant computational effort.\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tG_x * G_y  & = \\frac{1}{\\sqrt{2\\pi}\\sigma} e^{-\\frac{x^2}{2\\sigma^2}} * \\frac{1}{\\sqrt{2\\pi}\\sigma} e^{-\\frac{y^2}{2\\sigma^2}}\\\\\n\t\t\t& = \\frac{1}{2\\pi\\sigma^2} e^{-\\frac{x^2 + y^2}{2\\sigma^2}}\\\\\n\t\t\t& = G_{xy}\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item \\textit{Prove that a 2D box filter (size $3\\times 3$) is linearly separable.}\n\t\n\t\\underline{Answer}: We can show this by simply computing the convolution.\n\t\\begin{equation*}\n\t\\begin{split}\n\t\t\\left[\\begin{array}{ccc}1 & 1 & 1\\end{array}\\right] *  \n\t\t\\left[\\begin{array}{c}1 \\\\ 1 \\\\ 1\\end{array}\\right] & = \\left[\\begin{array}{ccc}\n\t\t1 & 1 & 1\\\\ 1 & 1 & 1\\\\ 1 & 1 & 1\n\t\t\\end{array}\\right]\\\\\n\t\\end{split}\n\t\\end{equation*}\n\t\\item \\textit{Check whether the following 2D filter is linearly separable:}\n\t$$h = \\left[\\begin{array}{ccc}\n\t1 & -2 & 1\\\\ -2 & 4 & -2\\\\ 1 & -2 & 1\n\t\\end{array}\\right]$$\n\t\n\t\\underline{Answer}: The way to check that is looking for symmetric patterns in $x$ and $y$ direction which are independent of the other dimension. In this case, we can easily spot the pattern:\n\t\\begin{equation*}\n\t\\begin{split}\n\t\\left[\\begin{array}{ccc}1 & -2 & 1\\end{array}\\right] *  \n\t\\left[\\begin{array}{c}1 \\\\ -2 \\\\ 1\\end{array}\\right] & = \\left[\\begin{array}{ccc}\n\t1 & -2 & 1\\\\ -2 & 4 & -2\\\\ 1 & -2 & 1\n\t\\end{array}\\right]\\\\\n\t\\end{split}\n\t\\end{equation*}\n\t\\item \\textit{Check whether the following 2D filter is linearly separable:}\n\t$$h = \\left[\\begin{array}{ccc}\n\t1 & 8 & 3\\\\ 7 & 6 & 2\\\\ 4 & 9 & 5\n\t\\end{array}\\right]$$\n\t\n\t\\underline{Answer}:  No, this kernel is not linearly separable.\n\\end{itemize}\n\\subsection{Object detection}\n\n\\subsection{Convolutional Neural Networks}\n\\subsubsection{Amount of parameters, output size and computational cost}\n\\begin{itemize}\n\t\\item \\textbf{Output size}: the spatial output size of a convolutional layer depends on the kernel size $k$, the padding $p$ (per side), the stride $s$ and the input size $w_i$. The output size is then calculated by $w_o = (w_i + 2\\cdot p - k)/s + 1$\n\t\\begin{itemize}\n\t\t\\item \\textit{What is the size of the output volume with stride $3$, kernel $5\\times 5$, number of neurons $5$ and input size $32\\times 32\\times 3$ (no padding)?}\n\t\t\n\t\t\\underline{Answer}: The output size is $w_0 = (32 + 2\\cdot 0 - 5)/3 + 1 = 10$.\n\t\t\n\t\t\\item \\textit{What padding size is required to keep the output size equals to the input size for a kernel $k$ and stride $s$?}\n\t\t\n\t\t\\underline{Answer}: we have to reverse the equation above to:\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\tw_o = w_i & = (w_i + 2\\cdot p - k)/s + 1\\\\\n\t\t\t\t\\Leftrightarrow (w_i - 1) \\cdot s & = w_i + 2\\cdot p - k\\\\\n\t\t\t\t\\Leftrightarrow p & = \\frac{1}{2}\\left(w_i \\cdot \\left(s-1\\right) - s + k\\right)\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tHence, if stride is $s=1$, the necessary padding is $p=\\frac{k-1}{2}$.\n\t\t\n\t\t\\item \\textit{How many output frames do we get for a 3D convolution of $3\\times 3\\times 3$ (stride $s=3$ and padding $p=1$ in temporal dimension) on a input video size of $16\\times 256\\times 256\\times 3$?}\n\t\t\n\t\t\\underline{Answer}: We can apply the same formula as before: $l_o = (16 + 2\\cdot 1 - 3)/3 + 1 = 6$ output frames.\n\t\\end{itemize}\n\t\\item \\textbf{Number of parameters}: a 2D convolution contains $k\\times k\\times c_F \\times c_G$ parameters where $k$ is the kernel size, and $c_F$ and $c_G$ the number of input and output channels. For a 3D convolution, we multiply it by another $k$. Note that all these three $k$'s can be different (e.g. $3\\times 3\\times 1$, $5\\times 1 \\times 1$, ...)\n\t\\begin{itemize}\n\t\t\\item \\textit{How many parameters are learned in a convolutional layer with an RGB input image, $5\\times 5$ kernel size and $100$ different filters?}\n\t\t\n\t\t\\underline{Answer}: We learn $5\\times 5\\times 3\\times 100 = 7,500$ parameters for the filters, and $100$ biases. Thus, we have overall $7,600$ parameters.\n\t\t\n\t\t\\item \\textit{How many parameters are learned if we set the padding to $p=2$ and stride $s=2$?}\n\n\t\t\\underline{Answer}: The number of parameters is independent of the stride and the padding.\n\t\\end{itemize}\n\t\\item \\textbf{Computational cost}: The computational cost of a layer is the cost of a single filter application (the filter size) times the number of output neurons.\n\t\\begin{itemize}\n\t\t\\item \\textit{Given the input $w_F \\times h_F \\times c_F$ and output $w_G \\times h_G \\times c_G$, what is the computational cost of a 2D convolution with kernel size $k\\times k$ between these two layers?}\n\t\t\n\t\t\\underline{Answer}: The cost of applying a single filter once is $k\\times k\\times c_F$. We then have to move the filter over $x$ and $y$ dimension, and repeat it for $c_G$ filters. Thus, the overall cost is determined by:\n\t\t$$k\\times k\\times c_F\\times c_G\\times w_G\\times h_G$$\n\t\t\n\t\t\\item \\textit{Given the input $256 \\times 256 \\times 3$, what is the computational cost of a 2D convolution with kernel size $7\\times 7$, $32$ output channels, stride $s=3$ and padding $p=0$?}\n\t\t\n\t\t\\underline{Answer}: We first have to calculate the output size $w_G = (w_F + 2\\cdot p - k)/s + 1 = (256 + 0 - 7)/3 = 83$ and $h_G = 83$. Next, we can apply our previous formula:\n\t\t$7\\times 7\\times 3\\times 32\\times 83\\times 83$\n\t\t\n\t\t\\item \\textit{What are two ways to reduce the number of computations for 2D convolutions?}\n\t\t\n\t\t\\underline{Answer}: Same as in case of 3D convolutions. We can either do depth-wise convolutions (\\textit{MobileNet}), or do pseudo 2D convolutions by separating the filter $k\\times k$ to a $1\\times k$ and $k\\times 1$ convolution (\\textit{InceptionV2}).\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Other general questions}\n\\begin{itemize}\n\t\\item \\textbf{Locally constrained layer}: A convolutional layer where we don't share weights over spatial dimensions.\n\t\\begin{itemize}\n\t\t\\item \\textit{How many parameters are needed for a locally constrained layer, where each neuron looks at a $10\\times10$ window, when using $W=H=100$, and stride of $5$?}\n\t\t\n\t\t\\underline{Answer}: The spatial output size is $(100 - 10) / 5 + 1 = 19$ so that we have $19\\times 19=361$ different kernels. Combined with the kernel/window size, we get overall $10\\times 10\\times 361=36,100$ parameters.\n\t\t\\item \\textit{Describe a scenario where weight sharing as done in plain convolutional layers is not beneficial for recognition}\n\t\t\n\t\t\\underline{Answer}: Weight sharing works most effectively, if the input is transitional invariant. However, if this is not the case and we have stationary data, we should for example use locally constrained layers where the weights are not shared. This may lead to more parameters but reduces the required amount of channels (restricted number of possible objects per position). Example: face recognition with standardized position (eyes and mouth filters at different parts of the image). \n\t\\end{itemize}\n\\end{itemize}\n"
  },
  {
    "path": "Computer_Vision_1/cv_applications.tex",
    "content": "\\section{Applications}\nNot in the exam :-)"
  },
  {
    "path": "Computer_Vision_1/cv_deep_learning.tex",
    "content": "\\section{Deep Learning}\n\\begin{itemize}\n\t\\item Deep Neural Networks perform hierarchical feature learning and classification in a single architecture\n\\end{itemize}\n\\subsection{Convolutional Neural Networks}\n\\begin{itemize}\n\t\\item Key layer of CNNs are convolutions. The weights are surface-wise local, but depth-wise global.\n\t\\item Multiple neurons look at the same position, but using different kernels (channels)\n\t\\item Parameters of a convolutional layer\n\t\\begin{itemize}\n\t\t\\item \\textit{Kernel size}: size of the filter which is learned. If size is $k\\times k$, we learn overall $k^2$ parameters per channel\n\t\t\\item \\textit{Input channels}: number of input channels $c_i$. Every filter has the size of $k\\times k\\times c_i$\n\t\t\\item \\textit{Output channels}: number of output channels $c_o$. Represent the number of different filters learned.\n\t\t\\item \\textit{Stride} with which we slide the filter over the image. Stride of $s=1$ means we apply a filter on every pixel as usual, $s=2$ would skip every second pixel and $s=4$ takes only every fourth pixel as center of an filter application. Default: $s=1$.\n\t\\end{itemize}\n\t\\item Overall, we learn $(k\\times k\\times c_i + 1)\\times c_0$ \\textbf{parameters} in a convolutional layer (the 1 extra parameter for bias)\n\t\\item The \\textbf{output size} is calculated by $$h_o = (h_i + 2\\cdot p - k) / s + 1$$ where $p$ is the padding (number of extra pixels on each side)\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.2\\textwidth]{figures/cv_deep_learning_convolution_operator.png}\n\t\t\\caption{Convolutional layer in a CNN}\n\t\\end{figure}\n\t\\item Activation layers like ReLU ($\\max(0,x)$) introduce non-linearity\n\t\\item Pooling aggregates multiple values into a single value making it invariant to small transformations. Reduces the size of the next output layer while keeping the most important information \n\\end{itemize}\n\\subsubsection{Transfer Learning}\n\\begin{itemize}\n\t\\item Reuse information gained on a large dataset (e.g. ImageNet) on a new one\n\t\\item Depending on the amount and similarity of data with the pretrained one, we should fine-tune different layers (see Figure~\\ref{fig:transfer_learning})\n\t\\item Transfer Learning can greatly influence the performance of a network. Low level features (first layers) are almost always the same for images as we have to detect edges, colors, etc.\n\\end{itemize}\n\\begin{figure}[ht!]\n\t\\centering\n\t\\includegraphics[width=0.5\\textwidth]{figures/cv_deep_learning_transfer_learning.png}\n\t\\caption{Transfer learning}\n\t\\label{fig:transfer_learning}\n\\end{figure}\n% cv_deep_learning_transfer_learning.png\n\\subsection{GANs}\n\\begin{itemize}\n\t\\item Capture the underlying data distribution and being able to generate new samples\n\t\\item Next to generative adversarial networks, we can also apply Variational Autoencoders or PixelCNN/RNN for this task\n\t\\item GANs are trained by a minimax game between two neural networks (Discriminator $D$ and Generator $G$). $G$ wants to fool $D$ by generating realistic images. $D$ tries to distinguish between generated and real images/data:\n\t$$\\min_G \\max_D V(G,D) = \\mathbb{E}_{\\bm{x}\\sim p_{\\text{data}}(\\bm{x})} \\left[\\log \\left(D\\left(\\bm{x}\\right)\\right)\\right] + \\mathbb{E}_{\\bm{z}\\sim p_{z}(\\bm{z})} \\left[\\log\\left(1 - D\\left(G\\left(\\bm{z}\\right)\\right)\\right)\\right] $$\n\t\\item The standard/plain GAN architecture uses a noise vector $\\bm{z}$ as input to the generator. Note that it is also possible to put  and condition the GANs input on the output (aka \\textit{conditional GANs}). To ensure that the generator learns a relation from input to output, we might need to add an additional loss term like MSE to a label\n\t\\item The training procedure consists of two steps which can be alternated or repeated by themselves for multiple times\n\t\\begin{enumerate}\n\t\t\\item \\textit{Fix $G$ and train $D$}: in order to train the discriminator, we let $G$ generate fake images and feed the discriminator both the fake and sampled real data. Note that we need to fix $G$ to not backpropagate the error of $D$ through $G$.\n\t\t\\item \\textit{Fix $D$ and train $G$}: $G$ is trained by generating images and backpropagating the error of the prediction of $D$ (towards prediction of a real image). Although the gradients flow back through $D$, we do not update any weights of the discriminator as we otherwise cheat (train $D$ to optimize loss of $G$)\n\t\\end{enumerate}\n\\subsubsection{Stability and Training problems}\n\t\\item In general, it is hard to train a GAN. There are a lot of problems that can occur\n\t\\item \\textbf{Vanishing gradients} during training:\n\t\\begin{itemize}\n\t\t\\item If the discriminator is too bad, the generator does not get valid/accurate feedback and can therefore not learn properly\n\t\t\\item If the discriminator is perfect, the generator has very low gradients as a small change does not influence the discriminator\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.4\\textwidth]{figures/cv_deep_learning_GAN_vanishing_gradients.jpeg}\n\t\t\t\\caption{Vanishing gradients problem for training with KL-divergence. When the distance between the two distributions $p$ and $q$ (respectively $P_g$ and $P_r$) is too huge, the KL divergence is very close to zero. Hence, is does not provide any strong gradients in these regions.}\n\t\t\\end{figure}\n\t\\end{itemize}\n\t\\item \\textbf{Reaching the equilibrium}\n\t\\begin{itemize}\n\t\t\\item We know that the nash equilibrium of the minimax game is $P_g=P_r$ meaning the distribution of the real data is equal to the generated data. In that case, $D$ return 0.5 no matter what example we put in (as both distributions are equal).\n\t\t\\item However, it has been shown that such cost functions may not converge when using gradient descent. An example is shown in Figure~\\ref{fig:GAN_reaching_equilibrium}.\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.4\\textwidth]{figures/cv_deep_learning_GAN_oscillating.png}\n\t\t\t\\caption{Oscillating behavior of a non-cooperative game where $\\min_x \\max_y V(x,y) = x\\cdot y$. The equilibrium $x=y=0$ is never reached.}\n\t\t\t\\label{fig:GAN_reaching_equilibrium}\n\t\t\\end{figure}\n\t\\end{itemize}\n\t\\item \\textbf{Mode collapse}\n\t\\begin{itemize}\n\t\t\\item A GAN suffers from a mode collapse if the generator limits its predictions/generated distribution to a few samples/modes.\n\t\t\\item For example in case of the MNIST dataset, this would mean that the generator only creates numbers of one or two different digits. Although a full mode collapse is rarely the case, partial mode collapses frequently occur\n\t\t\\item In order to create a mode collapse, the gradients regarding the noise $\\bm{z}$ must be very low/close to zero. This can for example happen if we fix the discriminator and the generator converges to the optimal image $\\bm{x}^*$ that fools the discriminator the most\n\t\t\\item Once the generator collapse to one mode, the discriminator will learn that this mode is purely/mostly generated and thus changes its predictions. The generator will address that by changing the mode (note that as $\\partial L/\\partial \\bm{z}\\approx 0$, we will just collapse to the next mode and are not able to escape this loop).\n\t\t\\item In the end, this turns into a cat-and-mouse game between the generator and discriminator, and will not converge (see Figure~\\ref{fig:GAN_mode_collapse}).\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.6\\textwidth]{figures/cv_deep_learning_GAN_mode_collapse.png}\n\t\t\t\\caption{\\textit{Top row}: optimal convergence of generator distribution to 8 modes. \\textit{Bottom row}: Sample of a mode collapse after 10k iterations. The generator is only able to generate a single mode.}\n\t\t\t\\label{fig:GAN_mode_collapse}\n\t\t\\end{figure}\n\t\\end{itemize}\n\\end{itemize}"
  },
  {
    "path": "Computer_Vision_1/cv_deep_video.tex",
    "content": "\\section{Deep Video}\n\\begin{itemize}\n\t\\item Understanding a video requires to analyze spatial and temporal information. Thus, also more data is needed to fully train such a network whereas we cannot label every single frame (too expensive)\n\t\\item Grid-like data can be processed by a CNN, temporal mostly by RNN, and for unstructured data a fully connected network is most suitable\n\t\\item Easiest solution for video understanding would be to classify (sample/all) frames independently by standard CNN, and then perform average pooling over predictions. However, this approach does not capture temporal structure\n\\end{itemize}\n\\subsection{Recurrent Neural Networks}\n\\begin{itemize}\n\t\\item In Recurrent Neural Networks, a hidden state flows over time steps. The vanilla RNN formula is\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\th_t & = \\tanh \\left(W_{hh}h_{t-1} + W_{xh} x_{t}\\right)\\\\\n\t\t\ty_t & = W_{hy} h_t\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Weights are shared over time (also $W_{hh}$) so that a RNN can process an arbitrary sequence length. Also, it reduces the number of parameters and thus the chance of overfitting \n\t\\item However, weight sharing can also lead to vanishing gradients as if we backpropagate from $h_t$ to $h_k$, we have a factor $\\theta$ that lets the gradients vanish if it's lower than one, and explode if it is greater than one:\n\t$$\\frac{\\partial h_t}{\\partial h_k} = \\theta^{(t-k)} \\sum f(\\cdot)$$\n\t\\item Vanilla RNNs have troubles capturing long-term dependencies. A possible solution is using LSTMs that control the information flow by three gates (see Figure~\\ref{fig:deep_video_LSTM}):\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\text{Forget gate:  } & f_t = \\sigma\\left(W_f \\cdot \\left[h_{t-1}, x_t\\right] + b_f\\right)\\\\[7pt]\n\t\t\t\\text{Input gate:  } & i_t = \\sigma\\left(W_i \\cdot \\left[h_{t-1}, x_t\\right] + b_i\\right)\\\\\n\t\t\t& \\tilde{c}_t = \\tanh\\left(W_c \\cdot \\left[h_{t-1}, x_t\\right] + b_c\\right)\\\\\n\t\t\t& c_t = f_t * c_{t-1} + i_t * \\tilde{c}_t\\\\[7pt]\n\t\t\t\\text{Output gate:  } & o_t = \\sigma\\left(W_o \\cdot \\left[h_{t-1}, x_t\\right] + b_o\\right)\\\\\n\t\t\t& h_t = o_t * \\tanh\\left(c_t\\right)\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/cv_deep_video_LSTM.png}\n\t\t\\caption{Visual representation of a LSTM chain.}\n\t\t\\label{fig:deep_video_LSTM}\n\t\\end{figure}\n\\end{itemize}\n\\subsection{3D convolutions}\n\\begin{itemize}\n\t\\item We can extend standard convolutions to 3D by moving the filter over the time dimension as well (channels are now 4th dimension over which filter is still global)\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.3\\textwidth]{figures/cv_deep_video_3D_convs.png}\n\t\t\\caption{A 3D convolution is local over spatial and temporal dimensions, but still global over channels (i.e. RGB).}\n\t\t\\label{fig:deep_video_3d_convs}\n\t\\end{figure}\n\t\\item Example: extending a 2D kernel by temporal dimension:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t200\\times 200\\times 3 \\textcolor{blue}{\\times 16} \\xrightarrow{\\text{filter }3\\times3\\textcolor{blue}{\\times 3}} 200\\times 200\\times 256 \\textcolor{blue}{\\times 16}\n\t\t\t\\Rightarrow \\underbrace{3\\times 3}_{\\text{ spatial }}\\underbrace{\\textcolor{blue}{\\times 3}}_{\\text{ temporal }}\\underbrace{\\times 3}_{\\text{ input channels }}\\underbrace{\\times 256}_{\\text{ output channels}}\\text{ parameters}\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Such convolutions learn combined temporal and spatial information. \n\t\\item Alternative is to concatenate all input frames over the channel dimension and pass it to a simple 2D network (also called \\textit{early fusion}). Note that this approach loses the temporal information very fast\n\t\\item Consecutive 3D convolutions can be seen as hierarchical combination of frames. Low level layers therefore capture low level motions, while high level layers (close to output) are able to reason about a longer set of frames and thus high level motion.\n\t\\item Still, it is hard to learn long term dependencies with 3D convolutions as it does not have any gates and thus no explicit control over the information flow\n\t\\item Note that in general, video-based networks are more likely to suffer from overfitting as the input space has a much higher dimensionality and the network has more parameters\n\\end{itemize}\n\\subsection{State-of-the-art}\n\\subsubsection{Two Stream Network}\n\\begin{itemize}\n\t\\item Earliest proposed network for action recognition was \\textbf{Two stream network}\n\t\\item The architecture consists of two networks. One takes a single frame (\\textit{spatial} stream net), and the other processes the concatenated optical flow over the set of frames (\\textit{temporal} stream net). Both predictions are in the end combined\n\t\\item The biggest problem here is that the spatial and temporal information is processed independently, and the very late fusion makes it impossible to reason about both\n\t\\item Other disadvantages include a higher computational cost (two networks plus optical flow), only capturing short motion (early fusion of optical flow), noisy optical flow, and higher probability of overfitting due to number of parameters\n\t\\item Approach can be slightly improved by repeatedly applying the network on small snippets of the network, and combining the prediction afterwards\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/cv_deep_video_two_stream_network.png}\n\t\t\\caption{Architecture of the two stream network.}\n\t\t\\label{fig:deep_video_two_stream_net}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{I3D}\n\\begin{itemize}\n\t\\item Inspired by the success of the 2D version (GoogLeNet), current state-of-the-art networks apply 3D inception modules (see Figure~\\ref{fig:deep_video_I3D_module})\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.35\\textwidth]{figures/cv_deep_video_I3D.png}\n\t\t\\caption{\\textit{Left}: Standard Inception module of the I3D network. \\textit{Right}: Inception module with 3D temporal separable convolutions.}\n\t\t\\label{fig:deep_video_I3D_module}\n\t\\end{figure}\n\t\\item It is pretrained on ImageNet where the 2D filters are (after pretraining) inflated to a third dimension by repeating the values $N$ times over the time dimension, and rescaled by dividing by $N$\n\\end{itemize}\n\\subsubsection{Efficient 3D convolutions}\n\\begin{itemize}\n\t\\item The main drawback of I3D and all other 3D convolutional networks are the huge amount of parameters. There are three ways to efficiently reduce the number of parameters\n\\end{itemize}\n\\begin{enumerate}\n\t\\item \\textbf{Pseudo 3D convolutions}\n\t\\begin{itemize}\n\t\t\\item The idea behind this operation is that the spatial and the temporal dimension do not correlate in every detail, but the temporal dimension is more important locally for the spatial dimension\n\t\t\\item Thus, we split 3D convolution into a 2D spatial and a consecutive 1D temporal convolution. The concept is visualized in Figure~\\ref{fig:deep_video_pseudo_3D_convs}\n\t\t\\item The number of operations applied on input size $l_F \\times w_F \\times h_F \\times c_F$ to output $l_G \\times w_G \\times h_G \\times c_G$ is:\n\t\t\\begin{equation*}\n\t\t\t\t\\underbrace{k \\times k \\times 1 \\times c_F \\times c_I \\times l_F \\times w_G \\times h_G}_{\\text{Spatial 2D convolution}} + \\underbrace{1\\times 1\\times k \\times c_I \\times c_G \\times l_G \\times w_G \\times h_G}_{\\text{Temporal 1D convolution}}\n\t\t\\end{equation*}\n\t\t\\item The speedup by this operation is about $\\frac{1}{k}\\cdot \\frac{c_I}{c_G} \\cdot \\frac{l_F}{l_G} + \\frac{1}{k^2} \\cdot \\frac{c_I}{c_F}\\approx \\frac{1}{k}\\cdot \\frac{c_I}{c_G}$\n\t\\end{itemize}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.55\\textwidth]{figures/cv_deep_video_pseudo_3D_conv.png}\n\t\t\\caption{Pseudo 3D convolutions split the operation into a spatial part (2D) and a temporal (1D) convolution.}\n\t\t\\label{fig:deep_video_pseudo_3D_convs}\n\t\\end{figure}\n\t\\item \\textbf{Depth-wise separable convolutions}\n\t\\begin{itemize}\n\t\t\\item This operation is inspired by the MobileNet architecture and removes the property of convolutions being depth-wise global\n\t\t\\item We consider every input channel independently, and apply a different filter on each of them. For example, if we have an RGB input, we would apply three filters, each processing a different input channel\n\t\t\\item To still allow interaction/combination of multiple channels, we apply a local $1\\times 1\\times 1$ convolution afterwards. Hence, an output channel depends again on all input channels.\n\t\t\\item The number of operations applied on input size $l_F \\times w_F \\times h_F \\times c_F$ to output $l_G \\times w_G \\times h_G \\times c_G$ is:\n\t\t\\begin{equation*}\n\t\t\t\\underbrace{k \\times k \\times k \\times 1 \\times c_F \\times l_G \\times w_G \\times h_G}_{\\text{Depth-wise 3D convolution}} + \\underbrace{1\\times 1\\times 1 \\times c_F \\times c_G \\times l_G \\times w_G \\times h_G}_{\\text{Local }1\\times 1\\times 1\\text{ convolution}}\n\t\t\\end{equation*}\n\t\t\\item The speedup by this operation is considerably bigger than for pseudo 3D, namely $\\frac{1}{c_G} + \\frac{1}{k^{3}} \\approx \\frac{1}{k^{3}}$\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.6\\textwidth]{figures/cv_deep_video_3D_depthwise_conv.png}\n\t\t\t\\caption{Depth-wise 3D convolutions apply one filter per input channel, and combine the different channels afterwards. Same architecture is applied in MobileNet for the 2D case.}\n\t\t\t\\label{fig:deep_video_depthwise_3D_convs}\n\t\t\\end{figure}\n\t\\end{itemize}\n\t\\item \\textbf{Partial 2D architecture} \n\t\\begin{itemize}\n\t\t\\item Depending on the kind of motion we want to detect, it might not be necessary to apply 3D convolutions at every stage of the network. \n\t\t\\item For example, if we are only interested in high-level motions, we might want ot use a \\textit{Top-heavy I3D} which applies 3D convolutions only on the last layers. \n\t\t\\item Similarly, for short motions, we might want to consider a \\textit{Bottom-heavy I3D}. \n\t\t\\item Figure~\\ref{fig:deep_video_I3D_architectures} summarizes the different network architectures.\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.6\\textwidth]{figures/cv_deep_video_different_I3D_architectures.png}\n\t\t\t\\caption{Different I3D network architectures.}\n\t\t\t\\label{fig:deep_video_I3D_architectures}\n\t\t\\end{figure}\n\t\\end{itemize}\n\\end{enumerate}\n\\subsection{Self-supervised learning}\n\\begin{itemize}\n\t\\item Learn to represent a video adequately in the network by using data and tasks where the labels are freely exploited. The great benefit is that we can use a lot of (unlabeled) data\n\t\\item This is mostly done as a pre-training step as the network learns to deal and analyze with videos on a huge dataset. There are various tasks we can perform self-supervised learning on:\n\t\\begin{itemize}\n\t\t\\item \\textbf{Visual tracking}: If we have given a tracking system, we can train a network to predict whether two patches are similar or not. Therefore, we create labels by the tracking system by setting it to 1 if two patches are the same object over time, or otherwise to 0 (we sample a random other patch from the image and compare the scores).\n\t\t\\item \\textbf{Learning by shuffling}: The network is given a set of frames, and its tasks is it to determine whether it is in the correct temporal order or not. The supervision signals are easily generated by labeling the real videos as positive, and shuffle their frame order to create a negative example. The goal is that the network learns to understand poses and motions over frames.\n\t\t\\item \\textbf{Learning by arrow of time}: The task of the network is to predict whether a video is played forwards of backwards (binary classification). This is a very challenging task as it requires the network to understand laws of physics (water only flows downwards, not upwards) by analyzing different motions in the video. One can cluster afterwards what clues the network had extracted which lead to a prediction of forward or backward (called \\textit{arrow of time}). This approach gave the best self-supervised pre-training results so far, but is still not able to beat a supervised ImageNet pre-training.   \n\t\\end{itemize}\n\\end{itemize}"
  },
  {
    "path": "Computer_Vision_1/cv_imgformation.tex",
    "content": "\\section{Image formation}\n\\label{sec:img_formation}\n\\begin{itemize}\n\t\\item To fully understand/analyze an image, we first have to examine how it was created (note that an image is a 2D representation of a 3D world)\n\t\\item Various challenges occur in CV \n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/cv_image_formation_challenges_cv.png}\n\t\t\\caption{Challenges in Computer Vision}\n\t\\end{figure}\n\t\\item The two main parts of how an image is formed are:\n\t\\begin{itemize}\n\t\t\\item \\textit{Geometry} of the projection of a 3D environment to a 2D image. This defines which pixel belongs to which object (part/location). \n\t\t\\item \\textit{Physics of light} which determines the brightness of a point in the image plane as a function of illumination and surface properties. Thus, the light source has a crucial influence on an object's appearance \n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Projective Geometry and Camera models}\n\\begin{itemize}\n\t\\item A camera can be abstracted by a pinhole model. Larger aperture/pinhole results in blurry images, smaller give sharp but noisy images (less energy of light is being passed) $\\Rightarrow$ Change between both by using different lenses\n\t\\item We represent an image by a projection plane. The intersection between the center of projection and the plane is determined by (note that $z$ is negative):\n\t$$(x,y,z)\\to (-\\frac{d}{z}\\cdot x, -\\frac{d}{z}\\cdot y, -d)$$ \n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\begin{subfigure}[b]{0.48\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.4\\textwidth]{figures/cv_image_formation_3D_model.png}\n\t\t\t\\caption{Projection plane}\n\t\t\\end{subfigure}\n\t\t\\begin{subfigure}[b]{0.48\\textwidth}\n\t\t\t\\includegraphics[width=0.8\\textwidth]{figures/cv_image_formation_3D_model_2.png}\n\t\t\t\\caption{Pinhole camera model}\n\t\t\\end{subfigure}\n\t\t\\caption{Abstract camera model in 3D coordinates}\n\t\\end{figure}\n\t\\item Model projection of 3D points to 2D image plane using homogeneous coordinates. The components we use for the projection are:\n\t\\begin{itemize}\n\t\t\\item \\textit{Viewport projection}: Convert plane points to image coordinates (top left corner $(0,0)$, resolution scaling $s_x$, $s_y$)\n\t\t\\item \\textit{Perspective projection}: 3D points to image plane (homogeneous coordinates)\n\t\t\\item \\textit{View transformation}: rotation and translation matrix $\\bm{R}$ and $\\bm{T}$ for modeling the position and orientation of the camera. Can be seen as changing the coordinate system\n\t\t\\item All together, we get the transformed points by:\n\t\t$$\\left[\\begin{array}{c}u\\\\v\\\\1\\end{array}\\right] = \\underbrace{\\left[\\begin{array}{ccc}s_x & 0 & u_0\\\\0 & -s_y & v_0\\\\0 & 0 & 1\\end{array}\\right]}_{\\text{Viewport}} \\cdot \\underbrace{\\left[\\begin{array}{cccc}1 & 0 & 0 & 0\\\\0 & 1 & 0 & 0\\\\0 & 0 & -1/d & 0\\end{array}\\right]}_{\\text{Perspective}} \\cdot \\underbrace{\\left[\\begin{array}{cc}\\bm{R} & \\bm{T} \\\\\\bm{0}^T_3 &  1\\end{array}\\right]}_{\\text{View}} \\cdot \\left[\\begin{array}{c}x\\\\y\\\\z\\\\1\\end{array}\\right]$$\n\t\\end{itemize}\n\t\\item Viewport and perspective projection depend on the camera (size and position of image plain) so that those are called \\textit{intrinsic} camera parameter. In contrast, the view transformation is determined by \\textit{extrinsic} camera parameters as it defines the camera position in the (original) coordinate system\n\\end{itemize}\n\\subsection{Light and Color models}\n\\label{sec:color_models}\n\\begin{itemize}\n\t\\item The appearance color of an object is influenced by three components\n\t\\begin{itemize}\n\t\t\\item \\textit{Light source}: spectral power distribution of light $e(\\lambda)$ \n\t\t\\item \\textit{Object}: the reflection distribution of an object $p(\\lambda)$ (how good certain wavelengths are reflected)\n\t\t\\item \\textit{Sensor}: Detection by the sensor of the distribution $e(\\lambda) p(\\lambda)$\n\t\\end{itemize}\n\t\\item The goal is to be invariant to light source $e(\\lambda)$ and sensor perspective\n\t\\item Two very simple approaches to make an image independent of light source\n\t\\begin{itemize}\n\t\t\\item \\textbf{Gray-world} assumption: the world is in average gray. So, we rescale every channel independently by $128/$mean of channel. Problematic if image is biased towards not being grey (high single channel, etc.)\n\t\t\\item \\textbf{Scale-by-max}/\\textbf{White-patch} assumption: there is always at least one white pixel in an image. Hence, the channels are rescaled by $255$/max of channel. Fails if there is actually no white pixel in the image (results in wrong maximum), or if white pixel is in the shadow $\\Rightarrow$ assumes whole image being shaded.\n\t\t\\item All models underly/use the von Kries model where we convert an unknown light source $u$ to a canonical $c$ (i.e. day light) by simple channel scaling:\n\t\t$$\\left(\\begin{array}{c}R^c\\\\G^c\\\\B^c\\end{array}\\right) = \\left(\\begin{array}{ccc}\n\t\t\\alpha & 0 & 0 \\\\\n\t\t0 & \\beta & 0\\\\\n\t\t0 & 0 & \\gamma\n\t\t\\end{array}\\right) \\cdot \\left(\\begin{array}{c}R^u\\\\G^u\\\\B^u\\end{array}\\right)$$\n\t\tNote that to simplify the calculation of $\\alpha$, $\\beta$ and $\\gamma$, and assume that the channels $R$, $G$ and $B$ are independent (thus only diagonal matrix), we approximate the integral as single wavelength for narrow-band filters.\n\t\\end{itemize}\n\t\\item As computer can't handle continuous distributions, the following integrals are approximate by for example the RGB model:\n\t$$R = \\int_\\lambda e(\\lambda) p(\\lambda) f_R(\\lambda) d\\lambda, \\hspace{2mm}G = \\int_\\lambda e(\\lambda) p(\\lambda) f_G(\\lambda) d\\lambda, \\hspace{2mm}B = \\int_\\lambda e(\\lambda) p(\\lambda) f_B(\\lambda) d\\lambda$$\n\tEvery spectral color (see below diagram in Figure~\\ref{fig:rgb_color_wavelength_distribution_RGB}) can be represented by an linear combination of RGB values.\n\tNote that human ganglion cells have similar functions, but are the most sensitive to green.\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\begin{subfigure}[b]{0.4\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.75\\textwidth]{figures/cv_image_formation_color_RGB_model.png}\n\t\t\t\\caption{RGB model}\n\t\t\t\\label{fig:rgb_color_wavelength_distribution_RGB}\n\t\t\\end{subfigure}\n\t\t\\begin{subfigure}[b]{0.24\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=\\textwidth]{figures/cv_image_formation_color_XYZ_model.png}\n\t\t\t\\caption{XYZ model}\n\t\t\t\\label{fig:rgb_color_wavelength_distribution_XYZ}\n\t\t\\end{subfigure}\n\t\t\\hspace{5mm}\n\t\t\\begin{subfigure}[b]{0.28\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.8\\textwidth]{figures/cv_image_formation_color_XYZ_diagram_2.png}\n\t\t\t\\caption{XYZ diagram}\n\t\t\t\\label{fig:rgb_color_wavelength_distribution_XYZ_diagram}\n\t\t\\end{subfigure}\n\t\t\\caption{Color matching functions $f_R$, $f_G$ and $f_B$ for the standard (a) RGB / (b) XYZ model. The colors represented by the XYZ system are shown in (c). Note that the line of purples contains colors that cannot be created by a monochromatic light source and needs a combination of fully saturated red and violet (max and min of spectrum).}\n\t\t\\label{fig:rgb_color_wavelength_distribution}\n\t\\end{figure}\n\t\\item The intensity of the RGB color space is calculated by the sum of the channels: $I=R+G+B$\n\t\\item Another color space is the XYZ system. The color matching functions $\\overline{x}(\\lambda), \\overline{y}(\\lambda), \\overline{z}(\\lambda)$ are similar but not the same as RGB (see Figure~\\ref{fig:rgb_color_wavelength_distribution_XYZ}). The values are calculated by:\n\t$$X = \\int_\\lambda e(\\lambda) p(\\lambda) \\overline{x}(\\lambda) d\\lambda, \\hspace{2mm}Y = \\int_\\lambda e(\\lambda) p(\\lambda) \\overline{y}(\\lambda) d\\lambda, \\hspace{2mm}Z = \\int_\\lambda e(\\lambda) p(\\lambda) \\overline{z}(\\lambda) d\\lambda$$\n\t\\item However, we can split these measurements into a brightness/luminance and chromaticity/color component specified by $x$ and $y$. The luminance is given by $Y$ ($XYZ$ was designed for that), and the chromaticity is determined as ($Z$ is implicitly given by $1-x-y$):\n\t$$x=\\frac{X}{X+Y+Z},\\hspace{2mm}y=\\frac{Y}{X+Y+Z}$$\n\t\\item The created colors can be visualized in an $xy$-diagram (see Figure~\\ref{fig:rgb_color_wavelength_distribution_XYZ_diagram}). \n\t\\item Given a reference light source $e$, we can determine the dominant wavelength (\\textit{hue}) of a point $p$ by a line from $e$ through $p$ towards the boundary. The \\textit{saturation} is given by the ratio of line length between $e$ and $p$ and $e$ to dominant wavelength boundary. Combining these with the luminance $Y$, a point $p$ can be converted into the HSI color space (see Figure~\\ref{fig:rgb_color_HSV_color_cone}).\n\t\\item HSV can be seen as applying non-linear functions on the wavelength distribution (see Figure~\\ref{fig:rgb_color_HSV_wavelength_dist}). Hue is defined as the dominant wavelength, saturation as the purity of the color (probably relation between max energy and mean), and the brightness/luminance (given by average)\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\begin{subfigure}[b]{0.4\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.5\\textwidth]{figures/cv_image_formation_color_HSV.png}\n\t\t\t\\caption{HSV color cone}\n\t\t\t\\label{fig:rgb_color_HSV_color_cone}\n\t\t\\end{subfigure}\n\t\t\\begin{subfigure}[b]{0.4\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.6\\textwidth]{figures/cv_image_formation_color_HSV_wavelength_dist.png}\n\t\t\t\\caption{HSV wavelength distribution}\n\t\t\t\\label{fig:rgb_color_HSV_wavelength_dist}\n\t\t\\end{subfigure}\n\t\t\\caption{HSV color space. }\n\t\\end{figure}\n\t\\item The $xy$-diagram in Figure~\\ref{fig:rgb_color_wavelength_distribution_XYZ_diagram} visualizes the gamut that is visible for an average person/human vision. Different color spaces/devices capture colors by defining three points and linearly interpolate between those. However, it can be seen that there is no such gamut that can include the whole human vision gamut.\n\\end{itemize}\n\\subsection{Reflection models}\n\\begin{itemize}\n\t\\item When a light source shines on an object, it might be differently perceived from different sensors/cameras although they have the same properties $\\Rightarrow$ object appearance by reflectance\n\t\\item The reflectance properties of an object/point can be specified by a \\textit{BRDF}: Bi-directional reflectance distribution function $f(\\theta_i, \\phi_i; \\theta_r, \\phi_r)$ ($\\theta_i$ and $\\phi_i$ define the angles between input light and surface normal in $x$-$z$/$x$-$y$ direction respectively, $\\theta_r$ and $\\phi_r$ for the outgoing direction).\n\t\\item A BRDF can be build up by different components, as visualized in Figure~\\ref{fig:reflection_models_brdf_reflection_components}. The main parts can be distinguished into \\textit{body reflection} (also referred to as mate appearance), and \\textit{surface reflection} (responsible for the glossy appearance)\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.8\\textwidth]{figures/cv_image_formation_reflectance_properties.png}\n\t\t\\caption{Different components of reflectance. Black arrow visualizes the input ray, and greyish/shaded arrows the output rays. Length of the output rays indicate their energy. }\n\t\t\\label{fig:reflection_models_brdf_reflection_components}\n\t\\end{figure}\n\t\\item There are different models that approximate/assume/deal with certain forms of BRDFs\n\\end{itemize}\n\\subsubsection{Lambertian model}\n\\begin{itemize}\n\t\\item The lambertian reflectance model assumes a BRDF that constant: $f(\\theta_i, \\phi_i; \\theta_r, \\phi_r) = \\frac{\\rho_d}{\\pi}$ where $\\rho_d$ is defined by the albedo of the object, and division of $\\pi$ as energy is equally distributed over hemisphere\n\t\\item The surface reflection/output radiance can be calculated by $L=\\frac{\\rho_d}{\\pi}I\\cos \\theta_i=\\frac{\\rho_d}{\\pi}I\\cdot (\\vec{n}\\cdot \\vec{s})$ where $I$ is light source intensity, $\\vec{n}$ the surface normal and $\\vec{s}$ the input ray direction.\n\t\\item Note that the factor $(\\vec{n}\\cdot \\vec{s})$ defines the ratio of energy/photons that interact with that point/surface\n\t\\item By assuming a Lambertian world, we can decompose an image into a shading part (surface normals) and the albedo (reflectance) of an object.\n\\end{itemize}\n\\subsubsection{Phong model}\n\\begin{itemize}\n\t\\item The Phong model extends the Lambertian model by taking glossy reflectance into account (note that mirror is mostly approximated by glossy as mirror only looks at a single output angle which is rarely met). See Figure~\\ref{fig:reflection_models_phong} for the components of the Phong model\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.7\\textwidth]{figures/cv_image_formation_phong_model.png}\n\t\t\\caption{The Phong model combines diffuse and glossy reflectance. Note that ambient gives the object a certain base brightness for approximating reflectance among objects/walls/...}\n\t\t\\label{fig:reflection_models_phong}\n\t\\end{figure}\n\t\\item The reflection component of the specularity is calculated by $L_s=I\\cdot \\rho_s \\left(\\cos \\phi\\right)^{n_{shiny}}=I\\cdot \\rho_s \\left(\\vec{r}\\cdot \\vec{v}\\right)^{n_{shiny}}$ where $r$ is the output ray/mirror direction (calculated by $\\vec{r}=2\\cdot \\vec{n} \\cdot (\\cos \\theta) - \\vec{s}$), and $v$ the view direction of the sensor. \n\t\\item Large values for $n_{shiny}$ lead to narrow, small dot reflections (close to mirror) while small $n_{shiny}$ give broad, big surface reflectance. Note that the intensity is capped at a highest value (e.g. 1 or 255), so that multiple points can have the maximum intensity although they have a slightly different angle\n\t\\item Also, Figure~\\ref{fig:reflection_models_phong} shows that the body reflection (diffuse and ambient) contain the object color while the specularity depends on the light source (highlights color from light source)\n\\end{itemize}\n\\subsubsection{Dichromatic reflection models}\n\\begin{itemize}\n\t\\item The previously discussed models only consider the light source intensity for the reflection. However, we can integrate the reflection in our color models:\n\t$$\\text{body}_C = m_b (\\vec{n}, \\vec{s})  \\int_\\lambda e(\\lambda) p(\\lambda) f_C(\\lambda) d\\lambda$$\n\t$$\\text{surface}_C = m_s (\\vec{n}, \\vec{s}, \\vec{v})  \\int_\\lambda e(\\lambda) c(\\lambda) f_C(\\lambda) d\\lambda$$\n\twhere $C$ is a specific channel (for example $R$, $G$ or $B$), $\\vec{n}$ is the surface normal, $\\vec{s}$ the input ray direction and $\\vec{v}$ the viewpoint. \n\t\\item The function $m_b$ models the diffuse body reflection (i.e. $m_b(\\vec{n}, \\vec{s})=\\cos \\theta = \\vec{n}\\cdot \\vec{s}$ as for Lambertian) whereas $m_s$ represents the glossy surface reflection (i.e. Phong model). \n\t\\item The diffuse reflectance depends on the albedo of the object $p(\\lambda)$ whereas $c(\\lambda)$ determines the specularity of the object for certain wavelengths.\n\t\\item The perceived color of an object is the sum of the body and the surface\n\t\\item Our goal is to map an input image into a space which is independent of the scene (i.e. independent of $m_b$, $m_s$, ...). Different color models can help:\n\t\\begin{itemize}\n\t\t\\item \\textbf{rgb}: Assuming a white light source, normalize RGB values by the intensity (i.e. $r=\\frac{R}{R+G+B}$). This leads to photometric invariance for pure matte objects ($m_b$ cancels out as it is the same for all channels when assuming $m_s=0$). Note that this approach fails if an object has no color (i.e. all gray tones are mapped to the same value).\n\t\t\\item \\textbf{c1c2c3}: color space is obtained from RGB manipulation and is invariant to shadowing effects of light interaction particularly for matte objects. It has similar properties as rgb, but is determined by $c_1(R,G,B) = \\arctan\\frac{R}{\\max\\left\\{G,B\\right\\}}$\n\t\t\\item \\textbf{HSV} can be invariant to specularity if we assume a white light source and thus white specularity. The dominant wavelength, i.e. the hue, stays the same for those points. However, note that this model is instable for gray and especially white points that commonly occur at maximum specularity, as the hue is undefined.\n\t\t\\item \\textbf{l1l2l3}: Similar behavior as HSV, but calculates the values by $l_1(R,G,B) = \\frac{(R-G)^2}{(R-G)^2 + (R-B)^2 + (G-B)^2}$\n\t\\end{itemize}\n\t\\item Figure~\\ref{fig:cv_image_formation_invariance_color_spaces} summarizes some invariance properties of common color spaces\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/cv_image_formation_invariance_color_spaces.png}\n\t\t\\caption{Overview of invariance in color spaces.}\n\t\t\\label{fig:cv_image_formation_invariance_color_spaces}\n\t\\end{figure}\n\t\\item Different color spaces have different instabilities. Normalized colors get unstable around black pixels ($R=1, G=0, B=0$ is considered as pure red in rgb although in RGB it is black) whereas Hue is instable for low saturation (any hue gives same color)\n\t\\item Another method to be invariant to shadows is filtering the image for smooth image intensity transitions as color transitions are harsh compared to that. The new image is recovered by summing up over gradients. Note that this method fails for sharp shadows and/or smooth color transitions\n\\end{itemize}"
  },
  {
    "path": "Computer_Vision_1/cv_imgprocessing.tex",
    "content": "\\section{Image processing}\n\\begin{itemize}\n\t\\item Apply various algorithms on image to analyze/improve the data\n\t\\item The simplest kind of image transformation are those independent to the spatial position (thus also called point processing) where the new image is calculated by $g=a\\cdot t(f)+ b$. Examples: gamma correction ($\\log x$ to boost small, black values more than high ones), histogram equalization\n\\end{itemize}\n\\subsection{Neighborhood processing}\n\\begin{itemize}\n\t\\item The most common way to process an image is by applying filters on it. A filter is a linear weighted sum of local input values. \n\t\\item A convolution of image $I$ and a linear filter $h$ is calculated by $$I_{out} = I \\ast h, \\hspace{1mm} I_{out}(i,j) = \\sum\\limits_{k,l} I(i-k, j-l) \\cdot h(k,l)$$\n\t\\item Depending on the size of the filter, we might not be able to apply the filter on the pixels at the border. Thus, we extend the image to have the same output shape. Common padding methods are zero/black, mirror/copy edge or wrap around.\n\t\\item There are a lot of different filters that can be applied on an image. Filters can for example also be used for translation if wanted/needed. 1D example: $\\left[\\begin{array}{ccc}\n\t0 & 0 & 1\n\t\\end{array}\\right]$\n\t\\item In general, we distinguish between \\textit{low}-pass filters (smoothing) and \\textit{high}-pass filters (edge detection, sharpening). The frequency is thereby the change of pixel values, and the passed wavelengths describe to what the filters react the most. Note that there are also \\textit{band}-pass filters (low-pass filter convolved with high-pass filter)\n\t\\item For example, unicolor images stay mostly unchanged when they are processed by an low-pass filter. In contrast, applying a high-pass filter on such images leads to very low activations.\n\\end{itemize}\n\\subsubsection{Smoothing filters}\n\\begin{itemize}\n\t\\item \\textit{Box filter}: replace every pixel by the average of its neighborhood. \n\t$$h = \\frac{1}{9}\\left[\\begin{array}{ccc}\n\t1 & 1 & 1\\\\\n\t1 & 1 & 1\\\\\n\t1 & 1 & 1\\\\\n\t\\end{array}\\right]$$\n\tConvolving a box filter with itself results in a filter in a shape of a Gaussian\n\t\\item \\textit{Gaussian filter}: weight contributions of neighboring pixels by distance: $G_\\sigma = \\frac{1}{2\\pi \\sigma^2} e^{-\\frac{(x^2 +y^2)}{2\\sigma^2 }}$. A $3\\times 3$ Gaussian with $\\sigma=0.5$ has the following values:\n\t$$h= \\left[\\begin{array}{ccc}\n\t0.011 & 0.084 & 0.011\\\\\n\t0.084 & 0.619 & 0.084\\\\\n\t0.011 & 0.084 & 0.011\\\\\n\t\\end{array}\\right]$$\n\tNote that convolving a Gaussian with another Gaussian is again a Gaussian. Thus, we can separate a 2D Gaussian into two 1D filters which are sequentially applied on the image $\\Rightarrow$ reduce computational effort from $n^2$ to $2n$.\n\t\\item \\textit{Sharpening filter}: reverses the process of smoothing by accentuates differences with local average\n\t$$h = \\left[\\begin{array}{ccc}\n\t0 & 0 & 0\\\\\n\t0 & 2 & 0\\\\\n\t0 & 0 & 0\\\n\t\\end{array}\\right]-\\frac{1}{9}\\left[\\begin{array}{ccc}\n\t1 & 1 & 1\\\\\n\t1 & 1 & 1\\\\\n\t1 & 1 & 1\\\\\n\t\\end{array}\\right]$$\n\t\\item \\textit{Median filter}: A non-linear filter that selects the median value in the kernel window. The advantage of this filter is that its robust against outliers (good for filtering out salt-and-pepper noise)\n\\end{itemize}\n\\subsubsection{Edge detection filters}\n\\begin{itemize}\n\t\\item \\textit{Simple gradient filter}: The simplest gradient/edge detector is in 1D: $h = \\left[\\begin{array}{cc}-1 & 1\\end{array}\\right]$ \n\t\\item \\textit{Sobel filter}: a derivative filter that also takes nearby pixels into account for better approximation. $h_x$ detects vertical edges (gradients over $x$-direction) and $h_y$ detects horizontal edges.\n\t$$h_x = \\left[\\begin{array}{ccc}\n\t1 & 0 & -1\\\\\n\t2 & 0 & -2\\\\\n\t1 & 0 & -1\\\\\n\t\\end{array}\\right] \\text{\\hspace{5mm}and\\hspace{5mm}}h_y = \\left[\\begin{array}{ccc}\n\t1 & 2 & 1\\\\\n\t0 & 0 & 0\\\\\n\t-1 & -2 & -1\\\\\\end{array}\\right] $$ \n\t\\item \\textit{Derivative of a Gaussian}: the derivative of a Gaussian is highly suitable for edge detection as it represents a band-pass filter (Gaussian filter convolved with discrete gradient filter although derivative mostly calculated by continuous). Similar to sobel, but weights the pixels nearby a bit different. Note that we also have different filters for $x$ and $y$ direction.\n\t\\item \\textit{Laplacian of Gaussian}: Laplacian operator $\\nabla^2 f = \\frac{\\partial^2 \n\tf}{\\partial x^2} + \\frac{\\partial^2 f}{\\partial y^2}$ applied on a Gaussian. Is invariant to the direction of the gradient (circular symmetric). The shape of the function is also often described as a Mexican hat (see Figure~\\ref{fig:cv_image_processing_gaussian_filters}). Is highly responsive to blobs (blob detection) but is sensitive to the scale. To be invariant of the scale, we can apply multiple LoG filters with different values of $\\sigma$ and stack the results together.\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.6\\textwidth]{figures/cv_image_processing_gaussian_filters.png}\n\t\t\\caption{Visualization of different Gaussian filters.}\n\t\t\\label{fig:cv_image_processing_gaussian_filters}\n\t\\end{figure}\n\\end{itemize}\n\\subsection{Harris corner detector}\n\\begin{itemize}\n\t\\item Detect interest points in an image to perform matching or similar tasks. Corners are suitable to serve as interest points as they have a unique 2D position compared to edges and points\n\t\\item The initial idea is derived from performing autocorrelation on a small window of the image, and test which ones are unique/expressive. Now we are looking for small changes in $x$ and $y$ direction, how much the image changes. Based on that information, we can decide whether a pixel represents a corner or not.\n\t\\item Steps in the Harris corner detector\n\t\\begin{enumerate}\n\t\t\\item Compute the derivatives $I_x$ and $I_y$ of the image\n\t\t\\item Compute the products of the derivatives at every pixel: $I_x^2$, $I_y^2$, $I_{xy}=I_{x}\\cdot I_{y}$ \n\t\t\\item Compute sums of products over the window size and align them in the Harris matrix:\n\t\t$$H = \\left[\\begin{array}{cc}\n\t\t\\sum_W I_x^2 & \\sum_W I_x \\cdot I_y\\\\\n\t\t\\sum_W I_x \\cdot I_y & \\sum_W I_y^2 \\\\\n\t\t\\end{array}\\right]$$\n\t\tNote that the sum represents the application of a box filter. It is equally possible to apply Gaussian filters etc. \n\t\t\\item Determine the response of the detector at each pixel:\n\t\t$$R = \\det(H) - k\\cdot \\left(\\text{trace}(H)\\right)^2$$\n\t\t\\item If $|R|$ is small, the region is probably flat. Otherwise, if $R<0$ (and greater a certain threshold) we have an edge, and $R>0$ indicates a corner.\n\t\t\\item Perform non-maximum suppression if corner detector is calculated pixel-wise.\n\t\\end{enumerate}\n\t\\item Determining the \\textit{cornerness} of a point is based on the eigenvalues of the matrix $H$: $R=\\lambda_1 \\lambda_2 - k\\cdot (\\lambda_1 + \\lambda_2)^2$. The maximum eigenvalue is the gradient of the direction with the fastest change, and the minimum eigenvalue the gradient of the direction with the smallest change. Note that this models an ellipse for the gradients (see Figure~\\ref{fig:cv_image_processing_harris_ellipse})\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.25\\textwidth]{figures/cv_image_processing_harris_ellipse.png}\n\t\t\\caption{Visualization of relation between eigenvalues and gradients.}\n\t\t\\label{fig:cv_image_processing_harris_ellipse}\n\t\\end{figure}\n\t% cv_image_processing_harris_ellipse.png\n\t\\item If we have an edge, one eigenvalue is considerably greater than the other as in one direction we have a large gradient, whereas in the other (90 degrees) the pixels stay the same. Here, $R$ is smaller than 0 as $\\lambda_1\\lambda_2$ is small but $\\lambda_1 + \\lambda_2$ is large.\n\t\\item Thus, we only have a corner if in both directions we have a (equally) high change. In that case, $R$ is positive as $\\lambda_1\\lambda_2$ is large.\n\t\\item Other properties of the Harris Corner detector\n\t\\begin{itemize}\n\t\t\\item Partial invariance to \\textit{affine intensity} change. As only derivatives are used, a bias term $I+b$ does not influence result. When multiplying an image by a factor $I\\cdot a$, we scale the eigenvalues and thus the cornerness as well. We therefore might only have to adapt the threshold.\n\t\t\\item \\textit{Rotation invariant} as only the ellipse rotates but the eigenvalues stay the same\n\t\t\\item \\textit{Scaling sensitive}: The Harris corner detector is sensitive to scale as it usually applies LoG/Derivatives of Gaussians for determining $I_x$ and $I_y$. To make the corner detector invariant to scale, we can apply multiple gradient filters with different values for $\\sigma$ and stack them together (3D output instead of 2D). We then perform the detector on various scales, and take in the end the maximum response over scales for every pixel.\n\t\\end{itemize}\n\t\\item Applications\n\t\\begin{itemize}\n\t\t\\item \\textit{Image stitching} as for combining separate photos into a panorama. We therefore detect interest points in all images, and try to match those (description by e.g. SIFT/histogram/...)\n\t\t\\item \\textit{Object recognition} by comparing local features that were found for a specific object with the ones from another image.\n\t\\end{itemize}\n\\end{itemize}"
  },
  {
    "path": "Computer_Vision_1/cv_intro.tex",
    "content": "\\section{Introduction}\n\\subsection{Challenges in Computer Vision}\n\\begin{itemize}\n\t\\item \n\\end{itemize}"
  },
  {
    "path": "Computer_Vision_1/cv_object_rec.tex",
    "content": "\\section{Object recognition}\n\\begin{itemize}\n\t\\item Challenges in object recognition\n\t\\begin{itemize}\n\t\t\\item Huge dimensionality (large input size)\n\t\t\\item Image formation process (see Section~\\ref{sec:img_formation})\n\t\t\\item Images are stationary signal and share features, but have to distinguish it from noise\n\t\\end{itemize}\n\t\\item Hard to define explicit rules, but easy to collect examples $\\Rightarrow$ Machine learning\n\\end{itemize}\n\\subsection{Image representations}\n\\begin{itemize}\n\t\\item Need to find an image representation that is able to capture the semantics of an image and hence makes it easy to recognize objects\n\t\\item For normal pixel values, the euclidean distance does not reflect the similarity of images well. A change of illumination or translation has a huge impact on the metric although it is the same object\n\t\\item Global histograms over whole image are scale and translation invariant, but are not really distinctive (different images have same histogram)\n\t\\item The best way is to find \\textit{local features} that images share. They are more descriptive and reoccur in different images. In the next step, we have to describe these features to get a final representation. \n\t\\item One way to describe them are using SIFT (Scale-invariant feature transform) which creates a local histogram of gradients in the neighborhood (see Figure~\\ref{fig:SIFT})\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.6\\textwidth]{figures/cv_object_detection_SIFT.png}\n\t\t\\caption{SIFT descriptors for a $2\\times 2$ histogram patch (normally $4\\times 4$ see Figure~\\ref{fig:descriptor_SIFT}).}\n\t\t\\label{fig:SIFT}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{Histogram of Gradients (HoG)}\n\\begin{itemize}\n\t\\item A HoG descriptor abstracts a patch by a histogram of gradient orientations. \\item The steps for calculating a HoG descriptor for a given patch are\n\t\\begin{enumerate}\n\t\t\\item Determine pixel-wise gradients $I_x$ and $I_y$ by e.g. applying a Sobel filter (or rather simple $[1,-1]$ derivative filter)\n\t\t\\item Determine the orientation $\\theta = \\arctan \\frac{I_y}{I_x}$ and magnitude $I=\\sqrt{I_y^2 + I_x^2}$ of the pixel-wise gradients\n\t\t\\item Report gradients as a histogram. For example, if we take a 9 bin histogram, we map every gradient to the closest value of $0^{\\circ}$, $45^{\\circ}$, $90^{\\circ}$ etc. Note that the 9th bin is for zero gradients which have no orientation.\n\t\\end{enumerate}\n\t\\item An improvement to simply counting the number of gradients is considering their magnitude as well, or using a non-hard counting ($30^{\\circ}$ counts for $0^{\\circ}$ and $45^{\\circ}$).\n\t\\item A disadvantage of HoG is that it is not rotational invariant\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/cv_image_processing_HoG.jpg}\n\t\t\\caption{The HoG descriptor takes the gradients in a patch and group them into a histogram of orientations.}\n\t\t\\label{fig:descriptor_HoG}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{Scale Invariant Feature Transform (SIFT)}\n\\begin{itemize}\n\t\\item SIFT is a combination of detector and descriptor which is (mostly) both rotation and scale invariant\n\t\\item The first step of SIFT is getting a scale-invariant response map. This is done by extracting features by LoG (or rather DoG due to runtime) on various scales (see Figure~\\ref{fig:SIFT_scale_invariance})\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/cv_image_processing_SIFT_scales.png}\n\t\t\\caption{SIFT}\n\t\t\\label{fig:SIFT_scale_invariance}\n\t\\end{figure}\n\t\\item We now look for local maxima in terms of both scale and location. This means that we search for points that are higher than all neighboring pixels in $x$-$y$ direction and scale (see Figure~\\ref{fig:SIFT_scale_invariance} green points on the right) $\\Rightarrow$ non-maximum suppression\n\t\\item Given these points, we check for their \\textit{cornerness}. Only at those points, we need to calculate the gradients and estimate the eigenvalues:\n\t$$\\frac{\\text{Tr}(\\bm{H})^2}{\\text{Det}(\\bm{H})} < \\frac{(r+1)^2}{r}$$\n\tThe term $(r+1)^2/r$ is just a new threshold that specifies the required ratio between first and second eigenvalue.\n\t\\item To guarantee rotation invariance, we look for the dominant gradient orientation in the patch. This is done by creating a weighted histogram of gradient orientations in the whole patch (weighted by the magnitudes of these gradients), and take the orientation with the highest value as orientation of the patch. If the patch has other orientations that have a value of at least 80\\% of the dominant orientation, we create another descriptor for those as well.\n\t\\item Once a point is selected as a key-point, we can group all gradients in small regions in a histogram and combine them into a $4\\times 4$ grid of histograms. Note that we adjust all gradients according to the orientation of the key-point. Our final descriptor has then 128 features ($4\\times 4$ histograms with each $8$ bins, see Figure~\\ref{fig:descriptor_SIFT})\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/cv_image_processing_SIFT.jpg}\n\t\t\\caption{A SIFT descriptor with $4\\times 4$ histogram patch.}\n\t\t\\label{fig:descriptor_SIFT}\n\t\\end{figure}\n\\end{itemize}\n\\subsection{Bag-of-Words}\n\\begin{itemize}\n\t\\item One approach for image representation is the visual Bag-of-Words (BoW). We therefore split an image into patches, describe each of these patches by one \"visual word\" (patch in our dictionary), and finally create a histogram out of it (see Figure~\\ref{fig:BoW_pipeline})\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/cv_object_detection_BoW_pipeline.png}\n\t\t\\caption{General pipeline for BoW approach.}\n\t\t\\label{fig:BoW_pipeline}\n\t\\end{figure}\n\t\\item There are four components of BoW for which a design choice has to be made\n\t\\begin{itemize}\n\t\t\\item \\textbf{Patch sampling}: which patches should be used to describe an image. Can be either descriptive patches/interest points, but then the number of patches can significantly differ from image to image. Alternatively, we can perform a grid-like selection of patches (\\textit{dense sampling}) on multiple scale (reduce size of image and sample again).\n\t\t\\item \\textbf{Patch description}: describe the patches/visual words by SIFT, RGB, HOG or similar. Goes along with image representation \n\t\t\\item \\textbf{Visual dictionary}: create a dictionary by sampling a lot of patches from a large set of images (training images), and cluster them in their descriptor space to find distinctive patches. Use these clusters as visual words. There are different cluster methods that can be applied. However, one hyper-parameter is usually the number of clusters. High number of clusters give very distinctive, but noise sensitive patches, whereas low number of clusters give general, but less distinctive patches.\n\t\t\\item \\textbf{Histogram creation}: the simplest approach is finding the nearest prototype/visual word for every sampled batch of the image by e.g. L2 on the descriptor, and record the number of occurrences for each visual word. There are many (more advanced) alternatives that for example take the distance to the cluster means into account, or calculating mean and stddev etc. \n\t\\end{itemize} \n\t\\item Advantages and drawbacks of visual BoW\n\t\\begin{itemize}\n\t\t\\item[+] Translation invariant\n\t\t\\item[+] Fixed length feature vector \n\t\t\\item[$-$] Loss of spatial information\n\t\t\\item[$-$] Quantization loses information (mapping to visual words)\n\t\\end{itemize}\n\t\\item In order to keep some spatial information, we can extend the histogram by using multiple scales (spatial pyramid) and concatenate those for an output feature vector. Another approach would be to use the spatial information ($xy$-position) as additional features for the patch descriptor, and use during matching/clustering.\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.45\\textwidth]{figures/cv_object_detection_BoW_spatial_pyramid.png}\n\t\t\\caption{Spatial pyramid for histogram creation. We concatenate all histogram to a longer feature vector.}\n\t\t\\label{fig:BoW_spatial_pyramid}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{Bag of Words for Retrieval}\n\\begin{itemize}\n\t\\item We can compare images for the retrieval task by their BoW histogram. This is more efficient and faster than checking for every interest point and try to compare those.\n\t\\item Offline, we have to create the BoW vocabulary and determine a histogram for every image in our database\n\t\\item When an image is entered as a query, we need to represent it by its BoW histogram and then compare it with every other.\n\t\\item We can apply other techniques from IR as well like TF-IDF, query expansion, stop word removal, inverted file index,...\n\t\\item To guarantee a good performance for the first retrieved examples, we can rerank the top $k$ by using geometrical verification (detect interest points and try to match those)\n\\end{itemize}\n\\subsection{Object detection}\n\\begin{itemize}\n\t\\item Localization of objects in an image. Often approximated by bounding boxes that should be predicted around the object.\n\t\\item A simple sliding window approach is too expensive as it generates 1) a lot of boxes over 2) a lot of scales with 3) different box ratios/shapes and 4) many classes.\n\t\\item Hence, the first challenge is to find a set of relevant boxes with ``object'' (also called \\textit{candidate boxes} all graded by an objectness score), and in a second step determine the class of the object in this candidate boxes\n\t\\item One approach for that is \\textbf{selective search} which is based on the property of images being hierarchical\n\t\\begin{itemize}\n\t\t\\item Segment image into small fragments based on simple approaches. Generate for all of these a candidate box\n\t\t\\item For multiple iterations (recursively), combine two fragments that are the most similar together and consider a box for the combined fragment as well. Repeat until only one region is left\n\t\t\\item Apply a classifier on those candidate boxes\n\t\\end{itemize}\n\t\\item A general pipeline for object detection is shown in Figure~\\ref{fig:cv_object_detection_BB_pipeline}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.45\\textwidth]{figures/cv_object_detection_BB_pipeline.png}\n\t\t\\caption{Pipeline for object detection with Bounding Boxes1.}\n\t\t\\label{fig:cv_object_detection_BB_pipeline}\n\t\\end{figure}\n\t% cv_object_detection_BB_pipeline.png\n\\end{itemize}"
  },
  {
    "path": "Computer_Vision_1/cv_summary.tex",
    "content": "\\documentclass[a4paper]{article} \n\\addtolength{\\hoffset}{-2.25cm}\n\\addtolength{\\textwidth}{4.5cm}\n\\addtolength{\\voffset}{-3.25cm}\n\\addtolength{\\textheight}{5cm}\n\\setlength{\\parskip}{0pt}\n\\setlength{\\parindent}{0in}\n\n\\usepackage{blindtext} % Package to generate dummy text\n\\usepackage{charter} % Use the Charter font\n\\usepackage[utf8]{inputenc} % Use UTF-8 encoding\n\\usepackage{microtype} % Slightly tweak font spacing for aesthetics\n\\usepackage[english]{babel} % Language hyphenation and typographical rules\n\\usepackage{amsthm, amsmath, amssymb} % Mathematical typesetting\n\\usepackage{float} % Improved interface for floating objects\n\\usepackage[final, colorlinks = true, \nlinkcolor = black, \ncitecolor = black]{hyperref} % For hyperlinks in the PDF\n\\usepackage{graphicx, multicol} % Enhanced support for graphics\n\\usepackage{xcolor} % Driver-independent color extensions\n\\usepackage{marvosym, wasysym} % More symbols\n\\usepackage{rotating} % Rotation tools\n\\usepackage{subcaption}\n\\usepackage{wrapfig}\n\\usepackage[makeroom]{cancel}\n\\usepackage{censor} % Facilities for controlling restricted text\n\\newcommand{\\note}[1]{\\marginpar{\\scriptsize \\textcolor{red}{#1}}} % Enables comments in red on margin\n\\usepackage{bm}\n\\usepackage{blkarray}\n\\usepackage{enumitem}\n\\usepackage{pgfplots}\n\\usepackage{tikz}\n\\definecolor{colkeyword}{rgb}{0,0.4,0}\n\\definecolor{colname}{rgb}{0.4,0.4,0}\n\\definecolor{coltype}{rgb}{0.4,0,0.4}\n\\definecolor{coloperators}{rgb}{0,0,1.0}\n\\definecolor{colscopes}{rgb}{0.4,0,0}\n\n\\setcounter{tocdepth}{2}\n% Title Page\n\\title{Summary Computer Vision 1}\n\\author{Phillip Lippe}\n\n\n\\begin{document}\n\\maketitle\n\\tableofcontents\n\\newpage\n\n% \\input{cv_intro.tex}\n\\input{cv_imgformation.tex}\n\\input{cv_imgprocessing.tex}\n\\input{cv_object_rec.tex}\n\\input{cv_deep_learning.tex}\n\\input{cv_deep_video.tex}\n\\input{cv_applications.tex}\n\\appendix\n\\newpage\n\\input{cv_appendix.tex}\n\\end{document}"
  },
  {
    "path": "Deep_Learning/cheat_sheet/main.tex",
    "content": "%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n% MatPlotLib and Random Cheat Sheet\n%\n% Edited by Michelle Cristina de Sousa Baltazar\n%\n% http://matplotlib.org/api/pyplot_summary.html\n% http://matplotlib.org/users/pyplot_tutorial.html\n%\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n\n\\documentclass[a4paper]{article}\n\\usepackage[landscape]{geometry}\n\\usepackage{url}\n\\usepackage{multicol}\n\\usepackage{amsmath}\n\\usepackage{amsfonts}\n\\usepackage{tikz}\n\\usetikzlibrary{decorations.pathmorphing}\n\\usepackage{amsmath,amssymb}\n\n\\usepackage{colortbl}\n\\usepackage{xcolor}\n\\usepackage{mathtools}\n\\usepackage{amsthm, amsmath, amssymb, amsfonts}\n\\usepackage{enumitem}\n\n\\title{Deep Learning cheat sheet}\n\\usepackage[english]{babel}\n\\usepackage[utf8]{inputenc}\n\\usepackage{bm}\n\n\\newcommand{\\pd}[2]{\\frac{\\partial #1}{\\partial #2}}\n\\newcommand{\\loss}[0]{\\mathcal{L}}\n\\newcommand{\\chain}[3]{\\frac{\\partial #1}{\\partial #2}\\frac{\\partial #2}{\\partial #3}}\n\\newcommand{\\eq}[1]{\\begin{equation*}\\begin{split}#1\\end{split}\\end{equation*}}\n\\newcommand{\\coderef}[0]{Please find the implementation in the folder with the code files.}\n\\newcommand{\\TODO}[1]{\\textbf{\\textcolor{red}{#1}}}\n\n\\definecolor{green}{RGB}{0,160,0}\n\\definecolor{blue}{RGB}{0,0,160}\n\\definecolor{red}{RGB}{160,0,0}\n\\definecolor{orange}{RGB}{200,160,0}\n\\definecolor{purple}{RGB}{170,0,200}\n\\definecolor{cyan}{RGB}{0,200,200}\n\\definecolor{lightred}{RGB}{200,50,50}\n\n\\advance\\topmargin-0.9in\n\\advance\\textheight3in\n\\advance\\textwidth3in\n\\advance\\oddsidemargin-1.5in\n\\advance\\evensidemargin-1.5in\n\\parindent0pt\n\\parskip2pt\n\\newcommand{\\hr}{\\centerline{\\rule{3.5in}{1pt}}}\n%\\colorbox[HTML]{e4e4e4}{\\makebox[\\textwidth-2\\fboxsep][l]{texto}\n\\begin{document}\n\\footnotesize\n\\begin{multicols*}{3}\n\n\\tikzstyle{mybox} = [draw=black, fill=white, very thick,\n    rectangle, rounded corners, inner sep=10pt, inner ysep=10pt]\n\\tikzstyle{fancytitle} =[fill=black, text=white, font=\\bfseries]\n%------------ CONTEÚDO CAIXA RANDOM ---------------\n\\begin{tikzpicture}\n\\node [mybox] (box){%\n    \\begin{minipage}{0.3\\textwidth}\n\t\\underline{Definition}: A family of \\textcolor{green}{parametric}, \\textcolor{lightred}{non-linear} and \\textcolor{blue}{hierarchical} \\textcolor{orange}{representation learning functions}, which are \\textcolor{red}{massively optimized with stochastic gradient descent} to \\textcolor{purple}{encode domain knowledge}, i.e. domain invariances, stationarity.\\\\\n\t\\vspace{-3mm}\n\t\\begin{itemize}[leftmargin=4mm]\n\t\t\\setlength\\itemsep{0.0em}\n\t\t\\item Neural Network is a directed acyclic graph\t\t\n\t\t% \\item Every module can be expressed by $a=h(x;w)$\n\t\t\\item Use loss function that matches output distribution to improve numerical stability and make gradients larger\n\t\t\\item Input and output distribution of every module should be the same to prevent inconsistent behavior and harder learning\n\t\\end{itemize}\n\t\\underline{Backprop}: chain rule $\\pd{z}{x_i}=\\sum_j \\chain{z}{y_j}{x_i}$, $\\nabla_{\\bm{x}} \\bm{z} = \\left(\\pd{\\bm{y}}{\\bm{x}}\\right)^T \\cdot \\nabla_{\\bm{y}} \\bm{z}$\n\t\\vspace{-1mm}\n\t\\begin{enumerate}[leftmargin=4mm]\n\t\\setlength\\itemsep{0.2em}\n\t\\item Compute forward: $a^{(l)} = h^{(l)}\\left(x^{(l)}\\right)$, $x^{(l+1)}=a^{(l)}$\n\t\\item Compute reverse: $\\pd{\\loss}{a^{(l)}} = \\left(\\pd{a^{(l+1)}}{x^{(l+1)}}\\right)^T \\cdot \\pd{\\loss}{a^{(l+1)}}$\\\\$\\pd{\\loss}{\\theta^{(l)}} = \\pd{a^{(l)}}{x^{(l+1)}} \\cdot \\left(\\pd{\\loss}{a^{(l)}}\\right)^T$\n\t\\item Update params: $\\theta^{(l)}_{t+1} = \\theta^{(l)}_{t}-\\eta \\nabla_{\\theta_t^{(l)}}\\loss$\n\t\\end{enumerate}\n\t\t\n%\t\\begin{center}\\small{\\begin{tabular}{lp{4.5cm} l}\n%\t\t\\textit{random():} & obtém o próximo número aleatório no intervalo [0.0, 1.0] \\\\ \\hline\n%\t\t\\textit{random(começo,fim):} & obter o próximo número aleatório no intervalo [começo, fim] \\\\ \\hline\n%\t\t\\textit{random(stop):} & obtém o próximo número aleatório no intervalo [0, fim]\n%\t\\end{tabular}}\\end{center}\n    \\end{minipage}\n};\n\\node[fancytitle, right=10pt] at (box.north west) {Modular Learning};\n\\end{tikzpicture}\n\n\n%------------ CONTEÚDO CAIXA MatPlotLib ---------------\n\\begin{tikzpicture}\n\\node [mybox] (box){%\n    \\begin{minipage}{0.3\\textwidth}\n\t\t\\underline{Pure optimization} very direct goal to optimize (e.g. scheduling). ML wants to optimize test error that is intractable/only indirectly optimizable. Reduce different cost function on training set, optimum might be not optimal for test set (overfitting).\\\\[4pt]\n\t\t\\underline{Gradient descent}: dataset mostly too large, slow, not better optimum/faster convergence. \\underline{SGD}: standard error $\\sigma/\\sqrt{m}$, noisy gradients act as regularizer, dynamicly changing data possible. \\\\[4pt]\n\t\t\\underline{Ill conditioning}: if 2nd order change is greater than 1st ($\\frac{1}{2}\\epsilon^2 g^THg>\\epsilon g^Tg$), loss increases. Later training, reduce lr\\\\\n\t\t\\underline{Pathological curvatures}: ravine region in loss surface, high gradients in suboptimal direction, oscillations, slow convergence\\\\[4pt]\n\t\t\\underline{Hessian}: requires large batch to be accurate, hard to compute\\\\\n\t\t\\underline{Momentum}: maintain momentum from previous updates to dampen oscillations: $u_{t+1}=\\gamma u_t - \\eta_t g_t$, $w_{t+1}=w_t+u_{t+1}$. Exponential averaging $\\Rightarrow$ more robust gradients, faster\\\\\n\t\t\\underline{Nesterov momentum}: take future gradients, better in theory.\\\\[2pt]\n\t\t\\underline{RMSprop}: adaptive lr, exp. averaging over norms, assuming directions of sensitivity axis aligned. $r_t = \\alpha \\cdot r_{t-1} + \\left(1 - \\alpha\\right) \\cdot g_t^2$, $\\eta_t = \\frac{\\eta}{\\sqrt{r_t} + \\epsilon}$, $w_{t+1} = w_{t} - \\eta_t \\cdot g_t$\\\\[3pt]\n\t\t\\underline{AdaGrad}: adaptive lr, \\textit{sums} norm, thus based on scale and frequency, bad for nonconvex. $r_t = r_{t-1} + \\text{diag}(g^2_t)$\\\\ [3pt]\n\t\t\\underline{Adam}: Combine adaptive lr and momentum (applied on unscaled gradients). Bias correction to account init at origin. \\\\[4pt]\n\t\t\\underline{Bayesian optimization}: gradient-free, educated trail and error guesser, determine next point on uncertainty and expectation\\\\\n\t\t\\underline{Normalization}: center data around 0, same variance, allows higher learning rate and better learning. \\textit{BatchNorm}: ensure Gaussian distribution of features over batches. $\\hat{y}_i = \\gamma \\cdot \\hat{x}_i + \\beta$\\\\$\\mu_B = \\frac{1}{m} \\sum\\limits_{i=1}^{m} x_i$, $\\sigma_B^2 = \\frac{1}{m} \\sum\\limits_{i=1}^{m} \\left(x_i - \\mu_B\\right)^2$, $\\hat{x}_i = \\frac{x_i - \\mu_B}{\\sqrt{\\sigma^2 + \\epsilon}}$\\\\\n\t\tReduce effect of 2nd order between layers, acts as regularizer by introducing noise, let network control mean and variance.\\\\\n\t\tDuring testing, take moving average of last training steps\n    \\end{minipage}\n};\n%------------ CAIXA PRELIMINARES ---------------------\n\\node[fancytitle, right=10pt] at (box.north west) {Deep Learning Optimization (1)};\n\\end{tikzpicture}\n%------------ CONTEUDO EXEMPLO BASICO ---------------------\n\\begin{tikzpicture}\n\\node [mybox] (box){%\n    \\begin{minipage}{0.3\\textwidth}\n    \t\\underline{Regularization}: objective during training to reduce test error\\\\\n    \t\\textit{$\\ell_2$}: introduce objective $\\frac{\\lambda}{2}\\sum_l ||w_l||^2$, weight decay for SGD\\\\\n    \t\\textit{$\\ell_1$}: sparse weights with $\\lambda\\sum_l ||w_l||$\\\\\n    \t\\textit{Others}: Dropout, Early stopping, Augmentation, Multitask\\\\[4pt]\n    \t\\underline{Weight initialization}: small weights to keep data at origin, large to have strong gradients, preserve variance of activations ($w\\sim \\mathcal{N}(0,\\sqrt{1/d})$), no learning if all same, prevent dead ReLU\n    \\end{minipage}\n};\n%------------ EXEMPLO BASICO BOX ---------------------\n\\node[fancytitle, right=10pt] at (box.north west) {Deep Learning Optimization (2)};\n\\end{tikzpicture}\n%------------ CONTEUDO DOIS EIXOS ---------------------\n\\begin{tikzpicture}\n\\node [mybox] (box){%\n    \\begin{minipage}{0.3\\textwidth}\n\t\tImages stationary signals with spatial structure and huge dimensionality. Dimensions highly correlated (translation inv)\\\\[4pt]\n\t\t\\underline{Transfer Learning}: use large datasets to learn useful features, prevent overfitting, fine-tune less layers if datasets similar, use lower lr for pre-trained layers as close to optimum\\\\[4pt]\n\t\t \\underline{Architectures}: small filter for less params and higher non-linearity (even $n\\times1$/$1\\times n$), different scales on same input (stack of convs prone to overfitting), vanishing gradients by intermediate classifiers or residual connections (learn difference instead of mapping) $H(x)=x+F(x)$, possibly with gates\\\\[4pt]\n\t\t \\underline{Tracking}: \\textit{Fast R-CNN} based on middle feature map, extract BB (selective search, NN for \\textit{Faster R-CNN}). RoI pooling to get fixed-size output. \\textit{Siamese}: train on similarity of BB patches.\n    \\end{minipage}\n};\n%------------ DOIS EIXOS BOX ---------------------\n\\node[fancytitle, right=10pt] at (box.north west) {(Modern) Convolutional Neural Networks};\n\\end{tikzpicture}\n%------------ CONTEÚDO COMANDOS DE TEXTO ---------------------\n\\begin{tikzpicture}\n\\node [mybox] (box){%\n    \\begin{minipage}{0.3\\textwidth}\n    \\underline{Backprop thorugh time}: gradients of weights on memory $W$: $\\pd{\\loss}{W} = \\sum\\limits_{t=1}^{T}\\sum\\limits_{k=1}^{t} \\chain{\\loss_t}{y_t}{c_t}\\left(\\prod\\limits_{i=k+1}^{t} \\pd{c_i}{c_{i-1}}\\right)\\frac{\\partial^{+}c_k}{\\partial W}$\\\\\n    Formulating RNN as $c_t = W \\cdot \\sigma(c_{t-1}) + U \\cdot x_{t-1}$ leads to:\\\\ $\\left\\lVert \\pd{c_{t+1}}{c_{t}}\\right\\rVert \\leq \\left\\lVert W^T\\right\\rVert \\cdot \\left\\lVert \\text{diag}\\left(\\pd{\\sigma\\left(c_t\\right)}{c_t}\\right)\\right\\rVert$. If norm of non-linearity bounded by $\\gamma$, and $\\left\\lVert W^T\\right\\rVert < 1/\\gamma$, then vanishing gradients. If $\\left\\lVert W^T\\right\\rVert \\gg 1/\\gamma$ and non-linearity not zero, then exploding gradients. Quick fix for second: clip gradient norm\\\\[4pt]\n    \\underline{LSTM}: Prevent vanishing gradient by gated skip connections over time. Forget, output, and input+candidate gate\\\\[4pt]\n    \\underline{GNN}: \\textit{Deep Walk}: latent repr. by random walks, skip gram on sequences, not dynamic. \\textit{GraphSage}: aggregate information from neighbors, can be mean/max pool with weights, LSTM.\n    \\textit{GCN}: $h(H^{(l)}, A) = \\sigma\\left(D^{-1/2}\\hat{A}D^{-1/2} H^{(l)}W^{(l)}\\right)$\n    \\end{minipage}\n};\n%------------ COMANDOS DE TEXTO BOX ---------------------\n\\node[fancytitle, right=10pt] at (box.north west) {Recurrent Neural Networks};\n\\end{tikzpicture}\n%------------ CONTEÚDO COMANDOS DE TEXTO ---------------------\n\\begin{tikzpicture}\n\\node [mybox] (box){%\n\t\\begin{minipage}{0.3\\textwidth}\n\t\\underline{Generative modeling}: learn joint probability $p(x,y)$ or density function $p(x)$. Task can be performed by Bayes: $p(y|x)$. Generalizes better, better modeling of causal relations, out-of-distribution detection $p(y|x)p(x)$ with $p(x)$ low. \\textit{Discriminative modeling}: learn pdf $p(y|x)$, task-oriented and mostly better\\\\[4pt]\n\t\\underline{Applications}: RL simulator, creating missing data (pixel patches), super-res., data augm., cross-modal transl. (sketch to img)\\\\[4pt]\n\t\\underline{Types}: \\textit{Explicit density}: maximize log likelihood of data by modeling pdf. Must be complex enough and computationally tractable. \\textit{Implicit density}: no explicit pdf, only sampling mechanism\n\t\\end{minipage}\n};\n%------------ COMANDOS DE TEXTO BOX ---------------------\n\\node[fancytitle, right=10pt] at (box.north west) {Deep Generative Models (1)};\n\\end{tikzpicture}\n%------------ CONTEÚDO COMANDOS DE TEXTO ---------------------\n\\begin{tikzpicture}\n\\node [mybox] (box){%\n\t\\begin{minipage}{0.3\\textwidth}\n\t\\underline{GAN}: implicit model, adversarial training. Mini-max game:\\\\ $\\min_G \\max_D V(G,D) = \\mathbb{E}_{\\bm{x}\\sim p_{r}(\\bm{x})} \\left[\\log \\left(D\\left(\\bm{x}\\right)\\right)\\right] + \\mathbb{E}_{\\bm{z}\\sim p_{z}(\\bm{z})} \\left[\\log\\left(1 - D\\left(G\\left(\\bm{z}\\right)\\right)\\right)\\right] $. Better loss for generator: $-\\log D(G(z))$. Otherwise vanishing gradients if D too strong.\\\\[2pt]\n\t\\textit{Problems}: reaching equilibrium (oscillation around Nash), mode collapse if $\\partial \\loss / \\partial z\\approx 0$, low dimensional support (JS assumes overlap of distributions).\\\\[2pt]\n\t\\textit{Improvements}: WGAN using Earth-Mover's distance (also good for non-overlapping), usage of labels $y$ like in conditional GANs, label smoothing for overconfident D, Virtual BatchNorm with reference batch to reduce intra-batch inference\\\\[4pt]\n\t\\underline{Boltzmann machines}: Pdf based on energy function we learn: $p(x)=1/Z \\exp(-E(x))$ where $Z=\\sum_{x'} \\exp(E(x'))$. $Z$ complex, $2^{n}$ pos. for binary data. Restrain to pairwise relations: $E(x)=-x^TWx-b^Tx$. \\textit{Restricted BM}: reduce $W$ by introducing $h$ latents: $E(x,h)=-x^TWh-b^Tx-c^Th$, $p(x)=1/Z\\sum_{h'} \\exp(-E(x,h'))$, higher-order relations. Can reformulate to $p(h_j|x,\\theta)=\\sigma(W_{:,j}x+b_j)$, $p(x_i|h,\\theta)=\\sigma(W_{i,:}h+c_i)$. Maximize log likelihood by contrastive divergence. Sample $h_0\\sim p(h|x)$, $x_1\\sim p(x|h_0)$, a.s.o.\\\\[4pt]\n\t\\underline{VAE}: Model $p(x,z)=p(x|z)p(z)$. Goal is to maximize $p(x)=\\int p(x,z)dz$ which is intractable. Use ELBO instead:\\\\\n\t$\\log p(x) > \\mathbb{E}_{q_{\\varphi}(z|x)}\\left[\\log p_{\\theta}(x|z)\\right] - \\text{KL}\\left(q_{\\varphi}(z|x)||p_{\\lambda}(z)\\right)$\\\\\n\tDifference is $ - \\text{KL}\\left(q_{\\varphi}(z|x)||p(z|x)\\right)$. \\textit{Reparameterization trick}: sample from external dist., and transform it to own. For Gaussian: $z=\\mu_q + \\epsilon \\cdot \\sigma_q$. Backprop through model params and lower variance than REINFORCE.\\\\\n\t\\textit{Improvements}: $q(z|x)$ with NF on top, ELBO is extended by NF term. Optimize prior $p_{\\lambda}(z)=\\frac{1}{K}\\sum_k q_{\\varphi}\\left(z|u_k\\right)$, $u_k$ trained\\\\[4pt]\n\t\\underline{NF}: Model $p(x)$ directly with series of invertible transformations shifting probability mass. Math expression of NF:\\\\\n\t $x = z_k = f_k \\circ f_{k-1} \\circ ... \\circ f_1 (z_0) \\to z_i = f_i(z_{i-1})$\\\\\n\t $p(z_i) = p(z_{i-1}) \\cdot \\left|\\det \\frac{f_{i}^{-1}}{z_i}\\right| \\implies p(x) = p(z_0) \\cdot \\prod_{i=1}^{K} \\left|\\det \\frac{f_{i}^{-1}}{z_i}\\right|$\\\\\n\t $\\log p(x) = \\log p(z_0) - \\sum_{i=1}^{K} \\log \\left|\\det \\frac{f_{i}}{z_i}\\right|$\\\\\n\t $f$ must be invertible and has simple $\\det$ Jacobian (triangular)\n\t\\end{minipage}\n};\n%------------ COMANDOS DE TEXTO BOX ---------------------\n\\node[fancytitle, right=10pt] at (box.north west) {Deep Generative Models (2)};\n\\end{tikzpicture}\n\n\\begin{tikzpicture}\n\\node [mybox] (box){%\n\t\\begin{minipage}{0.3\\textwidth}\n\tHold dist. per latent variable instead of single val. \\textit{Benefits}: ensemble modeling (better acc), uncertainty estimates, prevent overconfidence, model compression (prior towards 0)\\\\[4pt]\n\t\\underline{Epistemic uncertainty}: dataset limits, unseen data, important for safety-critical and small datasets. Posterior $p(w|x,y)$ intractable. \\textit{MC dropout}: apply DP during test (Bernoulli-dist over weights). Var approx. uncertainty. Any NN can be made Bayesian with that, but expensive and not accurate. Can also be motivated from Gaussian Processes. Over-param. models better uncert. estm.\\\\[4pt]\n\t\\underline{Aleatoric uncertainty}: data uncertainty due to noise (e.g. bad sensor). \\textit{Data-dependent/heteroscedastic}: specific raw inputs hard to interpret, predict uncert. per data point: $\\loss = \\frac{||y_i - \\hat{y}_i||^2}{2\\sigma_i^2} + \\log \\sigma_i$. \\textit{Task-dependent/homoscedastic}: introduced by task (e.g. depth estimation), Sol: train on multiple tasks. $\\loss = \\frac{||y_i - \\hat{y}_i||^2}{2\\sigma^2} + \\log \\sigma$\n\t\\end{minipage}\n};\n%------------ COMANDOS DE TEXTO BOX ---------------------\n\\node[fancytitle, right=10pt] at (box.north west) {Bayesian Deep Learning (1)};\n\\end{tikzpicture}\n\n\\begin{tikzpicture}\n\\node [mybox] (box){%\n\t\\begin{minipage}{0.3\\textwidth}\n\t\\underline{Bayes by Backprop}: approx. true posterior $p(w|\\mathcal{D})$ by $q(w|\\theta)$: $\\loss = \\log q(w_s|\\theta) - \\log p(w_s) - \\log p(\\mathcal{D}|w_s) \\hspace{2mm}\\text{ where }\\hspace{2mm} w_s\\sim q(w_s|\\theta)$\\\\\n\tExample: assume Gaussian variational posterior with softplus $w=\\mu + \\epsilon\\cdot \\log\\left(1+\\exp\\rho\\right)$, then learn $\\mu$ and $\\rho$ by SGD.\n\t\\end{minipage}\n};\n%------------ COMANDOS DE TEXTO BOX ---------------------\n\\node[fancytitle, right=10pt] at (box.north west) {Bayesian Deep Learning (2)};\n\\end{tikzpicture}\n\n\\begin{tikzpicture}\n\\node [mybox] (box){%\n\t\\begin{minipage}{0.3\\textwidth}\n\t\\underline{Autoregressive Models}: generative without latent variables, assuming order in data, conditional probs $p(x) = \\prod_k p(x_k|x_{<k})$. Not necessarily parameter sharing, $p(x)$ tractable, but slow\\\\[4pt]\n\t\\underline{NADE}: model output with single layer, $\\mathcal{O}(D\\times H)$ params\\\\$p(x_d=1|x_{<d})=\\sigma\\left(V_{d,:}\\cdot h_d+b_d\\right)$, $h_d=\\sigma\\left(W_{:,<d}\\cdot x_{<d} + c\\right)$\\\\\n\t\\underline{MADE}: Autoencoder with carefully masked connections. $y_d$ only depends on $x_{<d}$. Connections can be shared with future $d$\\\\[4pt]\n\t\\underline{PixelRNN}: row-wise pixel and sequential color generation\\\\\n\t$p(x_i|x_{<i}) = p(x_{i,R}|x_{<i})\\cdot p(x_{i,G}|x_{i,R}, x_{<i})\\cdot p(x_{i,B}|x_{i,R}, x_{i,G}, x_{<i})$\\\\\n\t\\textit{Row-LSTM}: next output depends on three hidden states above\\\\\n\t\\textit{Diagonal-BiLSTM}: use all pixels before (all prev rows and left)\\\\\n\t\\underline{PixelCNN}: masked convs to only see top and left. Causes blind spot. Use separated vertical and horizontal stack\\\\[4pt]\n\t\\underline{PixelVAE}: Standard VAE with PixelCNN as decoder\n\t\\end{minipage}\n};\n%------------ COMANDOS DE TEXTO BOX ---------------------\n\\node[fancytitle, right=10pt] at (box.north west) {Deep Sequential Models};\n\\end{tikzpicture}\n\n\\begin{tikzpicture}\n\\node [mybox] (box){%\n\t\\begin{minipage}{0.3\\textwidth}\n\tValue function $q^{\\pi}(s_t, a_t) = \\mathbb{E}_{\\pi}\\left[r_{t+1}+\\gamma r_{t+2} + \\gamma^2 r_{t+3} + ... | s_t, a_t\\right]$\\\\\n\tBellman equation $q^{\\pi}(s_t, a_t) = \\mathbb{E}_{\\pi}\\left[r_{t+1}+\\gamma q^{\\pi}| s_t, a_t\\right]$\\\\\n\tOptimal policy with $q^{*}(s_t,a_t) = r_{t+1} + \\gamma \\max_{a_{t+1}} q(s_{t+1}, a_{t+1})$\\\\[4pt]\n\t\\underline{Value-based}: learn $q^{*}$ to get $\\pi^{*}$. Q-Learning (off-policy):\\\\\n\t$\\mathcal{L} = \\mathbb{E}\\left[\\left(r + \\gamma \\max_{a_{t+1}} q(s_{t+1}, a_{t+1}, \\theta) -q(s_{t}, a_{t}, \\theta) \\right)^2\\right]$\\\\\n\tFor gradient calculation, bootstrapped val is fixed. \\\\\n\t\\underline{Stability problems}: bootstrap, target and policy always changing, oscillations; seq. data break iid assump.; scale of $q$ values hard to control, unstable gradients; \\\\\n\t\\underline{Solutions}: experience replay (store samples $\\langle s, a, r, s'\\rangle$ in dataset, sample from that, makes batch iid), freezing target network every $K$ iters to avoid oscillations, clip rewards, skip frames, control exploration vs. exploitation by annealing $\\epsilon$-greedy policy\\\\[4pt]\n\t\\underline{Policy-based}: learn $\\pi^{*}$ directly, avoid problems with $q$ vals (especially hard for continuous action space). Training steps:\\\\\n\t\\vspace{-3mm}\n\t\\begin{enumerate}[leftmargin=4mm]\n\t\\setlength\\itemsep{0.0em}\n\t\\item Determine $q$ by simulation: $q^{\\pi_w}(s_t, a_t) = \\mathbb{E}\\left[r_t + \\gamma r_{t+1}... | \\pi_{w}\\right]$\n\t\\item Maximize $q$ by $\\pd{\\mathcal{L}}{w} = \\mathbb{E}\\left[\\chain{q^{\\pi}(s,a)}{a}{w}\\right]$ (deterministic)\\\\\n\tor $\\pd{\\mathcal{L}}{w} = \\mathbb{E}\\left[\\pd{\\log \\pi^{w}(a|s)}{w} q^{\\pi}(s,a)\\right]$ (stochastic)\n\t\\end{enumerate}\n\t\\textit{Asynchronous Advantage Actor-Critic}: Learn both policy and value function at same time, run multiple agents simultaneously (more diverse samples), advantage estimates: use learned value function to compare actually gained $q$ value. If loss is higher, unexpected (good) things happened $\\Rightarrow$ exploration\\\\[4pt]\n\t\\underline{Model-based}: try to model environment and be aware of rules. E.g. AlphaGo with tree-search guided by CNNs. Two policy networks playing against each other, and a third network to predict $V(s_t)=\\sum_{a'} \\pi(a'|s_t) \\cdot q^{\\pi}(s_t, a')$.\n\t\\end{minipage}\n};\n%------------ COMANDOS DE TEXTO BOX ---------------------\n\\node[fancytitle, right=10pt] at (box.north west) {Deep Reinforcement Learning};\n\\end{tikzpicture}\n\n\\begin{tikzpicture}\n\\node [mybox] (box){%\n\t\\begin{minipage}{0.3\\textwidth}\n\t\\underline{Forward KL}: $D_{KL}(p||q)$, overstimate variance\\\\\n\t\\underline{Backward KL}: $D_{KL}(q||p)$, underestimate variance\\\\\n\t$D_{KL}(q||p) = \\int q(x) \\log \\frac{q(x)}{p(x)}dx \\Rightarrow$ if $p(x)=0$, then $q(x)=0$\\\\\n\t$D_{KL}(p||q) = \\int p(x) \\log \\frac{p(x)}{q(x)}dx \\Rightarrow$ if $p(x)>0$, then $q(x)>0$\\\\[2pt]\n\t\\underline{Jensen-Shannon}: $D_{JS}(p||q) = \\frac{1}{2}D_{KL}(p||M)+\\frac{1}{2}D_{KL}(p||M)$\\\\ $M = \\frac{p+q}{2}\\Rightarrow D_{JS}(p||q)=D_{JS}(q||p)$\\\\[4pt]\n\t$ a = Wx+b$, $\\pd{a_i}{W_{jk}} = 1(i=j)\\cdot x_k$, $\\pd{a}{b} = \\bm{I}$, $\\pd{a}{x} = W$\\\\\n\t\\end{minipage}\n};\n%------------ COMANDOS DE TEXTO BOX ---------------------\n\\node[fancytitle, right=10pt] at (box.north west) {Math to know};\n\\end{tikzpicture}\n\n\\begin{tikzpicture}\n\\node [mybox] (box){%\n\t\\begin{minipage}{0.3\\textwidth}\n\t\\underline{Compare non-linear activation functions}\n\t% \\vspace{-3mm}\n\t\\begin{description}[leftmargin=4mm]\n\t\\setlength\\itemsep{0.0em}\n\t\\item[ReLU] Strong gradient for $x>0$, non saturating \\textit{Drawbacks}: dead neurons\n\t% \\item Every module can be expressed by $a=h(x;w)$\n\t\\item[Sigmoid] probability distribution output \\textit{Drawbacks}: small gradients $<1/4$, saturating, shifts distribution\n\t\\item[Tanh] zero-centered in origin \\textit{Drawbacks}: saturating, only strong gradients around 0\n\t\\end{description}\n\t\\end{minipage}\n};\n%------------ COMANDOS DE TEXTO BOX ---------------------\n\\node[fancytitle, right=10pt] at (box.north west) {Old Exams};\n\\end{tikzpicture}\n\n\\begin{tikzpicture}\n\\node [mybox] (box){%\n\t\\begin{minipage}{0.3\\textwidth}\n\t\\underline{Differences between generative and discriminative models}\\\\\n\t1. Generative models are used to estimate the joint probability density function $p(x)$. Discriminative models are used, instead, to model the conditional $p(y|x)$.\\\\\n\t2. Generative models are often intractable because in the $p(x)=\\int p(x|z) p(z) dz$ the integral is not always possible to analytically compute.\\\\\n\t3. Discriminative models tend to yield better accuracies given a task, meaning they are optimized for the particular task, at the cost of potential overfitting.\\\\[5pt]\n\t\\underline{Advantages/Disadvantages of generative models}\\\\\n\t\\textbf{GAN}: Very good, realistic results, fast to sample from, no need to train on likelihood, very flexible to extension \\textit{Drawbacks}: no quantitative evaluation, hard to train (sensitive to hyperparameters, mode collapse, etc.), no real objective in terms of likelihood (and distribution is unknown)\\\\\n\t\\textbf{VAE}: \\textit{Benefits}: Usable for data compression, distribution known (calculate likelihood function), stable training (no mode collapse) \\textit{Drawbacks}: only approx. likelihood (ELBO), tends to give blurry instead of realistic images, need flexible enough encoder and prior\\\\\n\t\\textbf{NF}: \\textit{Benefits}: directly optimize $p(x)$, one-to-one mapping between $z$ and $x$ (knows exact embedding of any image in latent space) \\textit{Drawbacks}: high number of parameters, complexity restrained by requirement of reversible $f$\\\\[5pt]\n\t\\underline{Difference RNN/Autoregressive}\\\\\n\t\\textbf{RNN}: shares weigths over steps, applicable to any sequence length, compresses all previous inputs into single hidden state/memory, not necessarily generative\\\\\n\t\\textbf{Autoregressive}: does not necessarily share weights, fixed in sequence length, are generative\n\t\\end{minipage}\n};\n%------------ COMANDOS DE TEXTO BOX ---------------------\n\\node[fancytitle, right=10pt] at (box.north west) {Additional questions};\n\\end{tikzpicture}\n\n\\begin{tikzpicture}\n\\node [mybox] (box){%\n\t\\begin{minipage}{0.3\\textwidth}\n\t\\begin{minipage}{0.45\\textwidth}\n\t\\includegraphics[width=\\textwidth]{figures/NN_Zoo.png}\n\t\\end{minipage}\n\t\\begin{minipage}{0.45\\textwidth}\n\t\\includegraphics[width=\\textwidth]{figures/optimization_pathological_curvatures.png}\n\t\\includegraphics[width=\\textwidth]{figures/RNN_LSTM.png}\n\t\\end{minipage}\n\t\n\t\\begin{minipage}{\\textwidth}\n\t\\includegraphics[width=\\textwidth]{figures/NF_concept.png}\n\t\\end{minipage}\n\t\n\t\\begin{minipage}{\\textwidth}\n\t\\centering\n\t\\includegraphics[width=0.7\\textwidth]{figures/Autoregressive_PixelRNN.pdf}\n\t\\end{minipage}\n\t\n\t\\begin{minipage}{\\textwidth}\n\t\\centering\n\t\\includegraphics[width=0.7\\textwidth]{figures/GAN_generative_models_overview_2.png}\n\t\\end{minipage}\n\t\\end{minipage}\n};\n%------------ COMANDOS DE TEXTO BOX ---------------------\n\\node[fancytitle, right=10pt] at (box.north west) {Figures};\n\\end{tikzpicture}\n\n\\end{multicols*}\n\\end{document}"
  },
  {
    "path": "Deep_Learning/dl_appendix.tex",
    "content": "% \\section{Neural Network Zoo}\n\n\\begin{figure}[ht!]\n\t\\centering\n\t\\includegraphics[width=0.9\\textwidth]{figures/NN_Zoo_High.png}\n\\end{figure}"
  },
  {
    "path": "Deep_Learning/dl_autoregressive.tex",
    "content": "\\section{Deep Sequential Models}\n\\subsection{Autoregressive Models}\n\\begin{itemize}\n\t\\item Generative models without latent variables, but assuming an order in the data (if there is no, create an artificial order like image from left to right, top to bottom). The likelihood is the product of conditionals:\n\t$$p(x)=\\prod_{k=1}^{D} p(x_k|x_{j<k})$$\n\t\\item In contrast to RNNs, there is no/not necessarily parameter sharing, and the chain cannot be of infinite length because of that\n\t\\item \\textit{Advantages}: $p(x)$ is tractable\n\t\\item \\textit{Drawbacks}: training and generation is slow due to being sequential and not parallel\n\\end{itemize}\n\\subsubsection{NADE}\n\\begin{itemize}\n\t\\item Originally defined for binary inputs/data. Can be generalized for other spaces as well\n\t\\item Every output $x_d$ is modeled by a single layer that takes as input all previous data points, and generates based on that it's prediction:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tp(x_d=1|x_{<d}) & = \\sigma\\left(V_{d,:}\\cdot h_d + b_d\\right), h_d = \\sigma\\left(W_{:,<d}\\cdot x_{<d} + c\\right)\n\t\t\\end{split}\n\t\\end{equation*}\n\twhere $V\\in \\mathbb{R}^{D\\times H}, W\\in \\mathbb{R}^{H\\times D}, b\\in \\mathbb{R}^{D}, c\\in \\mathbb{R}^{H}$ ($H$ hidden dimensionality, $D$ input dimensions)\n\t\\item Objective is minimizing log likelihood: $\\mathcal{L} = - \\log p(x) = - \\sum_{k=1}^{D} p(x_k|x_{<k})$\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.6\\textwidth]{figures/Autoregressive_NADE.pdf}\n\t\t\\caption{Concept of NADE.}\n\t\\end{figure}\n\t\\item \\textit{Teacher forcing}: During training, use ground truth as input for all levels. For testing, use generated samples as input (sequentially)\n\\end{itemize}\n\\subsubsection{MADE}\n\\begin{itemize}\n\t\\item Use an autoencoder where we carefully mask out connections so that the output $y_d$ only depends on inputs $x_{<d}$\n\t\\item Name ``autoencoder'' is only because we try to reproduce the input. However, note that we neither have a bottleneck nor we try to get sparsity. We just remove connections to make the outputs depending on certain inputs\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/Autoregressive_MADE.png}\n\t\t\\caption{Masked autoencoder for autoregressive models. We set certain weights to 0 (i.e. remove connections between neurons) so that the generation of $x_1$ only depends on $x_2$ and $x_3$, but not on $x_1$ itself (which would be cheating and prevent the model of being generative).}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{PixelRNN}\n\\begin{itemize}\n\t\\item Assume row-wise pixel and sequential color generation (first red channel, then green, afterwards blue):\n\t$$p(x_i|x_{<i}) = p(x_{i,R}|x_{<i})\\cdot p(x_{i,G}|x_{i,R}, x_{<i})\\cdot p(x_{i,B}|x_{i,R}, x_{i,G}, x_{<i})$$\n\t\\item Different ways of modeling it. LSTM variants mostly have 12 layers\n\t\\begin{itemize}\n\t\t\\item \\textit{Row LSTM}: to compute next output (i.e. next hidden state), we take into consideration the three hidden states of the row above a certain pixel as ``last hidden state''. We get therefore a tri-angular shape of context. However, it thereby misses context from the row itself, and further away context. As it does not use pixels in the same row, the computation can be parallelized for a row. \n\t\t\\item \\textit{Diagonal Bi-LSTM}: Uses all pixels that were generated before by using a Bi-LSTM. \n\t\\end{itemize}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/Autoregressive_PixelRNN.pdf}\n\t\t\\caption{Comparing different methods of PixelRNN and PixelCNN. The lower level is the previous layer, and the top is the next layer. If we have a single layer PixelRNN/CNN, the lower one would be the input and the upper the generated output.}\n\t\t\\label{fig:Autoregressive_PixelRNN}\n\t\\end{figure}\n\t\\item The architecture includes residual connections to speed up training\n\t\\item \\textit{Benefits}: good modeling of $p(x)$, reasonable image quality\n\t\\item \\textit{Disadvantages}: slow training and slow generation\n\\end{itemize}\n\\subsubsection{PixelCNN}\n\\begin{itemize}\n\t\\item Replace recurrence by convolutions to speed up (at least) training\n\t\\item Convolutions are masked so that only context from before (i.e. left and top) can be used. See Figure~\\ref{fig:Autoregressive_PixelRNN} left and Figure~\\ref{fig:Autoregressive_PixelCNN} for an example\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\begin{subfigure}{0.3\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.6\\textwidth]{figures/Autoregressive_Masked_Conv.png}\n\t\t\t\\caption{Example mask for $5\\times 5$ convolution}\n\t\t\\end{subfigure}\n\t\t\\hspace{2mm}\n\t\t\\begin{subfigure}{0.32\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=\\textwidth]{figures/Autoregressive_PixelCNN_blindspot_problem.png}\n\t\t\t\\caption{Blindspot of PixelCNN}\n\t\t\\end{subfigure}\n\t\t\\hspace{2mm}\n\t\t\\begin{subfigure}{0.32\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=\\textwidth]{figures/Autoregressive_PixelCNN_blindspot.jpg}\n\t\t\t\\caption{Solution to blindspot}\n\t\t\\end{subfigure}\n\t\t\\caption{Masked convolutions in PixelCNN}\n\t\t\\label{fig:Autoregressive_PixelCNN}\n\t\\end{figure}\n\t\\item Problem: worse results than PixelRNN because of limited context and blind spot (cascaded convolutions ignore right upper part)\n\t\\item Solution: use two convolutions, one vertical stack looking purely on the top part, and the horizontal stack looking to the right. Additionally, use gated convolutions (one half of the features go through tanh, the other through sigmoid)\n\t\\item \\textbf{PixelCNN++}: replace softmax with logistic mixture likelihood over 8 bits, use encoder-decoder architecture with skip connections\n\\end{itemize}\n\\subsubsection{PixelVAE}\n\\begin{itemize}\n\t\\item Standard VAE with PixelCNN as decoder/generator\n\t\\item However, generator is very powerful which can lead to the problem that it ignores the latent code, and just generates ``nice'' images\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.3\\textwidth]{figures/Autoregressive_PixelVAE.png}\n\t\t\\caption{Architecture of a PixelVAE}\n\t\\end{figure}\n\\end{itemize}"
  },
  {
    "path": "Deep_Learning/dl_bayesian.tex",
    "content": "\\section{Bayesian Deep Learning}\n\\begin{itemize}\n\t\\item Bayesian machine learning: holding a distribution per latent variable instead of single value\n\t\\item Benefits of Bayesian\n\t\\begin{itemize}\n\t\t\\item Ensemble modeling (better accuracies)\n\t\t\\item Uncertainty estimates, preventing overconfident networks\n\t\t\\item Model compression (have prior that pushes weights towards 0)\n\t\t\\item \\TODO{Think of more}\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Epistemic uncertainty}\n\\begin{itemize}\n\t\\item \\textit{Epistemic uncertainty}: dataset limits\n\t\\item Uncertainty that is introduced by dataset limits (unseen data $\\Rightarrow$ how certain are the weights)\n\t\\item Can be reduced by increasing the amount of data\n\t\\item Important for safety-critical applications and small datasets\n\t\\item Hard to model because posterior is usually intractable for complex functions like NN\n\t$$p(w|x,y) = \\frac{p(x,y|w)p(w)}{\\int p(x,y|w)p(w)dw}$$\n\t\\item \\textbf{Monte-Carlo Dropout}: apply dropout during testing (Bernoulli-distribution over weights as variational distribution). The variance/uncertainty derived from there approximates uncertainty gained by variational framework. \n\t\\begin{itemize}\n\t\t\\item \\textit{Advantages}: every standard NN can be turned into a Bayesian NN. Very easy to train and no inference network necessary\n\t\t\\item \\textit{Drawbacks}: expensive, have to rerun model several times on data. Not very accurate (depends on activation function etc.)\n\t\\end{itemize}\n\t\\item \\textbf{Deep Gaussian Process}: predict mean and variance for every data point.\n\t\\begin{itemize}\n\t\t\\item The predictive distribution is $p(y|x,X,Y) = \\int p(y|x,w)p(w|X,Y)dw$\n\t\t\\item The likelihood term is a Gaussian $p(y|x,w)=\\mathcal{N}(y; \\hat{y}(x,w), \\tau^{-1}I_D)$ where $\\hat{y}(x,w)$ is a NN and $\\tau^{-1}$ the model precision that can be derived from MC dropout\n\t\t\\item For the posterior, we use variational approximation: $p(w|X,Y)\\approx q(w)$. In case of MC dropout, we have $\\tilde{W}_i = W_i\\cdot \\text{diag}\\left(\\left[z_{i,j}\\right]_{1}^{K_i}\\right), z_{i,j}\\sim \\text{Bernoulli}\\left(p_i\\right)$ where $\\tilde{W}_i$ are the weights with applied dropout\n\t\t\\item Minimize loss $\\mathcal{L}= - \\int q(w)\\log p(Y|X,w)dw + KL\\left(q(w)||p(w|X,Y)\\right)$. First term is approximated by Monte-Carlo integration (equivalent to sampling dropout), and second can be approximated analytically\n\t\\end{itemize}\n\t\\item Over-paramterized models give better uncertainty estimates as they capture bigger class of models. However, they also need higher dropout rates\n\\end{itemize}\n\\subsection{Aleatoric uncertainty}\n\\begin{itemize}\n\t\\item \\textit{Aleatoric uncertainty}: data uncertainty\n\t\\item Uncertainty due to the nature of data (noise/hard to predict accurate. Example: depth estimation with bad sensor)\n\t\\item Can be reduced by better data (better sensors, multiple different sensors, etc.)\n\t\\item \\textit{Data-dependent/heteroscedastic aleatoric uncertainty}: specific raw inputs like images that are hard to interpret\n\t\\begin{itemize}\n\t\t\\item Can be modeled by predicting a variance term per data point to reduce loss\n\t\t$$\\mathcal{L} = \\frac{\\lVert y_i - \\hat{y}_i\\rVert^2}{2\\sigma_i^2} + \\log \\sigma_i$$\n\t\tIf variance low, the loss is weighted higher, but the $\\log$ term is smaller $\\Rightarrow$ trade-off\n\t\\end{itemize}\n\t\\item \\textit{Task-dependent/homoscedastic aleatoric uncertainty}: introduced by task like semantic segmentation or depth estimation (hard at edges). Possible solution: train on multiple tasks like edge detection\n\t\\begin{itemize}\n\t\t\\item We can as well introduce a variance term, but shared by all data points (task individual):\n\t\t$$\\mathcal{L} = \\frac{\\lVert y_i - \\hat{y}_i\\rVert^2}{2\\sigma^2} + \\log \\sigma$$\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Bayes by Backprop}\n\\begin{itemize}\n\t\\item Start from a NN with a distribution over its weights\n\t\\item Train weights to approximate the true posterior well (similar to ELBO just with $p(\\mathcal{D})=1 \\Rightarrow \\log p(\\mathcal{D}) = 0$)\n\t$$\\text{KL}\\left(q\\left(w|\\theta\\right)||p\\left(w|\\mathcal{D}\\right)\\right) = \\text{KL}\\left(q\\left(w|\\theta\\right)||p\\left(w\\right)\\right) - \\int q(w|\\theta) \\log p(\\mathcal{D}|w)dw$$\n\tFirst term pushes distributions towards prior, and second towards modeling the data well\n\t\\item Compute by Monte-Carlo integration (over distribution $q(w|\\theta)$) for \\textit{both} terms:\n\t$$\\mathcal{L} = \\log q(w_s|\\theta) - \\log p(w_s) - \\log p(\\mathcal{D}|w_s) \\hspace{2mm}\\text{ where }\\hspace{2mm} w_s\\sim q(w_s|\\theta)$$\n\t\\item Example: assume a Gaussian variational posterior on the weights $w=\\mu + \\epsilon \\cdot \\log(1 + \\exp\\rho))$ (standard deviation with softplus trick for always positive values). Learn parameters $\\mu$ and $\\rho$ per weight\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/Bayes_By_Backprop.png}\n\t\\end{figure}\n\t\\item In experiments, Bayesian NNs perform similar to plain NNs with dropout\n\\end{itemize}"
  },
  {
    "path": "Deep_Learning/dl_convnets.tex",
    "content": "\\section{Convolutional Neural Networks}\n\\begin{itemize}\n\t\\item Images are stationary signals with spatial structure and huge dimensionality\n\t\\item Input dimensions are highly correlated (e.g. translation invariant)\n\t\\item Preserve spatial structure by convolutional filters, local connectivity (with shared weights) and being robust to local variances by spatial pooling\n\\end{itemize}\n\\subsection{Transfer Learning}\n\\begin{itemize}\n\t\\item Use large datasets like ImageNet to learn useful features for other, smaller datasets\n\t\\item Prevent overfitting, even for large networks\n\t\\item Alternatively, we could also use a pre-trained network on task 1 as feature extractor for task 2 (same as freezing first layers)\n\t\\item Which layer(s) to fine-tune?\n\t\\begin{itemize}\n\t\t\\item If both task have the same labels, we can initialize all layers. Otherwise, the classification layer (last layer) must be newly trained. If there is only very few data available, only fine-tune this layer\n\t\t\\item If datasets are very different, the fully connected layers need to be replaced\n\t\t\\item First convolutional filters capture low-level information that mostly does not change over datasets. Mid-level convolutions can be fine-tuned if dataset is large enough\n\t\\end{itemize}\n\t\\item Use a smaller learning rate for pre-initialized layers as network starts already from a point close to the optimum. New layers can be trained with higher learning rate\n\\end{itemize}\n\\subsection{Standard classification architectures}\n\\subsubsection{VGGNet}\n\\begin{itemize}\n\t\\item All filter sizes are $3\\times 3$, as this is the smallest filter size, and is more parameter efficient to build up large filters, plus additional non-linearity between filters\n\t\\item $1\\times 1$ convolutions used to increase non-linearity/complexity without increasing receptive field\n\\end{itemize}\n\\begin{figure}[ht!]\n\t\\centering\n\t\\includegraphics[width=0.3\\textwidth]{figures/CNN_VGGnet.png}\n\t\\caption{VGG16 architecture}\n\t\\label{fig:CNN_VVGnet}\n\\end{figure}\n\\subsubsection{Inception}\n\\begin{itemize}\n\t\\item Receptive fields should vary in size as objects can appear in different scales\n\t\\item Naively stacking more convolutional operations on top of each other is expensive and prone to overfitting\n\t\\item Inception module applies different filter sizes on same input ($1\\times 1$ convolutions for feature reduction)\n\t\\item Architecture consists of 9 Inception blocks\n\t\\item Solution for vanishing gradients: have intermediate classifiers that amplify the gradient signal for early layers\n\t\\item InceptionV2: $5\\times 5$ replaced by two $3\\times 3$ filters\n\t\\item InceptionV3: $1\\times 3$ and $3\\times 1$ filters instead of $3\\times 3$\n\t\\item BatchNormalization has shown to be very helpful in this architecture\n\\end{itemize}\n\\begin{figure}[ht!]\n\t\\centering\n\t\\includegraphics[width=0.8\\textwidth]{figures/CNN_Inception_module.pdf}\n\t\\caption{Inception module}\n\t\\label{fig:CNN_Inception_module}\n\\end{figure}\n\\subsubsection{ResNet/DenseNet/HighwayNet}\n\\begin{itemize}\n\t\\item Deeper networks are harder to optimize, and might actually achieve worse results than shallow ones because of that (although learning identity in additional layers must lead to same results)\n\t\\item Better approach: try to model the difference that is learned in every layer $H(x) = F(x) + x$\n\t\\item Different ways for modeling $F(x)$. Most popular ones shown in Figure~\\ref{fig:CNN_ResNet_blocks}. BatchNormalization has been shown to be very important because of vanishing gradients\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.7\\textwidth]{figures/CNN_ResNet_blocks.png}\n\t\t\\caption{ResNet blocks}\n\t\t\\label{fig:CNN_ResNet_blocks}\n\t\\end{figure}\n\t\\item \\textbf{HighwayNet} introduces a gate with learnable parameters to determine the importance of a layer: $H(x) = F(x) \\cdot T(x) + x \\cdot \\left(1 - T\\left(x\\right)\\right)$\n\t\\item \\textbf{DenseNet} uses skip connections to multiple forward layers. Creates complex blocks where last layer sees the input of all previous layers\n\\end{itemize}\n\\subsection{Tracking/Object detection}\n\\subsubsection{Fast R-CNN}\n\\begin{itemize}\n\t\\item Based on middle feature map, get bounding boxes by e.g. selective search \n\t\\item RoI pooling returns fixed size feature map for selected bounding box (puts e.g. $3\\times 3$ mask on features and pools accordingly)\n\t\\item Features used to generate class prediction and location correction\n\t\\item During training, sample multiple candidate boxes from image and train on all of them. Makes it more efficient/faster, \\textit{but} batch elements might be highly correlated (in the paper, they report that they experienced it to be neglectable)\n\t\\item Very accurate and fast, but external box proposals needed\n\t\\item \\textbf{Faster R-CNN}: train network to propose box locations\n\\end{itemize}\n\\subsubsection{Siamese Network for Training}\n\\begin{itemize}\n\t\\item Use Siamese network to compare similarity of two patches\n\t\\item If we compare patches over time, we can find objects with the highest similarity $\\Rightarrow$ tracking of objects\n\t\\item Can be trained on rich video dataset, and can be applied to unseen categories/targets\n\\end{itemize}\n\\subsection{Spatial Transformer Network}\n\\begin{itemize}\n\t\\item ConvNets must be invariant/robust to pose/geometry changes. One simple way of doing it is data augmentation\n\t\\item Better: use spatial transformer network to learn rotation/scale transformation\n\t\\item Define grid on input. Scale, translation and rotation parameters are learned by the network and depend on the input. Finally, transform image based on the changed grid. \n\t\\item Operation is differentiable and thus can be learned\n\\end{itemize}"
  },
  {
    "path": "Deep_Learning/dl_deep_rl.tex",
    "content": "\\section{Deep Reinforcement Learning}\n\\subsection{Fundamentals of Reinforcement Learning}\n\\begin{itemize}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/RL_basic_concept.png}\n\t\t\\caption{Interaction model between environment and agent}\n\t\\end{figure}\n\t\\item The \\textbf{state} $s_t$ is the summary of all experience so far: $s_t = f(o_1, r_1, a_1, o_2, r_2, a_2, ..., o_t, r_t)$ ($o_i$ observable part of environment at time step $i$). If we have a fully observable environment, then $s_t = f(o_t)$.\n\t\\item The \\textbf{policy} of an agent determines its actions: $\\pi\\left(a_t|s_t\\right)$. Can be deterministic or stochastic\n\t\\item The \\textbf{value function} is the expected total reward under policy $\\pi$: $$q^{\\pi}(s_t, a_t) = \\mathbb{E}_{\\pi}\\left[r_{t+1}+\\gamma r_{t+2} + \\gamma^2 r_{t+3} + ... | s_t, a_t\\right]$$\n\t$\\gamma$ as discount factor as we are most certain about close rewards and sometimes are more interested in immediate rewards\n\t\\item \\textbf{Bellman equation} for value function:\n\t$$q^{\\pi}(s_t, a_t) = \\mathbb{E}_{s', a'}\\left[r + \\gamma q^{\\pi}\\left(s', a'\\right) | s_t, a_t\\right] = \\sum_{s'} p(s'|s_t,a_t)\\cdot \\left[r(s', a_t, s_t) + \\gamma \\sum_{a'} \\pi(a'|s') \\cdot q^{\\pi}\\left(s', a'\\right) \\right]$$\n\t\\item The optimal value function is therefore $q^{*}(s_t,a_t) = \\max_{\\pi} q^{\\pi}(s_t,a_t) = r_{t+1} + \\gamma \\max_{a_{t+1}}$\n\t\\item The \\textbf{environment} can be modeled by the agent (learned from experience), and used for planning and look ahead. This can be for example a simulator\n\\end{itemize}\n\\subsection{Deep RL approaches}\n\\subsubsection{Value-based approaches}\n\\begin{itemize}\n\t\\item Try to learn value function $q^*$ to get the optimal policy $\\pi^*$\n\t\\item The input to such models is usually the state, which should be as raw as possible (e.g. image frames). We can either add the action to the input and let the network predict its Q-value, or predict Q-values for all possible actions (second is faster and simpler)\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/RL_deep_QLearning.png}\n\t\t\\caption{Modeling of Q-value predictions}\n\t\\end{figure}\n\t\\item Optimization by SARSA-like loss:\n\t$$\\mathcal{L} = \\mathbb{E}\\left[\\left(r + \\gamma \\max_{a_{t+1}} q(s_{t+1}, a_{t+1}, \\theta) -q(s_{t}, a_{t}, \\theta) \\right)^2\\right]$$\n\t\\item For the gradients, we assume that the bootstrapped max value is fixed:\n\t$$\\pd{\\mathcal{L}}{\\theta} = \\mathbb{E}\\left[-2\\cdot \\left(r + \\gamma \\max_{a_{t+1}} q(s_{t+1}, a_{t+1}, \\theta) -q(s_{t}, a_{t}, \\theta) \\right) \\cdot \\pd{q(s_{t}, a_{t}, \\theta)}{\\theta}\\right]$$\n\t\\item Optimize with SGD by sampling one action and state, calculate q-values for all possible future actions, and use the maximum as bootstrap goal\n\\end{itemize}\n\\subsubsection{Stability problems}\n\\begin{itemize}\n\t\\item As we bootstrap, the target is always changing $\\Rightarrow$ policy changes fast, can lead to oscillations\n\t\\item The sequential data breaks the iid assumption on which SGD relies\n\t\\item The scale of Q-values is not easy to control, and is very task dependent $\\Rightarrow$ gradients are unstable and can be either too large or too small\n\t\\item \\textbf{Improving stability}\n\t\\begin{itemize}\n\t\t\\item \\textit{Experience replay}: store memories of $\\langle s, a, r, s'\\rangle$ (with e.g. a $\\epsilon$-greedy policy) in a dataset, and sample batches from there to train on. Breaks temporal dependency and helps SGD by i.i.d.\n\t\t\\item \\textit{Freezing target}: instead of having a moving target, we freeze the $Q$ network every $K$ iterations, and use that to generate our targets (Q-targets come now from a bit older network parameter setting, but is steady over $K$ iterations). Avoids oscillations\n\t\t\\item \\textit{Clipping rewards}: Normalize or clip rewards to be in range $[-1,+1]$ or any other stable range. Prevents unknown scales of $Q$\n\t\t\\item \\textit{Skipping frames}: a light version of experience replay is skipping $N$ frames between two data points to avoid too strong temporal dependency (two consecutive frames are very similar)\n\t\t\\item \\textit{Exploration vs Exploitation}: use a $\\epsilon$-greedy policy with annealing temperature. In the beginning, we will focus on exploration while slowly converging to exploitation\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Policy-based approaches}\n\\begin{itemize}\n\t\\item Try to learn the optimal policy $\\pi^*$ directly from experience (parameterized policy $\\pi_w(a_t|s_t)$)\n\t\\item Avoids learning the $q$ values which are hard for continuous action spaces, and tend to oscillate because of bootstrapping\n\t\\item Training steps\n\t\\begin{enumerate}\n\t\t\\item Determine Q-value for current policy by running a simulation:\\\\ $q^{\\pi_w}(s_t, a_t) = \\mathbb{E}\\left[r_t + \\gamma r_{t+1} + \\gamma^2 r_{t+2} + ... | \\pi_{w}\\right]$\n\t\t\\item Maximize q-values as loss function. \n\t\t\\begin{enumerate}\n\t\t\t\\item If policy is deterministic:\n\t\t\t$$\\pd{\\mathcal{L}}{w} = \\mathbb{E}\\left[\\chain{q^{\\pi}(s,a)}{a}{w}\\right]$$\n\t\t\t\\item If policy is stochastic:\n\t\t\t$$\\pd{\\mathcal{L}}{w} = \\mathbb{E}\\left[\\pd{\\log \\pi^{w}(a|s)}{w} q^{\\pi}(s,a)\\right]$$\n\t\t\\end{enumerate}\n\t\\end{enumerate}\n\t\\item Asynchronous Advantage Actor-Critic\n\t\\begin{itemize}\n\t\t\\item Learn both policy and value function\n\t\t\\item Multiple agents that simultaneously interact with (copy of) environment and learn\n\t\t\\item \\textit{Advantage estimates}: Use the learned value function to compare to your actually gained $q$ value. Loss is therefore higher if unexpected things happen $\\Rightarrow$ exploration\n\t\\end{itemize}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\begin{subfigure}{0.45\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.8\\textwidth]{figures/RL_A3C_multiple_workers.png}\n\t\t\\end{subfigure}\n\t\t\\begin{subfigure}{0.45\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.8\\textwidth]{figures/RL_A3C_cycle.png}\n\t\t\\end{subfigure}\n\t\t\\caption{Schematic overview of A3C}\n\t\t\\label{fig:RL_A3C}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{Model-based approaches}\n\\begin{itemize}\n\t\\item Try to model the environment to be aware of rules etc. \n\t\\item Example: AlphaGo relies on Tree-Search guided by CNNs. We use two policy networks to play against each other, and one value network that predicts the value function of a state\n\\end{itemize}"
  },
  {
    "path": "Deep_Learning/dl_generative_models.tex",
    "content": "\\section{Deep Generative Models}\n\\begin{itemize}\n\t\\item \\textit{Generative modeling}: learn the joint probability $p(x,y)$ or density function $p(x)$. Task can be performed with Bayes rule: $p(y|x)$. Generalize better (less prone to overfitting), and better modeling of causal relations. Members include GAN, VAE, etc.\n\t\\begin{itemize}\n\t\t\\item We can use generative models to predict uncertainty and out of distribution examples: $p(x,y) = p(y|x)p(x) \\Rightarrow$ if $x$ o.o.d., then $p(x)$ low!\n\t\\end{itemize}\n\t\\item \\textit{Discriminative modeling}: learn conditional pdf $p(y|x)$. Is usually task-oriented and gets better results. \n\t\\item Applications of generative models\n\t\\begin{itemize}\n\t\t\\item Simulating possible futures for reinforcement learning\n\t\t\\item Creating missing data  (e.g. pixel patches which are missing)\n\t\t\\item Super-resolution scaling for images\n\t\t\\item Data augmentation (replace e.g. car by bicyclist in a scene)\n\t\t\\item Cross-modal translation (sketch to image)\n\t\\end{itemize}\n\t\\item Different type of generative models (see Figure~\\ref{fig:GAN_generative_models_overview})\n\t\\begin{itemize}\n\t\t\\item \\textit{Explicit density}: maximize log likelihood of the data by modeling a probability density function. Function must be complex enough and computationally tractable\n\t\t\\item \\textit{Implicit density}: no explicit pdf needed, only a sampling mechanism\n\t\\end{itemize}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.45\\textwidth]{figures/GAN_generative_models_overview_2.png}\n\t\t\\caption{Overview of generative models}\n\t\t\\label{fig:GAN_generative_models_overview}\n\t\\end{figure}\n\\end{itemize}\n\\subsection{Generative Adversarial Networks}\n\\begin{itemize}\n\t\\item Adversarial training of generator vs discriminator\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.45\\textwidth]{figures/GAN_pipeline.png}\n\t\t\\caption{Pipeline of adversarial GAN training}\n\t\t\\label{fig:GAN_pipeline}\n\t\\end{figure}\n\t\\item The generator is a (mostly deconvolutional) network that takes noise $z$ as input, and creates fake images. The discriminator tries to distinguish between fake and real images\n\t\\item Trained in a minimax game fashion, the loss function resembles the Jensen-Shannon divergence:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\min_G \\max_D V(G,D) & = \\mathbb{E}_{\\bm{x}\\sim p_{\\text{data}}(\\bm{x})} \\left[\\log \\left(D\\left(\\bm{x}\\right)\\right)\\right] + \\mathbb{E}_{\\bm{z}\\sim p_{z}(\\bm{z})} \\left[\\log\\left(1 - D\\left(G\\left(\\bm{z}\\right)\\right)\\right)\\right] \\\\\n\t\t\tJ^{(D)} & = - \\frac{1}{2}\\mathbb{E}_{x\\sim p_{\\text{data}}}\\left[\\log D(x)\\right] - \\frac{1}{2}\\mathbb{E}_{z\\sim p_{z}}\\left[\\log 1 - D(G(z))\\right]\\\\\n\t\t\tJ^{(G)} & = - \\frac{1}{2}\\mathbb{E}_{z\\sim p_{z}}\\left[\\log D(G(z))\\right]\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Loss of generator is changed from $\\log 1 - D(G(z))$ because otherwise the gradients of the generator vanish for a too strong discriminator \n\t\\item Divergence is important and can strongly influence the behavior of model\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tD_{KL}\\left(p(x)\\lVert q^{*}(x)\\right) = \\int p(x) \\log \\frac{p(x)}{q^{*}(x)} dx & \\implies \\text{if } p(x)>0, \\text{ then } q(x)>0\\\\\n\t\t\tD_{KL}\\left(q^{*}(x)\\lVert p(x)\\right) = \\int q^{*}(x) \\log \\frac{q^{*}(x)}{p(x)} dx & \\implies \\text{if } p(x)=0, \\text{ then } q(x)=0\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\\end{itemize}\n\\subsubsection{GAN training problems}\n\\begin{itemize}\n\t\\item \\textbf{Vanishing gradients} during training:\n\t\\begin{itemize}\n\t\t\\item If the discriminator is too bad, the generator does not get valid/accurate feedback and can therefore not learn properly\n\t\t\\item If the discriminator is perfect, the generator has very low gradients as a small change does not influence the discriminator\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.4\\textwidth]{figures/cv_deep_learning_GAN_vanishing_gradients.jpeg}\n\t\t\t\\caption{Vanishing gradients problem for training with KL-divergence. When the distance between the two distributions $p$ and $q$ (respectively $P_g$ and $P_r$) is too huge, the KL divergence is very close to zero. Hence, is does not provide any strong gradients in these regions.}\n\t\t\\end{figure}\n\t\\end{itemize}\n\t\\item \\textbf{Reaching the equilibrium}\n\t\\begin{itemize}\n\t\t\\item We know that the nash equilibrium of the minimax game is $P_g=P_r$ meaning the distribution of the real data is equal to the generated data. In that case, $D$ return 0.5 no matter what example we put in (as both distributions are equal).\n\t\t\\item However, it has been shown that such cost functions may not converge when using gradient descent. An example is shown in Figure~\\ref{fig:GAN_reaching_equilibrium}.\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.4\\textwidth]{figures/cv_deep_learning_GAN_oscillating.png}\n\t\t\t\\caption{Oscillating behavior of a non-cooperative game where $\\min_x \\max_y V(x,y) = x\\cdot y$. The equilibrium $x=y=0$ is never reached.}\n\t\t\t\\label{fig:GAN_reaching_equilibrium}\n\t\t\\end{figure}\n\t\\end{itemize}\n\t\\item \\textbf{Mode collapse}\n\t\\begin{itemize}\n\t\t\\item A GAN suffers from a mode collapse if the generator limits its predictions/generated distribution to a few samples/modes.\n\t\t\\item For example in case of the MNIST dataset, this would mean that the generator only creates numbers of one or two different digits. Although a full mode collapse is rarely the case, partial mode collapses frequently occur\n\t\t\\item In order to create a mode collapse, the gradients regarding the noise $\\bm{z}$ must be very low/close to zero. This can for example happen if we fix the discriminator and the generator converges to the optimal image $\\bm{x}^*$ that fools the discriminator the most\n\t\t\\item Once the generator collapse to one mode, the discriminator will learn that this mode is purely/mostly generated and thus changes its predictions. The generator will address that by changing the mode (note that as $\\partial L/\\partial \\bm{z}\\approx 0$, we will just collapse to the next mode and are not able to escape this loop).\n\t\t\\item In the end, this turns into a cat-and-mouse game between the generator and discriminator, and will not converge (see Figure~\\ref{fig:GAN_mode_collapse}).\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.6\\textwidth]{figures/cv_deep_learning_GAN_mode_collapse.png}\n\t\t\t\\caption{\\textit{Top row}: optimal convergence of generator distribution to 8 modes. \\textit{Bottom row}: Sample of a mode collapse after 10k iterations. The generator is only able to generate a single mode.}\n\t\t\t\\label{fig:GAN_mode_collapse}\n\t\t\\end{figure}\n\t\\end{itemize}\n\t\\item \\textbf{Low dimensional support}\n\t\\begin{itemize}\n\t\t\\item The KL and JS divergence work best for overlapping distributions as neither of them is 0 (numerical instability)\n\t\t\\item However, during training, the training distribution is not perfect, and as we have high dimensional data, both distributions are less likely to overlap much\n\t\t\\item Also, it is easy for the discriminator to find a line in between them\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{GAN improvements}\n\\begin{itemize}\n\t\\item \\textbf{Wasserstein GAN}\n\t\\begin{itemize}\n\t\t\\item Instead of KL/JS, use Wasserstein (Earth Mover's) Distance:\n\t\t$$\\mathcal{W}(p_r, p_g) = \\inf\\limits_{\\gamma \\sim \\prod (p_r,p_g)} \\mathbb{E}_{(x,y)\\sim \\gamma}|x-y|$$\n\t\t\\item Intuitive explanation: how much do I have to move from one distribution to get the other one. Thus, the distance is even meaningful for non-overlapping distributions\n\t\\end{itemize}\n\t\\item \\textbf{Usage of labels}\n\t\\begin{itemize}\n\t\t\\item Learning a conditional model $p(y|x)$ often generates better samples than from a random distribution\n\t\t\\item One example are conditional GANs where we have given a ground truth\n\t\\end{itemize}\n\t\\item \\textbf{Label smoothing}\n\t\\begin{itemize}\n\t\t\\item Train the discriminator to predict $D(x)\\approx 1 - \\alpha$ instead of 1\n\t\t\\item Has been shown to be a good regularization by preventing the discriminator to be overconfident\n\t\t\\item In addition, the gradients of the generator do less likely explode\n\t\\end{itemize}\n\t\\item \\textbf{Virtual batch normalization}\n\t\\begin{itemize}\n\t\t\\item Batch Normalization can significantly help in neural networks\n\t\t\\item However, in GANs, it leads to high intra-batch correlation\n\t\t\\item Solution: \\textit{virtual batch normalization} where we select a reference batch which is fixed during training, and combine it with the statistics of the current batch. Reduces overfitting on reference batch and intra-batch correlation\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{GAN open questions}\n\\begin{itemize}\n\t\\item \\textbf{Mode collapse}: How to prevent a model to suffer from mode collapse. One idea is penalizing the model is features are too similar, or allowing discriminator to see across batch elements. But these solutions are more heuristic tries and no theoretical solution\n\t\\item \\textbf{Evaluation of GANs}: GANs are currently judged by their qualitative results/predictions, but there is no quantitative measurement yet\n\t\\item \\textbf{Discrete outputs}: The generator and discriminator need to be differentiable, and thus discrete outputs are not possible. There are some workarounds, but no real theoretically sound solution.\n\t\\item \\textbf{Semi-supervised classification}: How to combine a GAN training and discriminative model efficiently (discriminator predicts class and fake/real at the same time)\n\\end{itemize}\n\\subsection{Boltzmann machines}\n\\begin{itemize}\n\t\\item A Boltzmann distribution is defined by $p(x) = \\frac{1}{Z}\\exp\\left(-E\\left(x\\right)\\right)$ where $E(x)$ is a energy function described by our model, and $Z=\\sum\\limits_x \\exp\\left(E\\left(x\\right)\\right)$ a normalization constant\n\t\\item The benefit of defining a distribution like that is that our model can use any output values between $[-\\infty, \\infty]$ instead of being constrained to $[0,1]$\n\t\\item A problem is that even if $x$ is binary, the normalizing constant $Z$ gets out of hands (sum over $2^{n}$ combinations for $n$ dimensional $x$). Thus, we limit the computations by only considering pairwise relations\n\t\\item Pairwise relations modeled by $E(x)=-x^TWx-b^Tx$. Learning $W$ and $b$ by maximizing the likelihood of the data\n\t\\item Problem: $W$ is still of size $n^2$ which can be too large for e.g. images ($256\\times 256$ leads to $4.2$ billion parameters in $W$) $\\Rightarrow$ Restricted Boltzmann machines\n\\end{itemize}\n\\subsubsection{Restricted Boltzmann machines}\n\\begin{itemize}\n\t\\item Restrict model by additional bottleneck over $h$ latents\n\t$$E(x,h) = -x^T W h - b^T x - c^T h, \\hspace{2mm} p(x) = \\frac{1}{Z}\\sum_h \\exp\\left(-E\\left(x,h\\right)\\right)$$\n\t\\item This function is not in the form of a energy function anymore (because of the sum). We can rewrite it as:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tF(x) & = -b^T x - \\sum_i \\log \\sum_{h_i} \\exp\\left(h_i\\left(c_i + W_i x\\right)\\right)\\\\\n\t\t\tp(x) & = \\frac{1}{Z} \\exp\\left(-F(x)\\right)\\\\\n\t\t\tZ & = \\sum\\limits_x \\exp\\left(-F(x)\\right)\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Can be represented as a single MLP layer (undirected) with less hidden units\n\t\\item Compared to simple Boltzmann machine, we can express higher-order relations \n\t\\item Every hidden unit is independent of each other, and the same for input $x$:\n\t$$p(h|x) = \\prod_j p(h_j|x, \\theta), \\hspace{2mm} p(x|h) = \\prod_i p(x_i|h, \\theta) $$\n\t\\item We can now reformulate the conditional probabilities as sigmoids \\textbf{iff} $h$ and $x$ are still binary:\n\t$$p(h_j|x, \\theta) = \\sigma\\left(W_{:,j} x + b_j\\right), \\hspace{2mm}p(x_i|h, \\theta) = \\sigma\\left(W_{i,:} h + c_i\\right)$$\n\t\\item The loss is maximizing the log likelihood:\n\t$$\\mathcal{L}(\\theta) = \\frac{1}{N}\\sum_n \\log p(x_n|\\theta) = \\frac{1}{N}\\sum_n\\left[- F(x) - \\log Z\\right]$$\n\t\\item The gradients can be computed accordingly:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\pd{\\log p(x_n|\\theta)}{\\theta} & = -\\sum_h p(h|x_n, \\theta) \\pd{E(x_n,h| \\theta)}{\\theta} + \\sum_{\\tilde{x}, h} p(\\tilde{x}, h|\\theta) \\pd{E(\\tilde{x}, h|\\theta)}{\\theta}\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\tProblem: second term is sum over $x$ and $h$ $\\Rightarrow$ high-dimensional, hard to compute\n\t\\item One way to do it is using contrastive divergence: sample $h_0 \\sim p(h|x)$, and $x_1 \\sim p(x|h_0)$, etc. In practice, a single sample is mostly sufficient\n\t\\item \\textbf{Deep Belief Network}: RBM are still models of single layer, we can also use a stack of RBMs. First layer is directed, others not. Our joint pdf is $p(x, h_1, h_2) = p(x|h_1)\\cdot p(h_1|h_2)$\n\t\\item \\textbf{Deep Boltzmann machines}: also a stack of RBMs, but with undirected first layer\n\t\\begin{itemize}\n\t\t\\item Hence, we get $p(h_2^{k}|h_1, h_3) = \\sigma \\left(W_1^{:,k}h_1 + W_3^{k,:}h_3 \\right)$\n\t\t\\item Computing gradients is intractable $\\Rightarrow$ approximate by sampling\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Variational Autoencoders}\n\\begin{itemize}\n\t\\item We assume an underlying, lower-dimensional data distribution $p(z)$ with which we can model our data distribution $p(x,z)=p(x|z)p(z)$\n\t\\item Therefore, we need to model $p(z|x)$ which is often not easy to compute. In variational inference, we approximate the true posterior by $q_{\\varphi}(z)$ (approximated posterior does not have to depend on observed $x$, e.g. in VAE it does)\n\t\\item Our goal is to maximize $p(x)$. As this is intractable, we use the ELBO:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\log p(x) & = \\log \\int p(x,z)dz \\\\\n\t\t\t& = \\log \\int q_{\\varphi}(z) \\frac{\\int p(x,z)}{q_{\\phi}(z)} dz\\\\\n\t\t\t& = \\log \\mathbb{E}_{q_{\\varphi}(z)}\\left[\\frac{p(x,z)}{q_{\\varphi}(z)}\\right]\\\\\n\t\t\t& \\geq \\mathbb{E}_{q_{\\varphi}(z)}\\left[\\log \\frac{p(x,z)}{q_{\\varphi}(z)}\\right]\\\\\n\t\t\t& = \\mathbb{E}_{q_{\\varphi}(z)}\\left[\\log p(x|z)\\right] - \\text{KL}\\left(q_{\\varphi}(z)||p(z)\\right) = \\text{ELBO}_{\\theta, \\varphi}\\left(x\\right)\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item The distance between $\\log p(x)$ and the ELBO is the KL divergence to the true (unknown) posterior:\n\t$$\\log p(x) - \\text{KL}\\left(q_{\\varphi}(z)||p(z|x)\\right) = \\mathbb{E}_{q_{\\varphi}(z)}\\left[\\log p(x|z)\\right] - \\text{KL}\\left(q_{\\varphi}(z)||p(z)\\right)$$\n\t\\item Thus, maximizing the ELBO either increases the log likelihood or optimizes the approximated posterior\n\t\\item Variational Autoencoders make $q_{\\varphi}(z)$ dependent of $x$, and model $p_{\\theta}(x|z)$ as well:\n\t$$\\text{ELBO}_{\\theta, \\varphi}\\left(x\\right) = \\mathbb{E}_{q_{\\varphi}(z|x)}\\left[\\log p_{\\theta}(x|z)\\right] - \\text{KL}\\left(q_{\\varphi}(z|x)||p_{\\lambda}(z)\\right)$$\n\tNote that $p_{\\lambda}(z)$ is not optimized, and its parameters $\\lambda$ just describe the prior (e.g. standard Gaussian) \n\t\\item The loss function for a VAE is the negative ELBO, where we approximate the expectation by a single sample. The KL is mostly chosen to be analytically solvable (e.g. for two Gaussian) to prevent a Monte-Carlo approximation of the integral \n\t\\item However, we face a problem when we try to compute the gradients for $\\nabla_{\\varphi} \\mathcal{L}$. Using Monte-Carlo integration has high variance, and sampling is non-continuous operation\n\t\\item \\textbf{Reparameterization trick}: sample from external, constant distribution, and transform this sample into a sample of the modeled distribution. For Gaussian: $z = \\mu_q + \\sigma_q \\cdot \\epsilon$\n\\end{itemize}\n\\subsubsection{Improvements of VAE}\n\\begin{itemize}\n\t\\item \\textbf{Encoder distribution}\n\t\\begin{itemize}\n\t\t\\item Modeling $q(z|x)$ as Gaussian makes training and implementation easy, but assumes that true posterior is also Gaussian, or can be at least approximated by one\n\t\t\\item Simple option: use different task-specific distribution like e.g. hyperspherical, however not always suitable\n\t\t\\item We can improve the complexity of this posterior by plugging in a Normalizing flow on top of the encoder output\n\t\t\\begin{equation*}\n\t\t\\begin{split}\n\t\tz_0 \\sim q_0(z|x) & = \\mathcal{N}(z|\\mu(x), \\text{diag}(\\sigma^2(x)))\\\\\n\t\tq_K(z|x) & = q_0(z|x) \\cdot \\left|\\text{det}\\pd{f_K(z_{k-1})}{z_{k-1}}\\right|\\\\\n\t\t\\end{split}\n\t\t\\end{equation*}\n\t\t\\item The ELBO is added with an additional term during training\n\t\t$$\\text{ELBO} = \\mathbb{E}_{q_{\\varphi}(z|x)}\\left[\\log p_{\\theta}(x|z)\\right] - \\text{KL}\\left(q_{\\varphi}(z|x)||p_{\\lambda}(z)\\right) + \\mathbb{E}_{z_0 \\sim q_0(z_0|x)}\\left[\\sum_{k=1}^{K} \\log \\left|\\text{det}\\pd{f_k(z_{k-1})}{z_{k-1}}\\right|\\right]$$\n\t\\end{itemize}\n\t\\item \\textbf{Prior optimization}\n\t\\begin{itemize}\n\t\t\\item We assume a prior $p(z)$ which is for example Gaussian, but cannot make sure that every point of the prior actually has a realistic counterpart in the original $x$ space\n\t\t\\item The optimal prior is the averaged distribution over all data samples: $q^{*}(z) = \\frac{1}{N}\\sum_{n=1}^{N} q_{\\varphi}(z|x_n)$\n\t\t\\item However, summing over all data point is infeasible. Thus, approximate it by $K$ pseudo-inputs $u_k$ that are trained via standard SGD in the framework:\n\t\t$$p_\\lambda(z) = \\frac{1}{K} \\sum_{k=1}^{K} q_{\\varphi}(z|u_k)$$\n\t\\end{itemize}\n\t\n\\end{itemize}\n\\subsection{Normalizing flows}\n\\begin{itemize}\n\t\\item VAE cannot model $p(x)$ directly because of the intractable formulation ($p(x) = \\int p(x,z)dz$)\n\t\\item Normalizing Flows solve that problem by using a series of invertible transformation that allow more complex latent distributions than Gaussian\n\t\\item The models can therefore be trained on directly maximizing the log likelihood instead of using the ELBO or similar\n\t\\item A normalizing flow consists of multiple flows that transform a simple Gaussian distribution step by step in the data distribution (see Figure~\\ref{fig:NF_concept})\n\t\\item Every flow shifts the probability mass specified by parameters (determined by e.g. a NN, see Figure~\\ref{fig:NF_density_shift})\n\t\\begin{figure}[ht!]\n\t\t% NF_density_shift.png\n\t\t\\centering\n\t\t\\begin{subfigure}{0.7\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=\\textwidth]{figures/NF_concept.png}\n\t\t\t\\caption{General concept of stacking multiple flows}\n\t\t\t\\label{fig:NF_concept}\n\t\t\\end{subfigure}\n\t\t\\hspace{8mm}\n\t\t\\begin{subfigure}{0.2\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=\\textwidth]{figures/NF_density_shift.png}\n\t\t\t\\caption{Shifting density}\n\t\t\t\\label{fig:NF_density_shift}\n\t\t\\end{subfigure}\n\t\t\\caption{Outline of how a normalizing flow works}\n\t\t\\label{fig:NF}\n\t\\end{figure} \n\t\\item Mathematically, we can define a normalizing flow by:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tx & = z_k = f_k \\circ f_{k-1} \\circ ... \\circ f_1 (z_0) \\to z_i = f_i(z_{i-1})\\\\\n\t\t\tp(z_i) & = p(z_{i-1}) \\cdot \\left|\\det \\frac{f_{i}^{-1}}{z_i}\\right| \\implies p(x) = p(z_0) \\cdot \\prod_{i=1}^{K} \\left|\\det \\frac{f_{i}^{-1}}{z_i}\\right|\\\\\n\t\t\t\\log p(x) & = \\log p(z_0) - \\sum_{i=1}^{K} \\log \\left|\\det \\frac{f_{i}}{z_i}\\right|\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Requirements: $f$ must be invertible (dimensions of $x$ and $z$ equal), and the Jacobian must be easy to compute (i.e. triangular)\n\\end{itemize}"
  },
  {
    "path": "Deep_Learning/dl_intro.tex",
    "content": "\\section{Introduction}\n\\subsubsection{Perceptron}\n\\begin{itemize}\n\t\\item Single perceptron weights every input with a weight, and adds a bias term\n\t\\item Step function as output: if input sum greater zero, then output is 1, else 0 (or -1)\n\t\\item Problem: can only learn linear problems and not e.g. XOR\n\t\\item Overcoming by multi-layer perceptron \n\\end{itemize}"
  },
  {
    "path": "Deep_Learning/dl_modularity.tex",
    "content": "\\section{Modular Learning}\n\\begin{itemize}\n\t\\item \\textit{Definition}: A family of \\textcolor{green}{parametric}, \\textcolor{lightred}{non-linear} and \\textcolor{blue}{hierarchical} \\textcolor{orange}{representation learning functions}, which are \\textcolor{red}{massively optimized with stochastic gradient descent} to \\textcolor{purple}{encode domain knowledge}, i.e. domain invariances, stationarity.\n\t% \\item Although with two-layer (shallow) network, we can approximate all possible functions, a deep architecture tends to be more efficient and generalize better\n\t\\item A neural network is a series of hierarchically connected functions $\\Rightarrow$ Directed Acyclic graph\n\t\\item Note that it is not allowed to have loops except over time/additional dimension\n\\end{itemize}\n\\begin{figure}[ht!]\n\t\\centering\n\t\\includegraphics[width=0.2\\textwidth]{figures/modularity_example_network.png}\n\t\\caption{Example network with interweaved connections. The architecture can be made arbitrarily complex, and can also include recurrent connections.}\n\t\\label{fig:modularity_example_network}\n\\end{figure}\n\\subsection{Module}\n\\begin{itemize}\n\t\\item A module is the simplest mathematical component in a NN, and can be expressed by $a=h(x;w)$ where $a$ is the output, $x$ the input, $w$ trainable parameters and $h$ an activation function\n\t\\item $w$ mostly learned by gradient-based methods, usually maximizing the likelihood\n\t\\begin{itemize}\n\t\t\\item ML solution: $w^{*} = \\arg\\max\\limits_{w}\\prod\\limits_{x,y}p_{model}\\left(y|x;w\\right)$\n\t\t\\item For gradient-based methods, we can minimize the negative log likelihood:\\\\ $\\mathcal{L}(w) = -\\mathbb{E}_{x,y\\sim \\tilde{p}_{data}}\\left[\\log p_{model}\\left(y|x;w\\right)\\right]$\n\t\t\\item If output is Gaussian, we would get the $\\ell_2$ norm\n\t\t\\item If output is Laplacian, we would get the $\\ell_1$ norm\n\t\\end{itemize} \n\t\\item Using a loss function that matches the output distribution of the network helps, because:\n\t\\begin{itemize}\n\t\t\\item It makes math simpler (exponential cancels out)\n\t\t\\item Better numerical stability ($\\log$ with very small/negative values, helps for e.g. Softmax+CrossEntropy)\n\t\t\\item Makes gradients larger as exponential-like activations often lead to saturation, which means gradients are almost 0 (but not with $\\log$)\n\t\\end{itemize}\n\t\\item It is important that the input and output distribution of every module match, as otherwise we get inconsistent behavior and makes it harder to learn\n\t\\begin{itemize}\n\t\t\\item For activation functions, this means we prefer them to be mostly activated around the origin and centered\n\t\t\\item Otherwise, e.g. ReLU can be come a linear unit or set everything to 0\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Example modules}\n\\begin{itemize}\n\t\\item \\textbf{Linear module}: $a = wx$\n\t\\begin{itemize}\n\t\t\\item Simple gradients $\\frac{\\partial a}{\\partial w} = x$, $\\frac{\\partial a}{\\partial x} = w$\n\t\t\\item No activation saturation $\\Rightarrow$ strong, reliable gradients\n\t\\end{itemize}\n\t\\item \\textbf{Rectified Linear Unit}: $a = \\max(0,x)$\n\t\\begin{itemize}\n\t\t\\item Gradient is step function. $\\pd{a}{x} = \\begin{cases}\n\t\t0 & \\text{ if } x\\leq 0\\\\\n\t\t1 & \\text{ if } x > 0\\\\\n\t\t\\end{cases}$\n\t\t\\item Hence, strong, fast gradients\n\t\t\\item However, dead neurons might be an issue when initialization/weights produce outputs smaller 0 for every input \n\t\t\\item Different variations like LeakyReLU, Softplus ($\\ln(1+e^{x})$), NoisyReLU exist\n\t\\end{itemize}\n\t\\item \\textbf{Sigmoid}: $a=\\sigma(x)=\\frac{1}{1+e^{-x}}$\n\t\\begin{itemize}\n\t\t\\item Gradient easy to calculate: $\\pd{a}{x} = \\sigma(x)\\left(1-\\sigma\\left(x\\right)\\right)$\n\t\t\\item Can be used as output function for probability distribution between $[0,1]$\n\t\t\\item Saturates and has small gradients\n\t\t\\item Not centered around origin $\\Rightarrow$ not good choice for within a network\n\t\\end{itemize}\n\t\\item \\textbf{Tanh}: $a=\\tanh\\left(x\\right)=\\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}$\n\t\\begin{itemize}\n\t\t\\item Gradients $\\pd{a}{x}=1-\\tanh\\left(x\\right)^2$\n\t\t\\item Saturates as well, but has slightly higher gradients than sigmoid and is centered around origin\n\t\\end{itemize}\n\t\\item \\textbf{Softmax}: $a^{(k)} = \\text{softmax}\\left(x^{(k)}\\right) = \\frac{e^{x^{(k)}}}{\\sum_j e^{x^{(j)}}}$\n\t\\begin{itemize}\n\t\t\\item Probability distribution over multiple classes\n\t\t\\item Softmax trick for numerical stability: $\\frac{e^{x^{(k)}-\\mu}}{\\sum_j e^{x^{(j)}-\\mu}}$\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Backpropagation}\n\\begin{itemize}\n\t\\item Calculate gradients of all parameters in the network based on the loss on the last layer\n\t\\item Principle of chain rule: $\\pd{z}{x} = \\sum_j \\chain{z}{y_i}{x}$ (gradients from all possible paths)\n\t\\begin{itemize}\n\t\t\\item In vector notation: $\\nabla_{\\bm{x}} \\bm{z} = \\left(\\pd{\\bm{y}}{\\bm{x}}\\right)^T \\cdot \\nabla_{\\bm{y}} \\bm{z}$ with Jacobian $\\pd{\\bm{y}}{\\bm{x}} = \\left[\\begin{array}{ccc}\n\t\t\\pd{y_1}{x_1} & \\pd{y_1}{x_2} & \\pd{y_1}{x_3} \\\\[5pt]\n\t\t\\pd{y_2}{x_1} & \\pd{y_2}{x_2} & \\pd{y_2}{x_3} \\\\\n\t\t\\end{array}\\right]$\n\t\\end{itemize}\n\t\\item Steps of Backpropagation:\n\t\\begin{enumerate}\n\t\t\\item Compute forward propagations for all layers recursively:\n\t\t$a^{(l)} = h^{(l)}\\left(x^{(l)}\\right) \\text{ and } x^{(l+1)} = a^{(l)}$\n\t\t\\item Compute the reverse path. \n\t\t$$\\pd{\\mathcal{L}}{a^{(l)}} = \\left(\\pd{a^{(l+1)}}{x^{(l+1)}}\\right)^T \\cdot \\pd{\\mathcal{L}}{a^{(l+1)}}, \\hspace{4mm} \\pd{\\mathcal{L}}{\\theta^{(l)}} = \\pd{a^{(l)}}{\\theta^{(l)}} \\cdot \\left(\\pd{\\mathcal{L}}{a^{(l)}}\\right)^T$$\n\t\t\\item Use gradients $\\pd{\\mathcal{L}}{\\theta^{(l)}}$ to update parameters via SGD\n\t\\end{enumerate}\n\\end{itemize}"
  },
  {
    "path": "Deep_Learning/dl_optimization.tex",
    "content": "\\section{Deep Learning Optimizations}\n\\begin{itemize}\n\t\\item Pure optimization has a very direct goal, namely finding the optimum. However, in Machine Learning, we define a training goal. Thus, the ``optimal'' parameters might not necessarily be the optimum (e.g. overfitting)\n\\end{itemize}\n\\subsection{Stochastic Gradient Descent}\n\\begin{itemize}\n\t\\item Pushing the weights towards highest gradient change\n\t$$w_{t+1} = w_{t} - \\eta_t \\nabla_{w} \\mathcal{L}$$\n\t\\item \\textit{Gradient descent}: gradients on the full dataset. However:\n\t\\begin{itemize}\n\t\t\\item Dataset is mostly too large for this\n\t\t\\item No real guarantee that this leads to a good optimum and/or it will converge faster\n\t\\end{itemize}\n\t\\item \\textit{Stochastic gradient descent}: approximate gradients by averaging over a small batch. \n\t\\begin{itemize}\n\t\t\\item Standard error is inverse proportional to number of elements $m$ in a batch: $\\sigma / \\sqrt{m}$.\n\t\t\\item Noisy gradients help to escape local minima, acts as regularization\n\t\t\\item Does sample roughly representative gradients from dataset. Is better as training data is also just a rough approximation of what the test data might look like (optimum on training $\\neq$ optimum on test)\n\t\t\\item SGD is faster, especially in first iterations\n\t\t\\item SGD is able to adapt with dynamically changing datasets\n\t\\end{itemize}\n\t\\item \\textit{Ill conditioning}: if gradients are large, applying them can lead to worse performance. This is the case if the second order derivative changes faster \n\\end{itemize}\n\\subsection{Advanced optimizations}\n\\subsubsection{Gradient-based optimization}\n\\begin{itemize}\n\t\\item \\textit{Pathological curvatures}: move through a ravine towards minimum. SGD tends to oscillate between the walls because they have high gradients\n\t\\begin{itemize}\n\t\t\\item Second order optimization can help a lot for pathological curvatures: $$w_{t+1} = w_{t} - H_{\\mathcal{L}}^{-1} \\eta_t g_t$$\n\t\t\\item Hessian $H_{\\mathcal{L}}^{ij} = \\pd{\\mathcal{L}}{w_i\\partial w_j}$ works as adaptive learning rate per parameter\n\t\t\\item However, unfeasible in practice because Hessian gets very large\n\t\\end{itemize}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.3\\textwidth]{figures/optimization_pathological_curvatures.png}\n\t\t\\caption{Pathological curvature}\n\t\t\\label{fig:optimization_pathological_curvatures}\n\t\\end{figure}\n\t\\item \\textbf{Momentum}: maintain \\textit{momentum} from previous parameter updates to dampen the oscillations.\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tu_{t+1} & = \\gamma u_{t} - \\eta_t g_t \\\\\n\t\t\tw_{t+1} & = w_{t} + u_{t+1}\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\begin{itemize}\n\t\t\\item Works as a exponential averaging $\\Rightarrow$ more robust gradients, faster convergence\n\t\t\\item $\\gamma$ might be initialized lower and then increased over time to $0.9$\n\t\t\\item Standard values for $\\gamma$ are between $0.5$ and $0.9$ (note that a lower learning rate should be used compared to standard SGD)\n\t\\end{itemize}\n\t\\item \\textbf{RMSprop}: adapting learning rate on current loss surface.\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tr_t & = \\alpha \\cdot r_{t-1} + \\left(1 - \\alpha\\right) \\cdot g_t^2\\\\\n\t\t\t\\eta_t & = \\frac{\\eta}{\\sqrt{r_t} + \\epsilon} \\\\\n\t\t\tw_{t+1} & = w_{t} - \\eta_t \\cdot g_t\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\begin{itemize}\n\t\t\\item $r_t$ is the (exponentially) averaged gradient norm describing the size of the gradients (per dimension!)\n\t\t\\item The learning rate is then adapted by $\\eta_t$ at every time step for each dimension independently\n\t\t\\item $\\epsilon$ to prevent numerical instability and too large learning rates\n\t\t\\item With the adapted learning rate, we update our weights with SGD\n\t\\end{itemize}\n\t\\item \\textbf{Adam}: Combining adaptive learning rate and momentum\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tm^{(t)} & = \\beta_1 m^{(t-1)} + (1 - \\beta_1)\\cdot g^{(t)}\\\\\n\t\t\tv^{(t)} & = \\beta_2 v^{(t-1)} + (1 - \\beta_2)\\cdot \\left(g^{(t)}\\right)^2\\\\\n\t\t\t\\hat{m}^{(t)} & = \\frac{m^{(t)}}{1-\\beta^{t}_1}, \\hat{v}^{(t)} = \\frac{v^{(t)}}{1-\\beta^{t}_2}\\\\\n\t\t\tw^{(t)} & = w^{(t-1)} - \\frac{\\eta}{\\sqrt{v^{(t)}} + \\epsilon}\\circ \\hat{m}^{(t)}\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\begin{itemize}\n\t\t\\item Keeps track of the gradient norm for momentum $m^{(t)}$, and norm (also known as velocity) $v^{(t)}$\n\t\t\\item The hyperparameters $\\beta_1$ and $\\beta_2$ correlate with the $\\gamma$ and $\\alpha$ respectively from the previous approaches\n\t\t\\item The adaptive learning rate is expressed by $\\hat{v}^{(t)}$, and the exponentially averaged gradients by $\\hat{m}^{(t)}$\n\t\t\\item The division is to remove the bias of $m^{(0)}$ and $v^{(0)}$ being zero. Note that $\\beta_1^t$ means the value of $\\beta_1$ to the power $t$, and not at time step $t$\n\t\t\\item Adam is in general better for complex models, but might fail on easy/stupid tasks compared to simple methods like SGD\n\t\\end{itemize}\n\t\\item \\textbf{Adagrad}: adapting learning rate based on both gradient scale and frequency of updates\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tG_t & = G_{t-1} + \\text{diag}\\left(g_t^2\\right)\\\\\n\t\t\tw_{t+1} & = w_{t} - \\frac{\\eta}{\\sqrt{G_t + \\epsilon}}\\cdot g_t\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\begin{itemize}\n\t\t\\item Very similar to RMSprop, but sums the scales over all time steps ($G_t$) instead of exponentially averaging \n\t\t\\item Less sensitive to learning rate tuning, but it gets very small over training time annealing to 0\n\t\\end{itemize} \n\t\\item \\textbf{Nesterov momentum}: use the future gradient instead of the current gradient. Leads to better convergence in theory\n\\end{itemize}\n\\subsubsection{Bayesian optimization}\n\\begin{itemize}\n\t\\item Gradient-based optimizations have the problem of getting stuck in local minima\n\t\\item Bayesian optimization is a gradient-free, educated trial and error guesser that works in lower dimensional spaces (up to 1000, but mostly 20 to 50 parameters)\n\t\\item Determines the next point/parameter values to evaluate based on variance/uncertainty, and expected/predictive value. \n\t\\item Can be used for e.g. network architecture search\n\\end{itemize}\n\\subsection{Normalization}\n\\begin{itemize}\n\t\\item Data pre-processing\n\t\\begin{itemize}\n\t\t\\item Center data around 0 (activation functions are designed for that)\n\t\t\\item Scale input variables to have similar diagonal covariances (not if features are differently important)\n\t\t\\item De-correlate features if there is no inductive bias (e.g. sequence over time)\n\t\\end{itemize}\n\t\\item \\textbf{Batch normalization}: ensure Gaussian distribution of features over batches at every module input\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\mu_B = \\frac{1}{m} \\sum\\limits_{i=1}^{m} x_i, &\\hspace{5mm} \\sigma_B^2 = \\frac{1}{m} \\sum\\limits_{i=1}^{m} \\left(x_i - \\mu_B\\right)^2 \\\\\n\t\t\t\\hat{x}_i & = \\frac{x_i - \\mu_B}{\\sqrt{\\sigma^2 + \\epsilon}} \\\\\n\t\t\t\\hat{y}_i & = \\gamma \\cdot \\hat{x}_i + \\beta\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\begin{itemize}\n\t\t\\item Normalize feature to $\\hat{x}_i \\sim \\mathcal{N}(0,1)$, then rescale with trainable parameters $\\gamma$ (variance) and $\\beta$ (mean).\n\t\t\\item Helps the optimizer to control mean and variance of input distribution, and reduces effects of 2nd order between layers $\\Rightarrow$ easier, faster learning \n\t\t\\item Acts as regularizer as distribution depends on mini-batch and therefore introduces noise\n\t\t\\item During testing, take a moving average of the last training steps and use those for $\\mu_B$ and $\\sigma_B^2$\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Regularization}\n\\begin{itemize}\n\t\\item Weight regularization needed to prevent overfitting\n\t\\item \\textbf{$\\ell_2$-regularization}: Introduce objective term for minimizing weights\n\t$$w^{*}=\\arg\\min_w \\mathcal{L} + \\frac{\\lambda}{2}\\sum_l ||w_l||^2$$\n\t\\begin{itemize}\n\t\t\\item When using simple (stochastic) gradient descend, then $\\ell_2$ regularization is the same as weight decay: $$w_{t+1} = \\left(1-\\lambda \\eta_t\\right) w_{t} - \\eta_t \\nabla_{\\theta} \\mathcal{L}$$\n\t\\end{itemize}\n\t\\item \\textbf{$\\ell_1$-regularization}: use $\\ell_1$ objective, introduces sparse weights\n\t$$w^{*}=\\arg\\min_w \\mathcal{L} + \\lambda \\sum_l ||w_l||$$\n\t\\item \\textbf{Early stopping}: stop the training when test error increases but training loss continues to decrease. Can be counted to regularization as training steps are reduced\n\t\\item \\textbf{Dropout}: setting activations randomly to 0 during training with probability $p$ (mostly between $0.1$ and $0.5$)\n\t\\begin{itemize}\n\t\t\\item During test time, every activation is reweighted by $1 - p$\n\t\t\\item Reduces co-adaptations/-dependencies between neurons because none can solely depend on the other\n\t\t\\item Neurons get more robust $\\Rightarrow$ reduces overfitting\n\t\t\\item Effectively, a different network architecture is used every iteration. Testing can be seen as using model ensemble\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Weight initialization}\n\\begin{itemize}\n\t\\item There are two forces on the weight magnitude: small weights are needed to keep data around origin, but large weights are required to have strong learning signals\n\t\\item Initialization should preserve variance of activations (input variance $\\approx$ output variance to keep distribution between modules same)\n\t\\item Depends on non-linearity and data normalization\n\t\\item \\textbf{Xavier initialization}: to maintain data variance, the variance of the weights must be $1/d$ where $d$ is number of input neurons $\\Rightarrow$ sample weight values from $w\\sim\\mathcal{N}(0,\\sqrt{1/d})$\n\t\\item \\textbf{Initialization for ReLU}: ReLU set half of the output neurons to 0 $\\Rightarrow$ double the weight variance to compensate zero flat-area: $w\\sim\\mathcal{N}(0,\\sqrt{2/d})$\n\\end{itemize}"
  },
  {
    "path": "Deep_Learning/dl_rnn.tex",
    "content": "\\section{Recurrent and Graph Neural Networks}\n\\subsection{Backpropagation through time}\n\\begin{itemize}\n\t\\item Sequences are of arbitrary length. Standard networks like CNN mostly work on fixed input dimensionality\n\t\\item Usage of memory with shared weights $\\theta$: $$c_{t+1} = h_{\\theta}\\left(x_{t+1}, c_{t}\\right) = h_{\\theta}\\left(x_{t+1}, h_{\\theta}\\left(x_{t}, c_{t-1}\\right)\\right) = ...$$\n\t\\item Simple RNN cell: \n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tc_t & = \\tanh\\left(U\\cdot x_t + W \\cdot c_{t-1}\\right) \\\\\n\t\t\ty_t & = \\text{softmax}\\left(V \\cdot c_{t}\\right) \\\\\n\t\t\t\\loss & = \\sum\\limits_{t=1}^{T} y_t^{*} \\log y_t \\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Gradient for output weights $V$:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\pd{\\loss_t}{V} & = \\chain{\\loss_t}{y_t}{c_t}\\pd{c_t}{V} = \\left(y_t - y_t^{*}\\right) \\cdot \\left(c_t\\right)^T\\\\\n\t\t\t\\pd{\\loss}{V} & = \\sum\\limits_{t=1}^{T} \\pd{\\loss_t}{V}\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Gradient for memory weights $W$: \n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\pd{\\loss_t}{W} & = \\chain{\\loss_t}{y_t}{c_t}\\pd{c_t}{W}\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\begin{itemize}\n\t\t\\item In $\\pd{c_t}{W}$, $c_t$ depends on $c_{t-1}$ which again depends on $W$. Thus, we have a recurrence in the gradient calculation:\n\t\t$$\\pd{\\loss_t}{W} = \\sum\\limits_{k=1}^{t} \\chain{\\loss_t}{y_t}{c_t}\\chain{c_t}{c_k}{W}$$\n\t\twhere $\\pd{c_k}{W}$ only models the dependency exactly at time step $k$\n\t\t\\item The gradient $\\pd{c_t}{c_k}$ can be determined by the chain rule: $\\pd{c_t}{c_k} = \\prod\\limits_{i=k+1}^{t} \\pd{c_i}{c_{i-1}}$\n\t\t\\item All in all, the final loss is:\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\pd{\\loss}{W} & = \\sum\\limits_{t=1}^{T}\\sum\\limits_{k=1}^{t} \\chain{\\loss_t}{y_t}{c_t}\\left(\\prod\\limits_{i=k+1}^{t} \\pd{c_i}{c_{i-1}}\\right)\\pd{c_k}{W}\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\\end{itemize}\n\t\\item Gradient for input weights $U$ very similar to $W$: \n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\pd{\\loss}{U} & = \\sum\\limits_{t=1}^{T}\\sum\\limits_{k=1}^{t} \\chain{\\loss_t}{y_t}{c_t}\\left(\\prod\\limits_{i=k+1}^{t} \\pd{c_i}{c_{i-1}}\\right)\\pd{c_k}{U}\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item The problem with RNNs are that the gradients at time step $t$ depend on $c_{t-1}$ which also depends on $w$. However, the gradients are calculated with the assumption that $w$ stays the same for the previous time steps.\n\t\\item This error can easily accumulate over many time steps so that in very long sequences, the gradients for the last steps are inaccurate\n\t\\item Reduce learning rate/fewer updates, but this leads to slower training\n\\end{itemize}\n\\subsubsection{Vanishing gradients}\n\\begin{itemize}\n\t\\item The exact derivations can be found in \\href{http://proceedings.mlr.press/v28/pascanu13.pdf}{this paper}\n\t\\item We assume an alternative formulation for simplicity here: $c_t = W \\cdot \\sigma(c_{t-1}) + U \\cdot x_{t-1}$ where $\\sigma$ is an arbitrary activation function. Then, the partial derivative between two time steps is\\\\ $\\pd{c_{t}}{c_{k}} = \\prod\\limits_{i=k+1}^{t} \\pd{c_{i}}{c_{i-1}} = \\prod\\limits_{i=k+1}^{t} W^T \\cdot \\text{diag}\\left(\\pd{\\sigma\\left(c_t\\right)}{c_t}\\right)$\n\t\\item Hence, the magnitude of $\\pd{c_{t+1}}{c_{t}}$ is bounded by this derivative: \n\t$$\\left\\lVert \\pd{c_{t+1}}{c_{t}}\\right\\rVert \\leq \\left\\lVert W^T\\right\\rVert \\cdot \\left\\lVert \\text{diag}\\left(\\pd{\\sigma\\left(c_t\\right)}{c_t}\\right)\\right\\rVert$$\n\t\\item In case the derivative of our non-linearity is bounded to a value $\\gamma$ (which is 1 in case of tanh), we know that gradients vanish if the norm of the weight gradients are lower than $1/\\gamma$:\n\t$$\\left\\lVert \\pd{c_{t+1}}{c_{t}}\\right\\rVert \\leq \\left\\lVert W^T\\right\\rVert \\cdot \\left\\lVert \\text{diag}\\left(\\pd{\\sigma\\left(c_t\\right)}{c_t}\\right)\\right\\rVert < \\frac{1}{\\gamma}\\gamma = 1$$\n\t\\item This term is exponentiated with the number of time steps. Thus, long sequences suffer even more of vanishing gradients $\\Rightarrow$ learn only short-term relationships\n\t\\item If however $\\left\\lVert \\pd{c_{t+1}}{c_{t}}\\right\\rVert > 1$ because of $\\left\\lVert W^T\\right\\rVert \\gg 1/\\gamma$, then we can get exploding gradients\n\t\\item Quick fix for exploding gradients: clip gradient norm. However, there the counterpart can happen where we only focus on long-term relationships\n\\end{itemize}\n\\subsubsection{Long Short-Term Memory}\n\\begin{itemize}\n\t\\item Preventing vanishing gradients by gate mechanism\n\t\\item By simply adding features to memory and limiting memory by sigmoid we can get strong gradients for any sequence length. Note that the gradients get lower in expectation because sigmoid has mean $0.5$. Nevertheless, if long-term dependencies are important, the network can learn them now\n\t\\item \\textit{Forget gate}: regulating how much information is kept from last time step\n\t\\item \\textit{Input + candidate gate}: Regulating which, and how much new information should be added given the current time step\n\t\\item \\textit{Output gate}: What features are important for the current time step\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.3\\textwidth]{figures/RNN_LSTM.png}\n\t\t\\caption{Visualization of a LSTM cell}\n\t\t\\label{fig:RNN_LSTM}\n\t\\end{figure}\n\\end{itemize}\n\\subsection{Graph Neural Networks}\n\\begin{itemize}\n\t\\item Perform operation on graph-structured data (e.g. social networks or knowledge graphs)\n\\end{itemize}\n\\subsubsection{Deep Walk}\n\\begin{itemize}\n\t\\item Learning latent representations of vertices in a network\n\t\\item The Deep Walk algorithm consists of two simple steps:\n\t\\begin{enumerate}\n\t\t\\item Perform random walks on the graph to generate node sequences\n\t\t\\item Run skip-gram on sequence (with word window) to learn node embeddings\n\t\\end{enumerate}\n\t\\item \\textit{Drawback}: algorithm has to be re-run if a new node is added, not useful for dynamic graphs\n\\end{itemize}\n\\subsubsection{GraphSage}\n\\begin{itemize}\n\t\\item In every iteration, aggregate information of neighbors and the node itself to generate new embeddings\n\t\\item Aggregation techniques are taking the mean (with weight and non-linearity applied on it afterwards), max pooling, or using a LSTM\n\\end{itemize}\n\\subsubsection{Graph Convolutional Networks}\n\\begin{itemize}\n\t\\item A GNN layer takes as input the embeddings for every node $H^{(l)}$ and the adjacency matrix $A$, and create new embeddings $H^{(l+1)}$\n\t\\item Graph convolutional layers use for this a matrix multiplication where weights are shared over nodes\n\t\\item In the simplest form, a GCN layer can be defined as $h(H^{(l)}, A) = \\sigma\\left(A H^{(l)} W^{(l)}\\right)$\n\t\\item To make it more efficient, we add the identity matrix to $\\hat{A} = A + I$ so that nodes use their old embeddings as well, and take the mean instead of the sum over all neighbors (by degree matrix $D$):\n\t$$h(H^{(l)}, A) = \\sigma\\left(D^{-1/2}\\hat{A}D^{-1/2} H^{(l)} W^{(l)}\\right)$$\n\\end{itemize}"
  },
  {
    "path": "Deep_Learning/dl_summary.tex",
    "content": "\\documentclass[a4paper]{article} \n\\addtolength{\\hoffset}{-2.25cm}\n\\addtolength{\\textwidth}{4.5cm}\n\\addtolength{\\voffset}{-3.25cm}\n\\addtolength{\\textheight}{5cm}\n\\setlength{\\parskip}{0pt}\n\\setlength{\\parindent}{0in}\n\n\\usepackage{blindtext} % Package to generate dummy text\n\\usepackage{charter} % Use the Charter font\n\\usepackage[utf8]{inputenc} % Use UTF-8 encoding\n\\usepackage{microtype} % Slightly tweak font spacing for aesthetics\n\\usepackage[english]{babel} % Language hyphenation and typographical rules\n\\usepackage{amsthm, amsmath, amssymb, amsfonts} % Mathematical typesetting\n\\usepackage{float} % Improved interface for floating objects\n\\usepackage[final, colorlinks = true, \nlinkcolor = black, \ncitecolor = black]{hyperref} % For hyperlinks in the PDF\n\\usepackage{graphicx, multicol} % Enhanced support for graphics\n\\usepackage{xcolor} % Driver-independent color extensions\n\\usepackage{marvosym, wasysym} % More symbols\n\\usepackage{rotating} % Rotation tools\n\\usepackage{subcaption}\n\\usepackage{wrapfig}\n\\usepackage{censor} % Facilities for controlling restricted text\n\\newcommand{\\note}[1]{\\marginpar{\\scriptsize \\textcolor{red}{#1}}} % Enables comments in red on margin\n\\usepackage{bm}\n\\usepackage{blkarray}\n\\usepackage{enumitem}\n\\usepackage{pgfplots}\n\\usepackage{tikz}\n\n\\newcommand{\\pd}[2]{\\frac{\\partial #1}{\\partial #2}}\n\\newcommand{\\loss}[0]{\\mathcal{L}}\n\\newcommand{\\chain}[3]{\\frac{\\partial #1}{\\partial #2}\\frac{\\partial #2}{\\partial #3}}\n\\newcommand{\\eq}[1]{\\begin{equation*}\\begin{split}#1\\end{split}\\end{equation*}}\n\\newcommand{\\coderef}[0]{Please find the implementation in the folder with the code files.}\n\\newcommand{\\TODO}[1]{\\textbf{\\textcolor{red}{#1}}}\n\n\\definecolor{green}{RGB}{0,160,0}\n\\definecolor{blue}{RGB}{0,0,160}\n\\definecolor{red}{RGB}{160,0,0}\n\\definecolor{orange}{RGB}{200,160,0}\n\\definecolor{purple}{RGB}{170,0,200}\n\\definecolor{cyan}{RGB}{0,200,200}\n\\definecolor{lightred}{RGB}{200,50,50}\n\n\\setcounter{tocdepth}{2}\n% Title Page\n\\title{Summary Deep Learning}\n\\author{Phillip Lippe}\n\n\n\\begin{document}\n\\maketitle\n\\tableofcontents\n\\newpage\n\n\\input{dl_intro.tex}\n\\input{dl_modularity.tex}\n\\input{dl_optimization.tex}\n\\input{dl_convnets.tex}\n\\input{dl_rnn.tex}\n\\input{dl_generative_models.tex}\n\\input{dl_bayesian.tex}\n\\input{dl_autoregressive.tex}\n\\input{dl_deep_rl.tex}\n\\appendix\n\\newpage\n\\input{dl_appendix.tex}\n\n\\end{document}"
  },
  {
    "path": "Information_Retrieval_1/ir_boolean_retrieval.tex",
    "content": "\\section{Boolean Retrieval}\n\\begin{itemize}\n\t\\item \\textbf{Information retrieval} is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers)\n\t\\item \\textbf{Boolean retrieval model} is a model in which the queries are in the form of a Boolean expression of terms. Terms can be combined by the operators \\texttt{AND}, \\texttt{OR} and \\texttt{NOT} \n\\end{itemize}\n\\subsection{Inverted Index}\n\\begin{itemize}\n\t\\item \n\\end{itemize}"
  },
  {
    "path": "Information_Retrieval_1/ir_click_models.tex",
    "content": "\\section{Click models}\n\\begin{itemize}\n\t\\item User clicks can be used as evaluation of IR systems as clicks indicate the relevance of a document\n\t\\item However, clicks are highly biased (positional, textual, attention/visual,...) $\\Rightarrow$ click models try to remove these biases and help using clicks for evaluation\n\t\\item Click models are optimized/trained on click logs which record for a given query which documents were clicked\n\t\\item Most models are based on probabilistic graphical models (PGMs) that describe the probability of a click\n\t\\item They are mostly trained by either applying a MLE or EM algorithm\n\\end{itemize}\n\\subsection{Random click model}\n\\begin{itemize}\n\t\\item In random click models, every document on the result page has the same probability of being clicked: $$P(C_u = 1) = \\text{const} = \\rho$$\n\t\\item Therefore, the model contains only a single parameter, which can be optimized by applying MLE: $$\\rho = \\frac{\\#\\text{clicks}}{\\#\\text{shown docs}}$$\n\t\\item \\textit{Advantages}: simple and fast\n\t\\item \\textit{Disadvantages}: the random click model does not consider many aspects including the position and content of a document\n\t\\item There are different variations of this model (also called click-through rate models - CTR) considering more aspects\n\t\\begin{itemize}\n\t\t\\item \\textbf{Rank-based CTR} - modeling a probability for every rank on the result page: $P(C_{u_r} = 1) = \\rho_r$\n\t\t\\item \\textbf{Query-document CTR} - modeling a probability for every query-document pair in the dataset: $P(C_{u}=1) = \\rho_{uq}$\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Position-based model}\n\\begin{itemize}\n\t\\item Position-based models take the position \\textit{and} the document-query pair into account for modeling the probability of a click\n\t\\begin{itemize}\n\t\t\\item \\textit{Examination} - reading a snippet at a rank/position $\\implies$ $P(E_r = 1) = \\gamma_r$\n\t\t\\item \\textit{Attractiveness} - prob. for document-query relevance $\\implies$ $P(A_{uq} = 1) = \\alpha_{uq}$\n\t\t\\item The combined probability of clicking on a document is therefore: $$P(C_u = 1) = P(E_{r_u} = 1) \\cdot P(A_{uq} = 1)$$\n\t\\end{itemize}\n\t\\item The model is visualized in Figure~\\ref{img:click_models_PBM_pgm}\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.25\\textwidth]{figures/click_models_PBM_pgm.png}\n\t\t\\caption{Probabilistic graphical model of parameters for PBM}\n\t\t\\label{img:click_models_PBM_pgm}\n\t\\end{figure}\n\t\\item The examination models the position bias in user clicks while the attractiveness covers the document relevance\n\t\\item \\textit{Advantages}: Distinguishing between position bias and document relevance\n\t\\item \\textit{Disadvantages}: the Position-based model assumes that all clicks are independent of each other. Models that overcome this include:\n\t\\begin{itemize}\n\t\t\\item \\textit{User browsing model (UBM)} - examination is also based on the rank of the previously clicked document $\\implies$ $P(E_{r,r'}=1) = \\gamma_{r,r'} $ ($n + n\\cdot (n-1)/2$ parameters $\\to$ 55 parameters for $n=10$)\n\t\t\\item \\textit{Cascade model} - see next section\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Cascade model}\n\\begin{itemize}\n\t\\item The cascade model assumes that the user scans the documents from top to bottom until he finds a relevant document and clicks\n\t\\item Thus, the top document is always examined, while following documents are only examined if none of the previous ones were clicked\n\t\\item The cascade model can be summarized in the equations:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tP(A_r = 1) & = \\alpha_{u_r q}\\\\\n\t\t\tP(E_1 = 1) & = 1 \\textit{\\hspace{7mm} first element is always examined}\\\\\n\t\t\tP(E_r = 1|C_{r-1} = 1) & = 0 \\textit{\\hspace{7mm} stop if previous document is clicked}\\\\\n\t\t\tP(E_r = 1|E_{r-1} = 0) & = 0 \\textit{\\hspace{7mm} only examine if none of the documents before was clicked}\\\\\n\t\t\tP(E_r = 1|E_{r-1}=1, C_{r-1}=0) & = 1 \\textit{\\hspace{7mm} if no click was performed yet, examine next document}\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Therefore, the model has no parameters for examination and solely relies on attractiveness. The corresponding PGM is visualized in Figure~\\ref{img:click_models_CM_pgm}\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/click_models_CM_pgm.png}\n\t\t\\caption{Probabilistic graphical model of parameters for CM}\n\t\t\\label{img:click_models_CM_pgm}\n\t\\end{figure}\n\t\\item \\textit{Advantages}: Clicking on a document depends on previous decisions/documents\n\t\\item \\textit{Disadvantages}: No skips are allowed. Also, the cascade model only considers a single click $\\implies$ Dynamic Bayesian Networks\n\\end{itemize}"
  },
  {
    "path": "Information_Retrieval_1/ir_counterfactual_eval.tex",
    "content": "\\section{Counterfactual Evaluation and Learning to Rank}\n\\begin{itemize}\n\t\\item The term \\textit{counterfactual} relates to \\textit{off-policy} learning in RL\n\t\\item Thus, we try to evaluate an offline task by using online data obtained by another policy to estimate the performance of the new policy in a online setting\n\\end{itemize}\n\\subsection{Counterfactual Evaluation}\n\\begin{itemize}\n\t\\item In general, a user interactive system can be formalized as follows (see Figure~\\ref{img:counterfactual_user_interactive_system}):\n\t\\begin{itemize}\n\t\t\\item $x$: Feature vector describing the user and context (i.e. query)\n\t\t\\item $y$: Result the system returns based on its policy ($y=\\pi(x)$)\n\t\t\\item $\\delta$: Feedback signal from the actions a user took. The function encodes the metric (user utility function) and is defined as $\\delta: X\\times Y\\to \\mathbb{R}$\n\t\t\\item $\\pi$: Policy describing the ranking system which takes $x$ as input and maps it to output $y$: $\\pi:X\\to Y$\n\t\\end{itemize}\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/counterfactual_user_interactive_system.png}\n\t\t\\caption{Visualization fo a user interactive system}\n\t\t\\label{img:counterfactual_user_interactive_system}\n\t\\end{figure}\n\t\\item \\textit{Counterfactual evaluation}: perform offline evaluation of online metrics given online data from another system $\\pi_{\\text{production}}$. Thus, we try to estimate performance of $\\pi_{\\text{new}}$ with interaction data obtained with $\\pi_{\\text{production}}$.\n\t\\item The interactions data/log is structured as $D=\\left\\{\\left(x_1, y_1, \\delta_1\\right),...,\\left(x_n, y_n, \\delta_n\\right)\\right\\}$\n\t\\begin{itemize}\n\t\t\\item The actions $y_i$ were selected by $\\pi_{\\text{production}}:X\\to Y$\n\t\t\\item Note that we only have partial information feedback, and no complete supervision. Only for the chosen action, we know the feedback signal/user utility. Thus, the \"correct\"/optimal action is unknown (also called \"bandit feedback\" as it was sampled from only one arm)\n\t\\end{itemize}\n\t\\item We want to estimate $\\mathbb{E}_{y\\sim \\pi_{\\text{new}}}\\left[\\delta(x,y)\\right]$ given $D$ from $\\pi_{\\text{production}}$. For this, there are two approaches: \\textit{model the rewards} and \\textit{inverse propensity scoring}.\n\\end{itemize}\n\\subsubsection{Model the rewards}\n\\begin{itemize}\n\t\\item The intuition behind \\textit{model the rewards} is to learn the reward function $\\delta:X\\times Y\\to \\mathbb{R}$ from $D\\sim \\pi_{\\text{production}}$ directly\n\t\\item The task can be reduced to a regression problem: $$\\delta_w = \\arg\\min_{\\delta_w} \\sum\\limits_{i=1}^{N} \\mathcal{L}\\left(\\delta_w\\left(x_i, y_i\\right), \\delta_i \\right)$$\n\twhere $\\mathcal{L}$ is a loss function like MSE.\n\t\\item Once $\\delta_w$ is learned, we can estimate our goal by $\\mathbb{E}_{y\\sim \\pi_{\\text{new}}}\\left[\\delta(x,y)\\right] = \\frac{1}{n} \\sum\\limits_{i=1}^{N} \\delta_w \\left(x_i, \\pi_{\\text{new}}(x_i)\\right)$\n\t\\item However, learning $\\delta_w$ is in general very difficult, as:\n\t\\begin{itemize}\n\t\t\\item Input space $X\\times Y$ is very high-dimensional\n\t\t\\item Rewards are highly non-linear and noisy\n\t\t\\item Data is strongly biased to the actions that $\\pi_{\\text{production}}$ prefers\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Inverse Propensity Scoring}\n\\begin{itemize}\n\t\\item Instead of learning $\\delta_w$, is it possible to directly estimate the value of the new policy $\\pi_{\\text{new}}$?\n\t\\item Answer: only under the condition that the policy $\\pi_{\\text{production}}$ is stochastic: $y\\sim \\pi(y|x)$. The probability $p$ to choose the action $y$ is also called \\textit{propensity}. Note that $p>0$ must hold for all possible actions as we otherwise have no chance to discover/obtain feedback for all actions\n\t\\item For unbiased counterfactual evaluation, we need data samples with the propensity $p_i$ from policy $\\pi_{\\text{production}}$ describing the probability of selecting $y_i$ for given input $x_i$: $\\left(x_i, y_i, \\delta_i, p_i\\right)$\n\t\\item Use importance sampling to make distributions $\\pi_{\\text{production}}$ and $\\pi_{\\text{new}}$ comparable. This leads to the \\textbf{IPS-estimator}:\n\t$$\\frac{1}{n}\\sum\\limits_{i=1}^{N} \\delta_i \\frac{\\pi_{\\text{new}}(y_i|x_i)}{p_i}$$\n\\end{itemize}\n\\subsubsection{Proof of Unbiasedness}\n\\begin{itemize}\n\t\\item We want to proof that in expectation, the IPS estimator will lead to the correct value: $$\\mathbb{E}_{y\\sim \\pi_{\\text{production}}}\\left[\\frac{1}{n}\\sum\\limits_{i=1}^{N} \\delta_i \\frac{\\pi_{\\text{new}}(y_i|x_i)}{p_i}\\right] = \\mathbb{E}_{y\\sim \\pi_{\\text{new}}}\\left[\\delta(x,y)\\right]$$\n\t\\item First, we can put the sum outside the expectation:\n\t$$\\mathbb{E}_{y\\sim \\pi_{\\text{production}}}\\left[\\frac{1}{n}\\sum\\limits_{i=1}^{N} \\delta_i \\frac{\\pi_{\\text{new}}(y_i|x_i)}{p_i}\\right] = \\frac{1}{n}\\sum\\limits_{i=1}^{N} \\mathbb{E}_{y\\sim \\pi_{\\text{production}}}\\left[\\delta_i \\frac{\\pi_{\\text{new}}(y_i|x_i)}{p_i}\\right]$$\n\t\\item Next, we replace the expectation by a sum over actions weighted by their corresponding probabilities:\n\t$$\\frac{1}{n}\\sum\\limits_{i=1}^{N} \\mathbb{E}_{y\\sim \\pi_{\\text{production}}}\\left[\\delta_i \\frac{\\pi_{\\text{new}}(y_i|x_i)}{p_i}\\right] = \\frac{1}{n}\\sum\\limits_{i=1}^{N} \\sum\\limits_{y_i\\in Y}\\left[\\pi_{\\text{production}}(y_i|x_i)\\delta_i \\frac{\\pi_{\\text{new}}(y_i|x_i)}{p_i}\\right]$$\n\t\\item As $p_i$ is defined as $\\pi_{\\text{production}}(y_i|x_i)$, we can reduce the equation to:\n\t$$\\frac{1}{n}\\sum\\limits_{i=1}^{N} \\sum\\limits_{y_i\\in Y}\\left[\\pi_{\\text{production}}(y_i|x_i)\\delta_i \\frac{\\pi_{\\text{new}}(y_i|x_i)}{p_i}\\right] = \\frac{1}{n}\\sum\\limits_{i=1}^{N} \\sum\\limits_{y_i\\in Y}\\left[\\delta_i \\pi_{\\text{new}}(y_i|x_i)\\right]$$\n\t\\item Finally, we apply rules based on the definition of expectation:\n\t$$\\frac{1}{n}\\sum\\limits_{i=1}^{N} \\sum\\limits_{y_i\\in Y}\\left[\\delta_i \\pi_{\\text{new}}(y_i|x_i)\\right] = \\frac{1}{n}\\sum\\limits_{i=1}^{N} \\mathbb{E}_{y_i\\sim \\pi_{\\text{new}}}\\left[\\delta(x_i,y_i)\\right] = \\mathbb{E}_{y\\sim \\pi_{\\text{new}}}\\left[\\delta(x,y)\\right]$$\n\t\\item Note that the IPS estimator has a high variance which scales with $p_i^2$. Thus, if we have a very low probability for some actions, this can introduce a high error $\\implies$ many samples needed to approximate target accurately. There are different approaches to reduce the variance\n\\end{itemize}\n\\subsection{Counterfactual Learning to Rank}\n\\begin{itemize}\n\t\\item Learning to Rank: \\textit{offline} - train on labeled data, \\textit{online} - learn from user interactions, \\textit{counterfactual} - learn offline from online retrieved data obtained by another policy/ranker\n\t\\item The goal of counterfactual LTR is to learn a new ranker $\\pi_{\\text{new}}$ from the interaction data with $\\pi_{\\text{production}}$\n\t\\begin{itemize}\n\t\t\\item The data is specified by $D=\\left\\{(x_1,y_1,\\delta_1),...,(x_N,y_N,\\delta_N)\\right\\}$ where $\\delta_i$ indicates which document was clicked (we assume that only one document was clicked)\n\t\t\\item $y_i$ is the ranking selected by $\\pi_{\\text{production}}:X\\to Y$\n\t\\end{itemize}\n\t\\item Naive approach: assume click indicates relevance and learn as if it would be a supervised dataset: $$\\pi_{\\text{new}} = \\arg\\min_{\\pi} \\sum\\limits_{i=1}^{N} \\text{rank}\\left(\\pi(x_i),y_i,\\delta_i\\right)$$\n\tThe objective function is to reduce the rank of the relevant document given the new ranking of $\\pi_{\\text{new}}$ and the previous ranking by $y_i$. Can be solved by pairwise LTR objective.\n\t\\item However, data obtained by online Learning to Rank is commonly noisy and biased\n\t\\item We can take these biases into account by using the inverse propensity scores:\n\t$$\\pi_{\\text{new}} = \\arg\\min_{\\pi} \\sum\\limits_{i=1}^{N} \\frac{\\text{rank}\\left(\\pi(x_i),y_i,\\delta_i\\right)}{p(\\textit{observing }\\delta_i)}$$\n\tThis formula can be motivated from a probabilistic click model perspective:\n\t$$p(\\textit{click}) = p(\\textit{observation})\\times p(\\textit{relevant}) \\implies p(\\textit{relevant}) = \\frac{p(\\textit{click})}{p(\\textit{observation})}$$\n\tLeft side is what we want to get, and on the right side it is specified what we actually optimize.\n\\end{itemize}\n\\subsubsection{Propensity estimation}\n\\begin{itemize}\n\t\\item However, the question remains how we calculate $p(\\textit{observing }\\delta_i)$. We can either approximate it by using click models, or by performing a randomization test\n\t\\item \\textit{RandTopN}\n\t\\begin{itemize}\n\t\t\\item Randomly shuffle the top $N$ documents\n\t\t\\item Measure clicks people have performed on the data (online experiment)\n\t\t\\item Aggregate clicks for infinite samples\n\t\t\\item Infer $\\hat{p} \\propto p(\\textit{observing} \\delta_i)$\n\t\\end{itemize} \n\t\\item \\textit{RandPair}\n\t\\begin{itemize}\n\t\t\\item Randomly swap top document with random top $N$ documents\n\t\t\\item Infers $\\frac{p(\\textit{observing} \\delta_i)}{p(\\textit{observing} \\delta_j)}$ for swapped documents $i$ and $j$\n\t\\end{itemize}\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\begin{subfigure}[b]{0.45\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.8\\textwidth]{figures/counterfactual_LTR_RandTopN.png}\n\t\t\t\\caption{RandTopN}\n\t\t\\end{subfigure}\n\t\t\\begin{subfigure}[b]{0.45\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.8\\textwidth]{figures/counterfactual_LTR_RandPair.png}\n\t\t\t\\caption{RandPair}\n\t\t\\end{subfigure}\n\t\t\\label{img:counterfactual_propensity_estimation}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{The Variance problem}\n\\begin{itemize}\n\t\\item The problem of solving the counterfactual approach is that if $p(\\textit{observing} \\delta_i)$ heads to $0$, the overall objective will be heavily biased towards this example $\\implies$ overfitting on single data point\n\t\\item One way to overcome this problem is using a variance regularizer which prevents the policy to deviate too much from the original production policy $\\pi_{\\text{production}}$:\n\t$$\\pi_{\\text{new}} = \\arg\\min_{\\pi} \\sum\\limits_{i=1}^{N} \\frac{\\text{rank}\\left(\\pi(x_i),y_i,\\delta_i\\right)}{p(\\textit{observing }\\delta_i)} + \\lambda \\sqrt{\\frac{\\mathcal{V}[\\pi, \\pi_{\\text{production}}]}{n}}$$\n\t\\item However, this optimization problem cannot be solved by SGD anymore and iterative methods must be applied (new learning framework \\textit{counterfactual risk minimization})\n\\end{itemize}"
  },
  {
    "path": "Information_Retrieval_1/ir_language_models.tex",
    "content": "\\section{Introduction to Retrieval models}\n\\begin{itemize}\n\t\\item Mathematical framework for defining query-document matching\n\\end{itemize}\n\\subsection{TF-IDF}\n\\begin{itemize}\n\t\\item In a vector space model, documents and queries are represented in vector space\n\t\\item Axes are mostly terms/vocabulary so that a document or query is represented by terms they contain (or their frequency)\n\t\\item We can rank documents based on their cosine similarity with the query:\n\t$$\\text{score}(d,q) = \\frac{\\vec{q} \\cdot \\vec{d}}{||\\vec{q}||\\cdot ||\\vec{d}||}$$\n\t\\item Documents can be therefore represented as non-negative vector of term weights (raw frequency in doc):\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/language_models_tf_example.png}\n\t\t\\label{img:language_models_tf_example}\n\t\\end{figure}\n\t\\item However, the problem here is that terms with a higher frequency in documents are automatically more important, although this is not always the case (e.g. \"the\"). Thus, for identifying the important terms, we can report document frequency (no. of docs in which terms occurs):\n\t$$\\text{df}(t) \\coloneqq \\#\\left\\{d:\\text{tf}(t;d)>0\\right\\}$$\n\t\\item We can translate document frequencies to term weights by inverting them (inverted document frequency - \\textit{IDF}):\n\t$$\\text{idf}(t) = \\log \\frac{n}{\\text{df}(t)} = \\log n - \\log \\text{df}(t)$$\n\tThe log is applied to dampen the effect of IDF.\n\t\\item Also the term frequencies should be dampened by a monotonic, sub-linear transformation as a term occurring twice as often doesn't imply that the document is also twice as important/relevant. Together, we can define the tf-idf weights as follows:\n\t$$\\text{tf-idf}(t;d) = \\log \\left(1+\\text{tf}(t;d)\\right) \\log \\frac{n}{\\text{df}(t)}$$\n\t\\item Scores are normalized by euclidean distance of document. Alternatively, we could also apply tf-idf on the relative term frequencies.\n\\end{itemize}\n\\subsection{BM25}\n\\begin{itemize}\n\t\\item Probabilistic retrieval framework that extends the idea of tf-idf\n\t\\item Instead of the log, we use a different damping functions which are easier to control:\n\t$$w_t = \\frac{(k_1 + 1)\\cdot \\text{tf}(d;t)}{k_1 + \\text{tf}(d;t)}\\cdot \\text{idf}(t)$$\n\t\\item In addition, we normalize the term frequency by the document length: $\\text{tf}'(d;t) = \\text{tf}(d;t) \\cdot l_{avg}/l_{d}$ ($l_{avg}$ is the average document length of collection). By this we prevent copies of documents concatenated with each other being higher rated. Putting this into our original function, we get:\n\t$$w_t = \\frac{(k_1 + 1) \\cdot \\text{tf}(d;t)}{k_1 \\cdot (l_d / l_{avg})+ \\text{tf}(d;t)}\\cdot \\text{idf}(t)$$\n\t\\item However, longer documents also tend to contain more information. Thus, we introduce another parameter $b$ that controls the normalization:\n\t$$w_t = \\frac{(k_1 + 1) \\cdot \\text{tf}(d;t)}{k_1 \\cdot ((1-b) + b\\cdot (l_d / l_{avg}))+ \\text{tf}(d;t)}\\cdot \\text{idf}(t)$$\n\t\\item For very long queries, we also need to consider this normalization which can be done by multiplying another term $\\frac{(k_3 + 1)\\cdot \\text{tf}(q;t)}{k_3 \\cdot \\text{tf}(q;t)}$\n\t\\item In conclusion, the BM25 score is calculated as follows:\n\t$$\\text{BM25} = \\sum\\limits_{\\text{unique\\hspace{1mm}} t\\in q} \\frac{(k_1 + 1) \\cdot \\text{tf}(d;t)}{k_1 \\cdot ((1-b) + b\\cdot (l_d / l_{avg}))+ \\text{tf}(d;t)} \\cdot \\frac{(k_3 + 1)\\cdot \\text{tf}(q;t)}{k_3 + \\text{tf}(q;t)} \\cdot \\text{idf}(t)$$\n\t\\item Parameters $k_1$, $b$ and $k_3$ are tuned. Common defaults are $k_1 = 1.5$ and $b=0.75$\n\t\\item It is the most widely used ranking in IR but only loosely inspired by probabilistic models\n\\end{itemize}\n\\subsection{Statistical Language Models}\n\\begin{itemize}\n\t\\item Statistical language models are a probability distribution over word sequences $P(w_1, ..., w_m)$ with which documents and queries can be represented (and uncertainty quantified)\n\t\\item Thus, a language model describes the probability of e.g. $q$ being the given word sequence\n\t\\item Documents are ranked given a query by its similarity. Therefore we can use either document likelihood, query likelihood or KL-divergence\n\\end{itemize}\n\\subsubsection{Query likelihood}\n\\begin{itemize}\n\t\\item Given a document, what queries are most likely to be created for it? \n\t\\item We first have to ensure that the query likelihood correlates with document likelihood. Therefore, we apply the Bayes rule: $p(d|q) = \\frac{p(q|d)p(d)}{p(q)}$. As $p(q)$ is equal for all documents, and we assume a uniform prior for all documents (though not always the case), we retrieve $p(d|q)\\propto p(q|d)$\n\t\\item Thus, by generating a probability distribution of possible queries for a document, we can approximate how likely a document is given a query.\n\t\\item The scoring function is defined as follows:\n\t$$\\text{score}(d,q) = \\log \\left[p(q|\\theta_d)\\cdot p(d)\\right]$$\n\twhere $\\theta_d$ describes the document. There are mainly three modeling choices:\n\t\\begin{enumerate}\n\t\t\\item \\textit{How to define the generative process $p|\\theta_d$?}\n\t\t\\begin{itemize}\n\t\t\t\\item Given $\\theta_d$, what is the generative process for getting $q=w_1,...,w_{|q|}$?\n\t\t\t\\item Different distributions are possible\n\t\t\t\\item \\textit{Multiple Bernoulli} - bag of word perspective, every word in vocabulary has probability to be in query or not. The related probability is:\n\t\t\t$$p(q|\\theta_d) = \\prod\\limits_{w_i \\in q} p(X_i = 1 | \\theta_d) \\prod\\limits_{w_i \\not\\in q} \\left(1 - p\\left(X_i = 1 | \\theta_d\\right) \\right)$$\n\t\t\t\\item \\textit{Multinomial} - similar to bernoulli, but we know have a random variable for every word slot in the query and not one for every word in the vocabulary. Thus, the calculation is:\n\t\t\t$$p(q|\\theta_d) = \\prod\\limits_{w_i \\in q} p(w_i | \\theta_d) \\text{\\hspace{4mm}where\\hspace{4mm}} \\sum\\limits_{w_i \\in V} p(w_i|\\theta_d) = 1$$\n\t\t\t\\item \\textit{Multiple Poisson} - similar to bernoulli, but instead of presence or absence, we model the number of times we expect a word from the vocabulary to occur in the query of length $|q|$ by a Poisson distribution:\n\t\t\t$$p(q|\\theta_d) = \\prod\\limits_{w_i \\in V} \\frac{e^{-\\lambda_i |q|} (\\lambda_i |q|)^{\\text{tf}(w_i;d)}}{\\text{tf}(w_i;d)!}$$\n\t\t\\end{itemize}\n\t\t\\item \\textit{How to estimate $\\theta_d$ based on document $d$?}\n\t\t\\begin{itemize}\n\t\t\t\\item To estimate $\\theta_d$ we perform MLE: $\\hat{\\theta}_d = \\arg \\max_{\\theta_d} p(d|\\theta_d)$\n\t\t\t\\item In case of a multinomial distribution, we would get:\n\t\t\t$$p(d|\\theta_d) = \\prod\\limits_{w_i \\in V} p(w_i | \\theta_d)^{\\text{tf}(w_i;d)} \\implies \\log p(d|\\theta_d) = \\sum\\limits_{w_i \\in V} \\text{tf}(w_i;d) \\log p(w_i | \\theta_d)$$\n\t\t\t\\item Note that this is a constrained optimization problem with $\\sum\\limits_{w_i \\in V} p(w_i|\\theta_d) = 1$.\n\t\t\t\\item By using lagrangian multiplier, we get $p_{MLE}(w_i|d) = \\frac{\\text{tf}(w_i;d)}{|d|}$\n\t\t\\end{itemize}\n\t\t\\item \\textit{How to compute prior $p(d)$?}\n\t\t\\begin{itemize}\n\t\t\t\\item The prior takes everything into account which is independent of a query.\n\t\t\t\\item This can include number of clicks, credibility, ...\n\t\t\\end{itemize}\n\t\\end{enumerate}\n\\end{itemize}\n\\subsubsection{Smoothing}\n\\begin{itemize}\n\t\\item How to deal with unseen words which have a probability of 0.\n\t\\item First, we assume a multinomial distribution again with the optimal parameters of $p(w_i|\\theta_d) = \\frac{\\text{tf}(w_i;d)}{|d|}$\n\t\\item \\textbf{Adaptive smoothing}: add a small extra count to every word:\n\t$$p(w_i|\\theta_d) = \\frac{\\text{tf}(w_i;d) + \\epsilon}{|d| + \\epsilon |V|}$$\n\tIn case of $\\epsilon=0$, we fall back to ML estimation. $\\epsilon=1$ is called Laplace smoothing.\n\t\\item \\textbf{Jelinek-Mercer smoothing}: linearly interpolate with \"background\" knowledge so that rare words also have smaller additives:\n\t$$p_{\\lambda}(w_i|\\theta_d) = \\lambda \\frac{\\text{tf}(w_i;d)}{|d|} + (1 - \\lambda) \\frac{\\text{tf}(w_i;C)}{|C|}$$\n\tThe context $C$ is approximated by the concatenation of all documents.\n\t\\item \\textbf{Dirichlet prior smoothing}: we assume that before seeing the document, we have a prior belief over all words $p(\\theta_d)$. We use the posterior which gets narrower the more words we see and therefore the more certain we are about the document distribution.\n\t\\begin{itemize}\n\t\t\\item Maximum A Posteriori estimate by $\\hat{\\theta}_d = \\arg\\max_{\\theta_d} p(\\theta_d|d) = \\arg\\max_{\\theta_d} p(d|\\theta_d) p(\\theta_d)$\n\t\t\\item Prior distribution $p_i\\sim \\text{Dir}(\\alpha) \\implies p(\\theta_d) = \\prod\\limits_{w \\in V} p(w|\\theta_d)^{\\alpha_w - 1}$\n\t\t\\item With a multinomial likelihood, we get:\n\t\t$$p(\\theta_d | d) \\propto \\prod\\limits_{w \\in V} p(w|\\theta_d)^{\\text{tf}(w;d)} \\prod\\limits_{w \\in V} p(w|\\theta_d)^{\\alpha_w - 1} = \\prod\\limits_{w \\in V} p(w|\\theta_d)^{\\text{tf}(w;d) + \\alpha_w - 1}$$\n\t\t\\item Thus, our new MAP solution is:\n\t\t$$p(w|\\theta_d) = \\frac{\\text{tf}(w;d) + \\alpha_w - 1}{|d| + \\sum_{w\\in V}\\alpha_w - |V|}$$\n\t\t\\item For $\\alpha_w = 1$, we get MLE estimation, and $\\alpha_w = 2$ represents Laplace smoothing.\n\t\t\\item We can also rewrite the smoothing similar to Jelinek-Mercer smoothing:\n\t\t$$p(w|\\theta_d) = \\frac{|d|}{|d|+ \\mu}\\frac{\\text{tf}(w;d)}{|d|} + \\frac{\\mu}{\\mu + |d|}p(w|C)$$\n\t\twhere $\\mu$ is the parameter depending on $\\alpha_w$. Thus, we interpolate with the background knowledge while taking the document length into account.\n\t\\end{itemize}\n\t\\item Next to Dirichlet prior smoothing, we can also use other distributions (for example a beta prior with multiple Bernoulli) which lead to slightly different smoothing functions. For example, with the beta prior, we get for a variable $\\alpha_w$ and $\\beta_w$ (without constraints!):\n\t$$p(w|\\theta_d) = \\frac{\\text{tf}(w;d) + \\alpha_w - 1}{\\alpha_w + \\beta_w - 1}$$\n\\end{itemize}\n\\subsubsection{Positional Language Models}\n\\begin{itemize}\n\t\\item There are variants of basic language models capturing term dependencies\n\t\\item Instead of having one language model representing the whole document, Positional Language Models define a LM for every word position\n\t\\item Thus we capture (small) \"fuzzy\" passages with which we can match our query\n\t\\item A term at each position can propagate its occurrence to close positions in word windows\n\t\\begin{itemize}\n\t\t\\item Example sentence: \\texttt{the black hat is not}...\n\t\t\\item With a equally weighted word window of one, we would retrieve the following language model (MLE params) for the position of word \"\\texttt{black}\": $p(\\texttt{black}|\\theta_p) = 1/3, \\hspace{2mm} p(\\texttt{the}|\\theta_p) = 1/3, \\hspace{2mm} p(\\texttt{hat}|\\theta_p) = 1/3$\n\t\\end{itemize}\n\t\\item We can weight the occurrences of every word based on the distance to the \"root\" of the language model (also called kernel):\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/language_models_positional.png}\n\t\t\\label{img:language_models_positional}\n\t\\end{figure}\n\t\\item In general, the term frequency of a word for a LM at position $j$ with kernel $k$ is determined as follows:\n\t$$\\text{tf'}(w,j;d) = \\sum\\limits_{i=1}^{|d|} \\text{tf}(w,i;d) \\cdot k(i,j) $$\n\t\\item The language model at every position is given by the corresponding MLE estimation:\n\t$$p(w|d,j) = \\frac{\\text{tf'}(w,j;d)}{\\sum_{w'\\in V} \\text{tf'}(w',j;d)}$$\n\t\\item Documents can now be scored by either their best matching language model with the query, or the average of the top-$k$ models\n\\end{itemize}"
  },
  {
    "path": "Information_Retrieval_1/ir_learning_to_rank.tex",
    "content": "\\section{Learning to Rank}\n\\begin{itemize}\n\t\\item Main issue in information retrieval is to determine whether document $d$ is relevant for query $q$\n\t\\item Common relevance signals include TF-IDF, BM25, document popularity etc.\n\t\\item But: what signals to use/how to combine these signals? There is not a single relevance signal \"to rule them all\" $\\implies$ combine all signals in a model\n\t\\item Simplest combination method: linear model $f(\\bm{d},\\bm{\\theta}) = \\sum\\limits_{i=1}^{|d|} \\theta_i d_i$ where $\\bm{d}$ represents the different signals for document-query pair\n\t\\item Task: find the optimal parameter set $\\bm{\\theta}$, commonly by Machine Learning techniques (linear regression)\n\\end{itemize}\n\\subsection{Offline Learning To Rank}\n\\begin{itemize}\n\t\\item Given an annotated dataset of relation document and relevance/ranking\n\t\\item There are three different approaches\n\t\\begin{enumerate}\n\t\t\\item \\textbf{Pointwise}: optimize models $f(\\bm{d},\\bm{\\theta})$ to predict relevancy of a document. This can be recasted in a regression problem with loss:\n\t\t$$\\mathcal{L}=\\sum_{\\bm{d}} \\left(f(\\bm{d},\\bm{\\theta}) - \\text{relevancy}(d,q)\\right)^2$$\n\t\tHowever, this approach does not consider the application of ranking where only the final order is important, but not the single scores.\n\t\t\\item \\textbf{Pairwise}: optimize regarding the total order of the documents and not specific relevance scores. The loss can be expressed by:\n\t\t$$\\mathcal{L}=\\sum_{d\\succ d'}\\left[f(\\bm{d'},\\bm{\\theta}) - f(\\bm{d},\\bm{\\theta})\\right]$$ \n\t\twhere $d\\succ d'$ means that $d'$ is the successor of $d$ in the labeled ranking. Nevertheless, this method does not take into account that only a subpart (top 10) of the collection is actually presented to the user.\n\t\t\\item \\textbf{Listwise}: optimize regarding ranking metrics like $DCG$. Thus, the loss could be:\n\t\t$$\\mathcal{L} = -nDCG(f(\\cdot,\\bm{\\theta}))$$\n\t\tThe problem is that most ranking metrics are not differentiable. There are heuristic approaches to still optimize with respect to such metrics. \n\t\\end{enumerate}\n\t\\item Problems with offline Learning to Rank: similar to offline evaluation in Section~\\ref{sec:offline_eval_problems}\n\t\\begin{itemize}\n\t\t\\item All described methods require an annotated dataset which contains either relevance labels for each document-query pair or a ranking over the whole collection.\n\t\t\\item Creating such is time consuming and expensive\n\t\t\\item Impossible to personalize for a user (everyone prefers a little bit different documents). Also, annotators and users might disagree in some points $\\implies$ dataset does not fully reflect user behavior\n\t\t\\item Can change over time\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Online Learning to Rank}\n\\begin{itemize}\n\t\\item Learn from implicit user feedback\n\t\\begin{itemize}\n\t\t\\item Might be noisy\n\t\t\\item Consider position bias (higher rank is more frequently clicked) and selection bias (only a limited set of documents is presented to the user)\n\t\\end{itemize}\n\t\\item Online Learning to Rank methods can learn from user interactions, \\textbf{and} control the results which are displayed/presented to the user\n\t\\item Thus, these methods can be more efficients as they control over what data is actually gathered\n\t\\item A general online learning to rank technique is visualized in Figure~\\ref{img:learning_to_rank_online_overview}\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/learning_to_rank_online_overview.png}\n\t\t\\caption{Overview of the general concept of online learning to rank}\n\t\t\\label{img:learning_to_rank_online_overview}\n\t\\end{figure}\n\t\\begin{itemize}\n\t\t\\item The user enters a query, for which the ranking algorithm generates a list of documents\n\t\t\\item The Online Learning to Rank system interacts with the results by adding and/or removing documents from the ranking. This can also include interleaving with another, slightly changed ranking algorithm\n\t\t\\item User interacts with the displayed result and gives implicit feedback.\n\t\t\\item The Online Learning to Rank algorithm updates the ranking parameters according to the analyzed feedback\n\t\\end{itemize}\n\t\\item \\textbf{Advantages}: learns directly from the user, is more responsive by immediately adapting its parameters\n\t\\item \\textbf{Risks}:\n\t\\begin{itemize}\n\t\t\\item Unreliable methods will affect/worsen user experiences immediately.\n\t\t\\item (Noisy) clicks can easily bias or even manipulate search engines\n\t\t\\item \\textbf{Self-confirming loop}\n\t\t\\begin{itemize}\n\t\t\t\\item If an irrelevant document was clicked by random, the system still perceives that this document is relevant and will change its parameters accordingly\n\t\t\t\\item Thus, the random document will be placed higher in future ranks. However, also similar documents to the irrelevant one will have an increased relevance score and will probably occur at a high position\n\t\t\t\\item Most likely, the next clicked document will be one of the highest ones which were irrelevant $\\implies$ entering a self-confirming loop\n\t\t\t\\item Due to bias and noise, an irrelevant document was clicked and inferred to be relevant\n\t\t\t\\item Due to noise, this inference is most likely to appear again\n\t\t\t\\item The algorithms confidence in this incorrect inference continues to increase\n\t\t\\end{itemize}\n\t\\end{itemize}\n\t\\item To prevent a self-confirming loop, we have to balance exploration and exploitation\n\t\\begin{itemize}\n\t\t\\item \\textit{Exploration}: collect feedback for learning from the most documents as possible\n\t\t\\item \\textit{Exploitation}: utilize what has been already learned \n\t\t\\item If systems only exploits, it misses out to obtain feedback for other documents that might be even better (danger to enter/staying in self-confirming loop)\n\t\t\\item To high exploration rate leads to a lot of irrelevant documents in ranking that worsen the user experience \n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Designing an Online Learning to Rank algorithm}\n\\begin{itemize}\n\t\\item To design a OLTR algorithm, we have to make design choices in four aspects (see Figure~\\ref{img:learning_to_rank_online_design})\n\\end{itemize}\n\\begin{figure}[ht]\n\t\\centering\n\t\\includegraphics[width=0.4\\textwidth]{figures/learning_to_rank_online_design.png}\n\t\\caption{General design components of an OLTR algorithm}\n\t\\label{img:learning_to_rank_online_design}\n\\end{figure}\n\\begin{enumerate}[label=(\\Alph*)]\n\t\\item \\textbf{Ranker}: the ranker maps documents to relevance scores. This module operates on feature level/document id's and can be for example a linear ranker/neural model/...\n\t\\item \\textbf{Exploration strategy}: define interactions with results of the ranker. No exploration would mean that the document ranking is simply passed and stays unchanged. A common strategy is \\textit{epsilon-greedy} where we inject random documents in random positions with ratio $\\epsilon$. Other algorithms include upper confidence bound etc.\n\t\\item \\textbf{Signal recording and interpretation}: algorithm can consider multiple signals (raw observation like clicks and dwell time, more complex metrics like time to success). Should remove bias/noise. When result list was constructed by using interleaving, the feedback would also consider which ranker has won based on user interactions.\n\t\\item \\textbf{Update mechanism}: update ranking algorithm given the user feedback. If ranker operates on document id's, we can update the document's specific relevance estimate for the query. If the ranker relies on features, we optimize a loss function like the ones shown in offline LTR.\n\\end{enumerate}\n\\subsubsection{Dueling Bandit Gradient Descent}\n\\begin{itemize}\n\t\\item One of the first OLTR algorithms was the \\textit{Dueling Bandit Gradient Descent}\n\t\\item The intuition is that we compare two rankers by online evaluation, and optimize our system towards the better performing one.\n\t\\item The method is structured in following steps (visualized in Figure~\\ref{img:learning_to_rank_online_DBGD})\n\t\\begin{enumerate}\n\t\t\\item From current feature state $\\theta_b$ of the ranker (shown in green), sample a new ranker/feature point $\\theta_c = \\theta_b + u$ laying on the unit sphere $||u||=1$ around the current one (shown in red)\n\t\t\\item Get the rankings of $\\theta_b$ and $\\theta_c$\n\t\t\\item Compare $\\theta_b$ and $\\theta_c$ using interleaving\n\t\t\\item If $\\theta_c$ wins comparison: update the current model by $\\theta_b \\leftarrow \\theta_b + \\eta (\\theta_c - \\theta_b)$\n\t\\end{enumerate}\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/learning_to_rank_online_DBGD.png}\n\t\t\\caption{Steps in Dueling Bandit Gradient Descent}\n\t\t\\label{img:learning_to_rank_online_DBGD}\n\t\\end{figure}\n\t\\item It can be shown that if there is only a single optimum, the Dueling Bandit Gradient Descent algorithm will be able to approximate the optimal model\n\\end{itemize}"
  },
  {
    "path": "Information_Retrieval_1/ir_neural_models.tex",
    "content": "\\section{Neural Retrieval Models}\n\\subsection{Distributed Word Representations}\n\\begin{itemize}\n\t\\item Latent, dense vector representation to model semantic similarity/relations\n\t\n\\end{itemize}\n\\subsubsection{Skip-gram}\n\\begin{itemize}\n\t\\item \\textbf{Skip gram}: learn to predict neighboring words in a small context window\n\t\\item Model probability by similarity between word and context vectors (two matrices):\n\t$$p(w_k|w_j) = \\frac{\\exp\\left(c_k \\cdot v_j\\right)}{\\sum_{i\\in|V|}} \\exp\\left(c_i \\cdot v_j\\right)$$\n\t\\item Denominator can be computationally expensive if vocabulary is quite large. Thus, we can approximate it by taking just a few negative examples $\\implies$ negative sampling\n\t\\item Overall, skip gram will learn two representations for each word (context $C$ and target words $V$) from which me most likely only use $V$ (visualized in Figure~\\ref{img:neural_ir_skip_gram})\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/neural_ir_skip_gram.png}\n\t\t\\caption{Visualization of skip gram method for learning word representations}\n\t\t\\label{img:neural_ir_skip_gram}\n\t\\end{figure}\n\t\\item Skip gram shows to capture relational meaning (\\texttt{KING - MAN + WOMAN = QUEEN})\n\\end{itemize}\n\\subsubsection{Using word embeddings in IR}\n\\begin{itemize}\n\t\\item \\textbf{Generalized Language Model}\n\t\\begin{itemize}\n\t\t\\item The standard language model assume that a term $t_q$ occurring in $q$ is being sampled from a document or a background collection (smoothing):\n\t\t$$p_{LM}(t_q|d) = \\lambda \\cdot p(t_q|d) + (1 - \\lambda) \\cdot p(t_q|C)$$\n\t\t\\item The generalized language model extends this idea by also considering terms that are similar to $t_q$ (for example synonyms):\n\t\t$$p_{LM}(t_q|d) = \\lambda \\cdot p(t_q|d) + \\alpha \\sum\\limits_{t'\\in d} p(t_q|t',d) p(t'|d) + \\beta \\sum\\limits_{t' \\in N_t} p(t_q|t',C) p(t'|C) + (1 - \\alpha - \\beta - \\lambda) \\cdot p(t_q|C)$$\n\t\twhere $$p(t_q|t',d) = \\frac{sim(t',t_q)}{\\sum_{t''\\in d} sim(t',t'')} \\text{\\hspace{2mm}and\\hspace{2mm}} p(t'|d) = \\frac{tf(t';d)}{|d|}$$\n\t\t$N_t$ is the set of the most similar words to $t_q$.\n\t\\end{itemize}\n\t\\item \\textbf{Word Mover's distance}\n\t\\begin{itemize}\n\t\t\\item For every word $w_i$ in the query $q$, look for the word with the highest similarity/smallest distance in document $d$\n\t\t\\item Score a document by the sum of the pairwise distances. The document with the smallest distance gets the highest rank\n\t\t\\item However, this approach doesn't care about the whole document but only the best matches\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Compositionality}\n\\begin{itemize}\n\t\\item To match queries and documents in the embedding space, we need to combine the words in each $\\implies$ compositionality\n\\end{itemize}\n\\subsubsection{Aggregate word vectors}\n\\begin{itemize}\n\t\\item Apply simple rules/arithmetic to combine word vectors\n\t\\item Example: \\textbf{Dual Embedding Space Model} \n\t\\begin{itemize}\n\t\t\\item represent a document by the centroid of its word vectors $\\bm{\\overline{D}} = \\frac{1}{|D|}\\sum_{\\bm{d}_j \\in D} \\frac{\\bm{d}_j}{||\\bm{d}_j||}$\n\t\t\\item The query-document similarity is the average over query words of cosine similarity:\\\\ $\\text{DESM}(Q,D) = \\frac{1}{|Q|}\\sum_{q_i \\in Q} \\frac{\\bm{q}_i^T \\bm{\\overline{D}}}{||\\bm{q}_i|| \\cdot ||\\bm{\\overline{D}}||}$\n\t\t\\item We can also use both the IN (word) and OUT (context) embeddings from skip-gram to optimize matching. What worked best was using IN representations for the query and OUT for document\n\t\t\\item In the ranking system, we either first rank documents by BM25 and rerank top $N$ with DESM, or use a linear combination of both scores\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Tune and Aggregate word vectors}\n\\begin{itemize}\n\t\\item Learn task-specific representations and not rely on pure skip-gram\n\t\\item \\textbf{Paragraph2vec}\n\t\\begin{itemize}\n\t\t\\item Generalizes word2vec to whole documents by embedding them in a fixed-size vector\n\t\t\\item Two different approaches. First is \\textit{distributed memory}:\n\t\t\\begin{itemize}\n\t\t\t\\item We are trying to predict the next word based on a few previous context word \\textit{and} a paragraph embedding. \n\t\t\t\\item Both the word and paragraph embeddings are learned during this process\n\t\t\t\\item Input embeddings can either be concatenated or averaged (commonly first one is applied)\n\t\t\t\\item Visualization in Figure~\\ref{img:neural_models_distributed_memory}\n\t\t\t\\begin{figure}[ht]\n\t\t\t\t\\centering\n\t\t\t\t\\includegraphics[width=0.3\\textwidth]{figures/neural_models_distributed_memory.png}\n\t\t\t\t\\caption{Distributed memory model. }\n\t\t\t\t\\label{img:neural_models_distributed_memory}\n\t\t\t\\end{figure}\n\t\t\\end{itemize}\n\t\t\\item Second method: \\textit{Distributed bag of words}\n\t\t\\begin{itemize}\n\t\t\t\\item In this approach, we don't consider the context words but try to predict all possible words in the paragraph given the embedding vector\n\t\t\t\\item This is done by sampling a random word at every SGD iteration from the small text windows, and train the classifier on predicting this word\n\t\t\t\\item Thus, we optimize the embedding regarding representing the word distribution in the paragraph\n\t\t\t\\item The distributed BOW is visualized in Figure~\\ref{img:neural_model_distributed_BOW}\n\t\t\t\\begin{figure}[ht]\n\t\t\t\t\\centering\n\t\t\t\t\\includegraphics[width=0.3\\textwidth]{figures/neural_model_distributed_BOW.png}\n\t\t\t\t\\caption{Distributed BOW model. }\n\t\t\t\t\\label{img:neural_model_distributed_BOW}\n\t\t\t\\end{figure}\n\t\t\\end{itemize}\n\t\\end{itemize}\n\t\\item \\textbf{Lexicographical definition}\n\t\\begin{itemize}\n\t\t\\item We can also use the lexicographical definitions of words to train and/or test the word embeddings\n\t\t\\item The word embeddings of the definition are combined by an (arithmetic) function $f_c$, and compared to the embedding of the word to be defined\n\t\t\\item The objective is to minimize the distance to the defined word, but maximize the distance to other words to distinguish between words\n\t\t\\item An example is shown in Figure~\\ref{img:neural_model_lexicographical_definition}\n\t\t\\begin{figure}[ht]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.3\\textwidth]{figures/neural_model_lexicographical_definition.png}\n\t\t\t\\caption{Lexicographical model for the example of \"\\textit{person}\" defined as \"\\textit{a human being}\". }\n\t\t\t\\label{img:neural_model_lexicographical_definition}\n\t\t\\end{figure}\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Tune word vectors and learn rules of composition}\n\\begin{itemize}\n\t\\item Deep, neural architectures which have different aspects to be designed\n\t\\item \\textbf{Architectures and representation}\n\t\\begin{itemize}\n\t\t\\item The simplest approach is to use neural networks to embed documents and queries to latent space, and then perform same similarity measures as before (like cosine similarity). This method is also referred to as \\textit{Projection to latent space} (see Figure~\\ref{img:neural_model_architecture_comparisons_projection_latent_space}). Possible network architectures are Convolutional NN, Recurrent NN or fixed Deep NNs if (max) input size is known\n\t\t\\item The next step is to replace the similarity measure by another neural network. Thus, this NN takes the composed embeddings of the document and query as input, and return a single real value indicating the similarity score. This approach is called \\textit{One Dimensional Matching} and visualized in Figure~\\ref{img:neural_model_architecture_comparisons_one_dim_matching}. The common architecture for the highest \n\t\t\\item Another architecture is spanning up a two-dimensional input by the query and document. Therefore, we compute similarity scores for every word in the query to every word in the document which results in a two dimensional matrix. On this, we apply a convolutional NN to end up in a fixed-size embedding. A consecutive fully-connected NN maps this embedding into a similarity score. Figure~\\ref{img:neural_model_architecture_comparisons_two_dim_matching} visualizes \n\t\\end{itemize}\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\begin{subfigure}[b]{0.3\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=\\textwidth]{figures/neural_model_latent_space.png}\n\t\t\t\\caption{Projection to Latent space}\n\t\t\t\\label{img:neural_model_architecture_comparisons_projection_latent_space}\n\t\t\\end{subfigure}\n\t\t\\hspace{2mm}\n\t\t\\begin{subfigure}[b]{0.3\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=\\textwidth]{figures/neural_models_one_dimensional_matching.png}\n\t\t\t\\caption{One dimensional matching}\n\t\t\t\\label{img:neural_model_architecture_comparisons_one_dim_matching}\n\t\t\\end{subfigure}\n\t\t\\hspace{2mm}\n\t\t\\begin{subfigure}[b]{0.3\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.8\\textwidth]{figures/neural_models_two_dim_matching.png}\n\t\t\t\\caption{Two dimensional matching}\n\t\t\t\\label{img:neural_model_architecture_comparisons_two_dim_matching}\n\t\t\\end{subfigure}\n\t\t\\label{img:neural_model_architecture_comparisons}\n\t\t\\caption{Comparison of different neural architectures for compositionality}\n\t\\end{figure}\n\t\\item \\textbf{Training}: depending on the available data, we can perform different levels of supervision\n\t\\begin{itemize}\n\t\t\\item \\textit{No supervision/labels}: If no labels are provided at all, we could autoencoders to reduce query and document to a latent space and check similarity metrics. Otherwise, we can make use of pretrained neural language models with techniques like ELMo and BERT\n\t\t\\item \\textit{Distant supervision}: We create pseudo-labels by sampling short word sequences from a document and considering this as query. The document from which we sampled is labeled as relevant/high similarity, while all other documents (from which we sample one randomly for training) are considered as being not relevant\n\t\t\\item \\textit{Weak supervision}: As alternative, we can use unsupervised ranking functions like BM25 to generate labels and use this scores for training (teacher-student architecture). In experiments, the neural network was even able to outperform BM25.\n\t\t\\item \\textit{Full supervision}: Labels are created by either annotators or using the click log as implicit feedback from the users. We can train the models in a standard supervised fashion. \n\t\\end{itemize}\n\\end{itemize}"
  },
  {
    "path": "Information_Retrieval_1/ir_offline_evaluation.tex",
    "content": "\\section{Offline evaluation}\n\\begin{itemize}\n\t\\item Evaluating an IR system without any interaction with user \n\t\\item Assumption: assessors can tell what is relevant\n\\end{itemize}\n\\subsection{Collection-based evaluation}\n\\begin{itemize}\n\t\\item Approximating user happiness by relevance of the found documents\n\t\\item There are different measures to do that\n\\end{itemize}\n\\subsubsection{Traditional Evaluation measures}\n\\begin{itemize}\n\t\\item We can view IR as a (binary) classification problem where every document is either relevant or not with respect to a query\n\t\\item The evaluation is performed by calculating precision ($\\frac{TP}{TP+FP}$) and recall ($\\frac{TP}{TP+FN}$)\n\t\\item However, the output of an IR system is a ranking and not a binary classification. Thus, we label the first $k$ documents the system proposes as relevant, and other as non-relevant $\\implies$ precision/recall $@$ cut-off ($P@k$/$R@k$)\n\t\\item The trade-off between precision and recall is task specific. For web searches, we want a high precision on the first few documents, but allow a worse recall (as a user doesn't want to find \\textit{all} relevant documents). In contrast, in medicine, we want to have a high recall to not miss an important document\n\t\\item Another way to incorporate precision and recall for ranking is using R/P curves. We start with a cut-off point of 0 where precision is 1 and recall 0. Then we increase the cut-off point one by one and record the new values in the curve (left, Figure~\\ref{img:offline_eval_RP_curves}). While the cut-off rank heads to infinity, recall goes to 1 and precision to 0. We interpolate the curves by taking the maximum value of all future values/values right from a given point.\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/offline_eval_RP_curves.png}\n\t\t\\caption{R/P curves for ranking}\n\t\t\\label{img:offline_eval_RP_curves}\n\t\\end{figure}\n\t\\item When having multiple queries, we would average the RP curves. \n\t\\item The area under the curve is the average precision which can also be calculated by taking the average of all precision values at ranks with relevant documents.\n\t\\item Usually, a binary scale of whether a document is relevant is not sufficient. For a graded relevance scale, we can use different evaluation measures\n\t\\begin{itemize}\n\t\t\\item \\textbf{Discounted Cumulative Gain (DCG)} - considers the relevance grade and position of every document. The total gain is accumulated at a certain rank $k$:\n\t\t$$DCG@k = \\sum\\limits_{\\text{rank} r=1}^{k} \\frac{2^{\\text{rel}_r} - 1}{\\log_2\\left(1 + r\\right)}$$\n\t\t\\item The numerator is the non-linear relevance score of the document at rank $r$, and the denominator the discount over ranking position\n\t\t\\item The score highly depends on the best possible ranking for a query. Thus, the DCG can be normalized by the value of the best ranking $\\implies$ $0\\leq nDCG \\leq 1$. This makes it easier to compare scores over different queries\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Model-based Evaluation measures}\n\\begin{itemize}\n\t\\item Another perspective of evaluation is looking at different aspects of possibles metrics. A model-based approach considers the following three components:\n\t\\begin{enumerate}\n\t\t\\item \\textbf{Browsing model} - describes how the user interacts with results, like the probability of a document being clicked/viewed $\\Rightarrow p(d)$\n\t\t\\item \\textbf{Model of document utility} - describes how a user derives utility from individual relevant documents. Similar to how to determine the graded relevance scale $\\Rightarrow g(d)$.\n\t\t\\item \\textbf{Utility accumulation model} - describes how a user accumulates utility in the course of browsing $\\Rightarrow E\\left[g(D)\\right] = \\sum_{r=1}^{\\infty}g(d) \\cdot p(d)$\n\t\\end{enumerate}\n\t\\item Examples for the browsing models\n\t\\begin{itemize}\n\t\t\\item \\textit{Position-based models} - the chance of observing a document depends on the position in the ranking. We can for example model it by $\\Rightarrow p(d_r)=(1-\\theta)^{r-1} \\theta$. The corresponding utility accumulation is described by Rank-biased Precision (RBP): $RBP = \\sum_{r=1}^{\\infty}\\text{rel}_r (1-\\theta)^{r-1} \\theta$\n\t\t\\item \\textit{Cascade-based models} - considers $\\theta$ as a function of the document at rank $r$. Mostly, the following function is used: $\\theta_r = \\mathcal{R}(\\text{rel}_r) = \\frac{2^{\\text{rel}_r}-1}{2^{\\text{max rel}}}$. The corresponding utility accumulation is the \\textit{Expected Reciprocal Relevance}: $ERR@k = \\sum\\limits_{r=1}^{k} \\frac{1}{r} \\cdot \\theta_r \\cdot \\prod\\limits_{i=1}^{r-1}\\left(1 - \\theta_i\\right)$\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Collection construction}\n\\begin{itemize}\n\t\\item To evaluate a system offline, we need labels of whether a document is relevant with respect to a query (or graded scale) $\\Rightarrow$ labels are created by humans\n\t\\item First step is to generate a huge document collection, and generate a set of topics/queries that should be evaluated. Mostly, queries are selected from very frequent, common and rare query bin sets of highly-used search engines \n\t\\item To be able to calculate measures like recall, we need to find all relevant documents in the collection. Can be done either deterministically or stochastically\n\t\\begin{itemize}\n\t\t\\item \\textbf{Depth-k pooling} - deterministic, standard method. Apply $M$ IR systems and take the union of the $k$ top results of all $M$ systems. This set of documents is labeled by humans, and all others are considered as not relevant. Note that we need the $M$ systems to be different/take another perspective on the data so that they don't find all the same documents. Otherwise, future IR algorithms can find relevant documents that the others haven't found yet and will be punished for that. $k$ is task specific, but a value of $100$ has shown to be sufficient\n\t\t\\item \\textbf{Random Sampling} - stochastic method. Simplest approach is for a query $q$, just sample a small set of documents out of the whole corpus and label those. Otherwise are considered as unlabeled, thus neglected in evaluation. Problem: significant sparsity of relevant documents in the corpus. \n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Challenges of offline evaluation}\n\\label{sec:offline_eval_problems}\n\\begin{itemize}\n\t\\item Expensive and slow to collect new data\n\t\\item Ambiguous queries are particularly hard to judge realistically (what intent is most popular?). Particularly hard for personalized searches\n\t\\item Judges need to correctly appreciate uncertainty/allow different intents\n\t\\item How to identify when relevance changes (temporal, query intent changes, ...)?\n\\end{itemize}\n\\subsection{Comparative evaluation}\n\\begin{itemize}\n\t\\item How do we compare different retrieval systems? Is the difference only due to random noise? $\\implies$ Statistical significance test\n\t\\item Two hypotheses where we want to prove that $H_0$ is wrong: $$H_0: \\text{MAP}_E - \\text{MAP}_P = 0, \\hspace{4mm} H_1: \\underbrace{\\text{MAP}_E - \\text{MAP}_P \\neq 0}_{\\text{two-sided}} \\text{\\hspace{2mm}or\\hspace{2mm}}\\underbrace{\\text{MAP}_E - \\text{MAP}_P > 0}_{\\text{one-sided}}$$\n\t\\item Compute the $p$-value that describes the probability of observing the test data given that $H_0$ is valid (low $p$-value disprove null hypothesis)\n\\end{itemize}\n\\subsubsection{Student's t-test}\n\\begin{itemize}\n\t\\item Statistic: $$t=\\frac{\\mu_{E-P}}{\\frac{\\sigma_{E-P}}{\\sqrt{N}}} = \\frac{\\overline{AP_E - AP_P}}{\\frac{\\sigma_{E-P}}{\\sqrt{N}}}$$\n\t\\item We assume that the mean measure follow a normal distribution (see Figure~\\ref{img:hypothesis_testing_t_test_t_dist})\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.3\\textwidth]{figures/hypothesis_testing_t_test_t_dist.png}\n\t\t\\caption{Distribution of $t$ values under the null hypothesis}\n\t\t\\label{img:hypothesis_testing_t_test_t_dist}\n\t\\end{figure}\n\t\\item The $p$-value is determined by the area under the distribution right from the determined value\n\t\\item If the $p$-value is lower than the significance level $\\alpha$, reject null hypothesis\n\t\\item There are two different error types for the t-test: \n\t\\begin{description}\n\t\t\\item[Type 1] rejecting null hypothesis although it was true (prob. is $\\alpha$)\n\t\t\\item[Type 2] not rejecting the null hypothesis although it was false (prob. is $\\beta$)\n\t\\end{description}\n\t\\item There are four aspects of the test that interact with each other. If one is unknown, it can be derived from the others\n\t\\begin{enumerate}\n\t\t\\item \\textit{Sample size} $N$\n\t\t\\item \\textit{Effect size} = diff. of means / std. dev.\n\t\t\\item \\textit{Significance level} = Type 1 error $\\alpha$\n\t\t\\item \\textit{Power} = 1 - Type 2 error $\\beta$. Prob. of finding an effect if it is there. \n\t\\end{enumerate}\n\\end{itemize}\n\\subsubsection{Sign test}\n\\begin{itemize}\n\t\\item Look at score/sample pairs from $A$ and $B$ and consider the null hypothesis $H_0: P(B>A)=P(A>B)=1/2$.\n\t\\item It is a discrete way of looking at the t-test. For a sample size of $N$, we get a binomial distribution with $N$ bins. The bins summed up from the measured number of $B$ winning over $A$ describe the p-value (see Figure~\\ref{img:hypothesis_testing_sign_test})\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/hypothesis_testing_sign_test.png}\n\t\t\\caption{Distribution of $t$ values under the null hypothesis}\n\t\t\\label{img:hypothesis_testing_sign_test}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{Distribution-free tests}\n\\begin{itemize}\n\t\\item Tests where we do not explicitly assume the underlying data to be sampled from a specific distribution\n\t\\item \\textbf{Randomization test}\n\t\\begin{itemize}\n\t\t\\item Given: a set of results for $N$ queries for algorithm $A$ and $B$\n\t\t\\item Repeat for many times:\n\t\t\\begin{itemize}\n\t\t\t\\item Randomly swap values for a query in algorithm $A$ and $B$\n\t\t\t\\item Compute average of both systems and their difference\n\t\t\t\\item Add difference to an array\n\t\t\\end{itemize} \n\t\t\\item The two systems are significantly different, if the actual difference without swapping is outside 95\\% of the differences in the array.\n\t\\end{itemize}\n\t\\item \\textbf{Bootstrap test}\n\t\\begin{itemize}\n\t\t\\item Same preparation as for randomization test\n\t\t\\item Repeat for many times:\n\t\t\\begin{itemize}\n\t\t\t\\item Randomly sample pair of scores (i.e. selecting queries) of $A$ and $B$ with replacement\n\t\t\t\\item Compute average of each systems in the set of pairs\n\t\t\t\\item Add difference to an array\n\t\t\\end{itemize}\n\t\t\\item The two systems are significantly different if the mean of the array can be shown to be significantly different from 0.\n\t\\end{itemize}\n\t\n\\end{itemize}"
  },
  {
    "path": "Information_Retrieval_1/ir_online_evaluation.tex",
    "content": "\\section{Online evaluation}\n\\begin{itemize}\n\t\\item In online evaluation, the system interacts with the user $\\implies$ user \"tells\" what is relevant, system analyzes the user's behavior for gaining that knowledge\n\t\\item The benefit of online evaluations is that they are mostly simpler and directly incorporate measuring the ranking quality\n\t\\item However, the downsides are that the results are worse to explain/interpret (why did users click less, different queries might rely on different metrics, ...). Also, evaluations might not be comparable over time so that we also need to ensure the same conditions/user population for both systems.\n\\end{itemize}\n\\subsection{Analyzing user behavior}\n\\begin{itemize}\n\t\\item A user provides various signals from which we can try to retrieve his \"happiness\" about the results. The following ones are mostly used:\n\t\\item \\textit{Clicks} - clicks are mostly noisy so that a click doesn't ensure that the document was actually relevant. Clicks have several biases:\n\t\\begin{itemize}\n\t\t\\item \\textit{Position bias} - a user tends towards clicking higher ranked results\n\t\t\\item \\textit{Contextual bias} - nearby results effect the click probability of a document\n\t\t\\item \\textit{Attention bias} - some results draw more attention to themselves by the usage of images, font size, ...\n\t\\end{itemize}\n\t\\item \\textbf{Time} - the time a user spends on a certain query before coming back to search engine\n\t\\begin{itemize}\n\t\t\\item \\textit{Dwell time} - time spent on a clicked page. If duration is more than 30 seconds, we assume that click satisfied\n\t\t\\item \\textit{Exit type} - how the user exists the page (closing browser, continue scrolling through results, putting in new query, ...) \n\t\\end{itemize}\n\t\\item \\textit{Mouse movement} - time on website is not sufficient. Mouse movement can indicate whether user is actually reading or only scrolling/scanning\n\t\\item \\textit{Reformulations} - if new query is entered, check for similarity with the previous one. Reformulated/Similar queries that were entered quickly after the first one, indicate that user was not satisfied with previous results.\n\\end{itemize}\n\\subsection{A/B Testing}\n\\begin{itemize}\n\t\\item When testing two systems in an online experiment, we need to make sure that both have the same preconditions so that the system improvements clearly correlate with the new click/sale numbers\n\t\\item In A/B Testing, users are split into two groups where each group is assigned to one of the algorithms. We analyze the users' behavior on both systems and calculate a metric based on that. Comparing the results for both systems on significance leads to a final decision.\n\t\\item \\textit{Challenges} in A/B Testing\n\t\\begin{itemize}\n\t\t\\item If one system is very different and probably bad, it will affect the \\textit{user experience} and damages website image $\\implies$ perform offline evaluation in advance to avoid testing a very bad system\n\t\t\\item It is hard to define \\textit{metrics} as they can contradict each other. For example, if we report number of clicks and sessions per users, a click increase can indicate better/more relevant results. However, if another systems provides snippets that already contain the information, the user will click less.\n\t\t\\item The metric should be as sensitive as possible. \\textit{Sensitivity} is the ability of the metric to detect the statistically significant difference when the treatment effect exists $\\implies$ how many queries/days/users/... for significance needed?\n\t\\end{itemize}\n\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/online_eval_AB_testing.png}\n\t\t\\caption{Visualization of A/B Testing}\n\t\t\\label{img:online_eval_AB_testing}\n\t\\end{figure}\n\\end{itemize}\n\\subsection{Interleaving}\n\\begin{itemize}\n\t\\item A/B testing introduces a high variance by letting different users evaluate different systems $\\implies$ Show interleaved results from both algorithms A and B without telling the user which document is from which model\n\t\\item The evaluation is based on the clicks of a user where the algorithm gets the credit that provided the clicked document\n\\end{itemize}\n\\subsubsection{Balanced interleaving}\n\\begin{itemize}\n\t\\item In balanced interleaving, we select randomly which algorithm starts (A or B). If A would start, we take the first document of A and place it in our interleaved ranking list. Then we pick the first document of B and continue with A again\n\t\\item If a document is already in the interleaved ranking, we skip this document and continue with picking the next document from the \\textit{other} ranking model \n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/online_eval_balanced_interleaving.png}\n\t\t\\caption{Formal algorithm describing balanced interleaving}\n\t\t\\label{img:online_eval_balanced_interleaving}\n\t\\end{figure}\n\t\\item Problem: balanced interleaving brakes under corner cases. Assume following ranking:\n\t$$A: \\left\\{d_1, d_2, d_3, d_4\\right\\}, B: \\left\\{d_2, d_3, d_4, d_1\\right\\}$$\n\tNo matter whether we start at model A or B, the interleaved list contains three documents assigned to B and only one to A. Thus, random clicking would lead to B winning $\\implies$ bias. Resolved by team-draft interleaving\n\\end{itemize}\n\\subsubsection{Team-draft interleaving}\n\\begin{itemize}\n\t\\item In team-draft interleaving we guarantee that both algorithms contribute equally to the interleaved ranking\n\t\\item At each stage, we flip a coin to determine whether to pick the next document from A or B first. Afterwards, the document of the other system is picked\n\t\\item If a document is already in the interleaved ranking, we look for the next document from the \\textit{same} ranking model until we find a new document.\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/online_eval_team_draft_interleaving.png}\n\t\t\\caption{Formal algorithm describing Team-draft interleaving}\n\t\t\\label{img:online_eval_team_draft_interleaving}\n\t\\end{figure}\n\t\\item There are also corner cases that can cause troubles in team-draft interleaving. However, in practice this rarely happens/has a significant effect. \n\\end{itemize}\n\\subsubsection{Probabilistic interleaving}\n\\begin{itemize}\n\t\\item To avoid biases completely, we can apply probabilistic models\n\t\\item Convert the ranking of each model to a probability distribution by applying softmax ($\\tau = 3$):\n\t$$p_i(d) = \\frac{\\frac{1}{r_i(d)^{\\tau}}}{\\sum_{d' \\in D} \\frac{1}{r_i(d')^{\\tau}}}$$\n\t\\item For every position in the interleaved ranking, flip a coin to determine whether to pick a document from A or B. Next, we sample from the corresponding softmax distribution a document without replacement, and add it to the interleaved list. The picked document is removed from the probability distributions of A and B. \n\t\\item We can perform evaluation by counting the clicks for documents sampled from A and B. We expect the same number of clicks for documents at the same position of both algorithms due to the same probability in the softmax. % both A and B based on the softmax, and compare which one has the higher probability sum of all clicked documents. Thus, for documents at the same rank, we expect a tie.\n\t\\item Note that compared to a hard assignment of 0 or 1 in balanced and team draft interleaving, the distribution of credit accumulated for clicks is smoothed based on the relative rank of the document in the original result lists (a click on any document leads to a non-zero credit for both rankings)\n\t\\item The algorithm is summarized in Figure~\\ref{img:online_eval_probabilistic_interleaving}\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/online_eval_probabilistic_interleaving.png}\n\t\t\\caption{Formal algorithm describing probabilistic interleaving}\n\t\t\\label{img:online_eval_probabilistic_interleaving}\n\t\\end{figure}\n\t\\item Another, more efficient evaluation method is by marginalizing over all possible assignments $a$. Therefore, we calculate the probability of the interleaved list $l$ given $a$ (and the query $q$) by successively multiplying the softmax probabilities at that point. For example, the first assignment $a=\\left\\{1,1,1,1\\right\\}$ leads to the following calculation:\n\t$$p(l_i|a=\\left\\{1,1,1,1\\right\\},q) = 0.85 \\cdot \\frac{0.1}{0.15} \\cdot \\frac{0.03}{0.05} \\cdot \\frac{0.02}{0.02} = 0.34$$\n\t\\item Normalizing all $p(l_i|a,q)$ by its sum lead to $p(a|l_i,q)$ $\\implies$ $p(a|l_i,q) = \\frac{p(l_i|a,q)}{\\sum_{a \\in A} p(l_i|a,q)}$\n\t\\item For every assignment, we add the value $o=-1$ times the probability $p(a|l_i,q)$ if more clicked documents were assigned to $A$. If $B$ has more clicks, we use $o=1$ as factor, and ignore it for a tie (or multiply by $o=0$). Thus, our expected number of wins $B$ has more than $A$ is given by:\n\t$$E[O] = \\sum_{a \\in A} o_a \\cdot p(a|l_i,q) \\text{\\hspace{2mm}where\\hspace{2mm}} o_a = \\begin{cases}\n\t -1 & \\text{if } c_A > c_B\\\\\n\t 0 & \\text{if } c_A = c_B\\\\\n\t 1 & \\text{if } c_A < c_B\n\t\\end{cases}$$\n\t\\item Figure~\\ref{img:online_eval_probabilistic_interleaving_2} visualizes an example for probabilistic interleaving\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=\\textwidth]{figures/online_eval_probabilistic_interleaving_2.png}\n\t\t\\caption{Visualization of probabilistic interleaving}\n\t\t\\label{img:online_eval_probabilistic_interleaving_2}\n\t\\end{figure}\n\\end{itemize}"
  },
  {
    "path": "Information_Retrieval_1/ir_semantic_matching.tex",
    "content": "\\section{Semantic matching}\n\\begin{itemize}\n\t\\item \\textit{Vocabulary gap}: query and document might use different lexical representation for the same entity $\\implies$ resolve by semantic matching\n\t\\item Represent query and document by their meaning, not lexical/word level\n\t\\item This will help to identify synonyms and/or semantic relatedness for computing similarity\n\t\\item To get such a representation, one idea is to apply dimensionality reduction. This relies on the assumption that the dimension of the data is actually lower than i.e. vocabulary size\n\t\\item Similar data in terms of semantic will be (hopefully) similar in reduced dimensions as well\n\\end{itemize}\n\\subsection{Latent Semantic Indexing}\n\\begin{itemize}\n\t\\item We represent all documents and queries in a term-document matrix:\n\t$$X = \\left[\\begin{array}{cccc}\n\t| & | &  & | \\\\\n\tx_1 & x_2 & \\dots & x_m \\\\\n\t| & | &  & | \n\t\\end{array}\\right]$$\n\twhere $m$ rows represent the documents, and $n$ rows the terms. A single cell in the matrix $X$ specifies the term frequency $\\text{tf}(w;d)$. Note that we would also add the queries in $X$ as documents.\n\t\\item On this matrix, we apply Singular Value Decomposition (SVD) so that $X_{n\\times m} = U_{n \\times r} \\Sigma_{r\\times r} V_{m\\times r}^T$\n\t\\begin{itemize}\n\t\t\\item $\\bm{U}_{n \\times r}$ represents the word/term embedding along rows (one word per row). The columns are the \"semantic\" dimensions that e.g. represent topics/hidden lower-dimensional space. \n\t\t\\item $\\bm{V}_{m \\times r}$ similarly represents the embedding of documents (one doc per row, but is transposed in calculation). \n\t\t\\item $\\bm{\\Sigma}_{r \\times r}$ is a square, diagonal matrix. The magnitude of a singular value represents the importance of the corresponding latent dimension in the collection/data. It is always sorted from the highest value in first place to lowest value in last place.\n\t\\end{itemize}\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/semantic_matching_SVD_example.png}\n\t\t\\caption{Example of SVD for 6 documents and 5 terms}\n\t\t\\label{img:semantic_matching_SVD_example}\n\t\\end{figure}\n\t\\item To reduce dimensions to $k$, we simply drop those with the lowest values in $\\Sigma$. These dimensions may be noise and make things dissimilar when they actually are on topic level. $k$ is hyperparameter.\n\t\\item In case of Figure~\\ref{img:semantic_matching_SVD_example} with $k=2$, we would drop the last three dimensions. This leads to our new embeddings $X'$ for the documents. The resulting matrices would look like as in Figure~\\ref{img:semantic_matching_SVD_example_2}.\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/semantic_matching_SVD_example_2.png}\n\t\t\\caption{Reduced dimensions by $k=2$ for 6 documents and 5 terms}\n\t\t\\label{img:semantic_matching_SVD_example_2}\n\t\\end{figure}\n\t\\item The similarity between documents/queries can be calculated by cosine similarity (dot product) between their new embeddings in $X'$. Furthermore, we can also compute the similarity between terms by using the rows.\n\t\\item Choice of $k$:\n\t\\begin{itemize}\n\t\t\\item The choice of $k$ is critical in IR. The ideal value of $k$ would be large enough to fit all the real structure in the data, but small enough to compress/group terms together that are very similar (less noise).\n\t\t\\item Typically, different values of $k$ are tested and compared by their performance. For example, a high precision but low recall suggests a poor generalization of the model. Therefore, we should decrease $k$ in this case.\n\t\\end{itemize}\n\t\\item LSI addresses synonymy by mapping similar words in the same dimensions. The cost of such a mapping is lower than for unrelated words as they occur similar/same documents. \n\t\\item \\textbf{Strengths} of LSI\n\t\\begin{itemize}\n\t\t\\item Using $X'$ instead of $X$ show performance increase as we filter out the noise\n\t\t\\item $X'$ represents the best approximation of $X$ with a matrix of rank $k$: $X' = \\argmin\\limits_{X':\\text{rank}(X')=k} ||X-X'||$\n\t\t\\item Is mostly combined with lexical methods like BM25 to not lose \"obvious\" matches\n\t\\end{itemize}\n\t\\item \\textbf{Weaknesses} of LSI\n\t\\begin{itemize}\n\t\t\\item A huge storage is required as the matrices $U$ and $V$ are dense (less zeros)\n\t\t\\item Representations are not interpretable, and it is not guaranteed that hidden dimensions represent topics\n\t\t\\item $k$ is often not easy to determine and requires multiple tests\n\t\t\\item SVD assumes orthogonal dimensions on which the variance is maximum which is not always the case\n\t\t\\item The model is not generative or probabilistic, which makes it hard to extend collection by new documents/queries (worst case: redo whole SVD)\n\t\\end{itemize}\n\t\\item One alternative is Non-negative Matrix Factorization which leads to smaller, positive matrices (but doesn't solve the other problems)\n\\end{itemize}\n\\subsection{Probabilistic Latent Semantic Indexing}\n\\begin{itemize}\n\t\\item (Pseudo-)generative model with which we try to detect key topics in the collection in an unsupervised fashion. The model describes how we would generate docs for certain topics\n\t\\item Every topic has its own language model/distribution over words in vocabulary in a unigram/bag-of-word style: $p(w|z=1), p(w|z=2),...$ where $z$ is the variable representing the topics. \n\t\\item A document is represented by the distribution of these topics in the document. Generating a document would require to first sample a topic based on the topic distribution of the document, and then generating a word based on the language model of the topic. The order of the words is not taken into account. An example is shown in Figure~\\ref{img:semantic_matching_PLSI_example}.\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/semantic_matching_PLSI_example.png}\n\t\t\\caption{Example of document representation in PLSI}\n\t\t\\label{img:semantic_matching_PLSI_example}\n\t\\end{figure}\n\t\\item For semantic matching, we calculate the cosine similarity between the topic distributions of the query and the document.\n\t\\item PSLA can have problems with stop words (common words that occur in every document frequently like \"the\" or \"and\"). To prevent that, we summarize all stop words in a separate background topic model which is equally shared by all docs. Before generating a word, we toss a biased coin to decide whether to retrieve a word from the background or standard topic model. Note that in matching, these stop words are masked out by that\n\\end{itemize}\n\\subsubsection{Retrieving distributions}\n\\begin{itemize}\n\t\\item Input: collection of $N$ documents, number of topics $K$\n\t\\item Output:\n\t\\begin{itemize}\n\t\t\\item Distributions over words $\\phi_{(z,w)} = p(w|z)$ for $z\\in \\left\\{1,...,K\\right\\}$ with $\\sum_{w\\in V} p(w|z) = 1$\n\t\t\\item Distributions over topics in all documents: $\\theta_{d,z} = p(z|d)$ for every $d$, with $\\sum_{z=1}^{K} p(z|d) = 1$\n\t\\end{itemize} \n\t\\item We try to solve problem by MLE. The probability of $w$ appearing at position $i$ in the document $d$ is:\n\t$$p(d_i = w | \\Phi, \\theta_d) = \\sum\\limits_{z=1}^{K} \\phi_{(z,w)} \\theta_{d,z}$$\n\tThe joint likelihood of the entire dataset is:\n\t$$p(W|\\Phi, \\Theta) = \\prod\\limits_{d\\in D}\\prod\\limits_{w\\in V}\\left(\\sum\\limits_{z=1}^{K} \\phi_{(z,w)} \\theta_{d,z}\\right)^{\\text{tf}(w;d)} $$\n\t\\item Taking the two constraints into account, we get an optimization problem which we can solve by the EM algorithm. For that, we assume that we know from which topic a word was generated at position $i$ in the document by $R_{d_i}$:\n\t$$p(W|R,\\Phi,\\Theta) = \\prod\\limits_{d\\in D}\\prod\\limits_{i=1}^{N_i}\\sum\\limits_{z=1}^{K} R_{(d_i,z)} \\left(\\phi_{(z,w)} \\theta_{d,z}\\right)$$\n\t\\item For the EM algorithm, we would update $R_{d_i}$ during the expectation step by $R_{d_i} = \\frac{\\phi_{(z,w_i)\\theta_{(d,z)}}}{\\sum_{z=1}^{K} \\phi_{(z,w_i)\\theta_{(d,z)}}}$. The maximization step consists of updating $\\Theta$ and $\\Phi$: $\\theta_{(d,z)} = \\frac{\\sum_{d_i} R_{(d_i,z)}}{|d|}$, $\\phi_{(z,w)} = \\frac{\\sum_{d \\in D} n(d,w) R_{(w,z)}}{\\sum_{w' \\in V} \\sum_{d \\in D} n(d,w')R_{(w',z)}}$\n\t\\item PLSA is able to learn topics with their corresponding word distributions. However, there are still some drawbacks:\n\t\\begin{itemize}\n\t\t\\item It is still not a fully generative model. After running PSLA, we have topic distribution for documents we initially had, but we cannot extend it to new documents (or only hardly with heuristics)\n\t\t\\item Prone to overfitting\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Latent Dirichlet Allocation (LDA)}\n\\begin{itemize}\n\t\\item Takes PLSA but makes it a generative model with Dirichlet prior. Can also be seen as a Bayesian treatment of PLSA\n\t\\item Instead of probabilities for every document, we now simply define two hyperparameters $\\alpha$ and $\\beta$\n\t\\item For every topic $z=1,...,K$, we draw a word distribution $\\phi_z \\sim \\text{Dir}(\\beta)$. Thus, $\\beta$ determines how words are distributed per topic\n\t\\item For each document $d$, we sample a topic distribution $\\theta_d \\sim \\text{Dir}(\\alpha)$. The alpha therefore controls the mixture of topics for any given document.\n\t\\item The words in the documents are then sampled by the probabilities $\\phi_z$ and $\\theta_d$. Note that this is a fully generative model as we can generate as many new documents as we need.\n\\end{itemize}\n\\subsubsection{Graphical Models and Probabilistic Topic Models}\n\\begin{itemize}\n\t\\item We can represent every probabilistic topic model as graphical model which abstracts the conditional independence relationships\n\t\\item For example, Figure~\\ref{img:semantic_matching_graphical_models_LDA} visualizes the graphical model of the Latent Dirichlet Allocation\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/semantic_matching_graphical_models_LDA.png}\n\t\t\\caption{Graphical model of LDA}\n\t\t\\label{img:semantic_matching_graphical_models_LDA}\n\t\\end{figure}\n\t\\item In this diagrams, it is easier to show extensions. For instance, Figure~\\ref{img:semantic_matching_graphical_models_author_model} visualizes the Author-Topic model where we add an observed variable of the author of a document. This observation affects the probability distributions over words for each topic in a document \n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/semantic_matching_graphical_models_author_model.png}\n\t\t\\caption{Graphical model of the Author-topic model}\n\t\t\\label{img:semantic_matching_graphical_models_author_model}\n\t\\end{figure}\n\t% semantic_matching_graphical_models_author_model.png\n\\end{itemize}\n\\subsection{Probabilistic Topic Models in IR}\n\\begin{enumerate}\n\t\\item \\textbf{Topic matching}\n\t\\begin{itemize}\n\t\t\\item Represent document by $p(z|d)$ and query by $p(z|q)$\n\t\t\\item Take the KL divergence or similar measure to find score/similarity between the distributions\n\t\t\\item But: query is very short so that topic model is harder to infer without having too much noise\n\t\\end{itemize}\n\t\\item \\textbf{Smoothing}\n\t\\begin{itemize}\n\t\t\\item Smooth probabilities according to the topics in the document:\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\tp(w|d) & = \\lambda p_{\\mu}(w|d) + (1 - \\lambda) p_{\\text{tm}}(w|d)\\\\\n\t\t\t\t& = \\lambda p_{\\mu}(w|d) + (1 - \\lambda) \\left(\\sum\\limits_{z=1}^{K} p(w|z) p(z|d)\\right)\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\t\\item Thus we apply Jelinek-Mercer smoothing where the context is replaced by the topic word distributions of the document\n\t\\end{itemize}\n\t\\item \\textbf{Query expansion}\n\t\\begin{itemize}\n\t\t\\item We can build an own language model for a given query by using the word distributions of the topics:\n\t\t$$p_{\\text{tm}}(w|q) = \\sum\\limits_{z=1}^{K} p(w|z) p(z|q)$$\n\t\\end{itemize}\n\\end{enumerate}"
  },
  {
    "path": "Information_Retrieval_1/ir_summary.tex",
    "content": "\\documentclass[a4paper]{article} \n\\addtolength{\\hoffset}{-2.25cm}\n\\addtolength{\\textwidth}{4.5cm}\n\\addtolength{\\voffset}{-3.25cm}\n\\addtolength{\\textheight}{5cm}\n\\setlength{\\parskip}{0pt}\n\\setlength{\\parindent}{0in}\n\n\\usepackage{blindtext} % Package to generate dummy text\n\\usepackage{charter} % Use the Charter font\n\\usepackage[utf8]{inputenc} % Use UTF-8 encoding\n\\usepackage{microtype} % Slightly tweak font spacing for aesthetics\n\\usepackage[english]{babel} % Language hyphenation and typographical rules\n\\usepackage{amsthm, amsmath, amssymb} % Mathematical typesetting\n\\usepackage{float} % Improved interface for floating objects\n\\usepackage[final, colorlinks = true, \nlinkcolor = black, \ncitecolor = black]{hyperref} % For hyperlinks in the PDF\n\\usepackage{graphicx, multicol} % Enhanced support for graphics\n\\usepackage{xcolor} % Driver-independent color extensions\n\\usepackage{marvosym, wasysym} % More symbols\n\\usepackage{rotating} % Rotation tools\n\\usepackage{subcaption}\n\\usepackage{censor} % Facilities for controlling restricted text\n\\newcommand{\\note}[1]{\\marginpar{\\scriptsize \\textcolor{red}{#1}}} % Enables comments in red on margin\n\\usepackage{bm}\n\\usepackage{blkarray}\n\\usepackage{enumitem}\n\\usepackage{pgfplots}\n\\definecolor{colkeyword}{rgb}{0,0.4,0}\n\\definecolor{colname}{rgb}{0.4,0.4,0}\n\\definecolor{coltype}{rgb}{0.4,0,0.4}\n\\definecolor{coloperators}{rgb}{0,0,1.0}\n\\definecolor{colscopes}{rgb}{0.4,0,0}\n\\usepackage{mathtools}\n\\DeclareMathOperator*{\\argmin}{arg\\,min}\n\n\\setcounter{tocdepth}{2}\n% Title Page\n\\title{Summary Information Retrieval 1}\n\\author{Phillip Lippe}\n\n\n\\begin{document}\n\\maketitle\n\\tableofcontents\n\\newpage\n\n% \\input{ir_boolean_retrieval.tex}\n\\input{ir_offline_evaluation.tex}\n\\input{ir_online_evaluation.tex}\n\\input{ir_click_models.tex}\n\\input{ir_language_models.tex}\n\\input{ir_semantic_matching.tex}\n\\input{ir_neural_models.tex}\n\\input{ir_learning_to_rank.tex}\n\\input{ir_counterfactual_eval.tex}\n\\appendix\n% \\newpage\n% \\input{nlp_appendix.tex}\n\\end{document}"
  },
  {
    "path": "Knowledge_Representation/kr_csp.tex",
    "content": "\\section{Constraint Satisfaction Problems}\n\\begin{itemize}\n\t\\item Knowledge Representation is focused on qualitative reasoning, not really quantitative\n\t\\begin{itemize}\n\t\t\\item Abstract description of the world is mostly easier for human to reason than an exact numerical definition \n\t\t\\item We can for example reason about time points and intervals in our system where we have relations between those\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Fundamentals of CSPs}\n\\begin{itemize}\n\t\\item To represent qualitative knowledge, we again have to define a:\n\t\\begin{itemize}\n\t\t\\item \\textit{Vocabulary}: finite set of relations, mostly binary. \n\t\t\n\t\tExample: $x$ equals $y$ $\\Rightarrow$ $x=y$, $x$ before $y$ $\\Rightarrow$  $x<y$, $x$ after $y$ $\\Rightarrow$  $x>y$\n\t\t\\item \\textit{Language}: sets of atomic formulae, perhaps restricted disjunction\n\t\t\n\t\tExample: define disjunction by $x\\left\\{<,=\\right\\}y$ (in maths $x\\leq y$), and formulae to describe configurations: $\\left\\{x\\left\\{<,=\\right\\}y, y\\left\\{z\\right\\}\\right\\}$\n\t\t\\item \\textit{Formal semantics}: interpretation of function and symbols\n\t\t\n\t\tExample: interpret time points and relations over rational (or real) numbers\n\t\\end{itemize}\n\t\\item On those, we can perform various reasoning tasks\n\t\\begin{itemize}\n\t\t\\item \\textit{Satisfiability}: Is this a consistent set of constraints? $\\Rightarrow$ find satisfying instantiation of all variables\n\t\t\\item \\textit{Deduction}: Does $x\\left\\{=\\right\\}y$ logically follow from the configuration?\n\t\t\\item \\textit{Minimal description}: What are the most constrained relations that describe the same set? \n\t\t\\item \\textit{Solving}: find one or all (optimal) solutions to the CSP\n\t\\end{itemize}\n\t\\item Formally, a Constraint Satisfaction Problem consists of:\n\t\\begin{itemize}\n\t\t\\item Variables $Y:=y_1, y_2, ..., y_k$\n\t\t\\item Domains $D_1, ..., D_k$ to which the variables belong ($y_i$ represents a possible value of $D_i$: $y_i\\in D_i$)\n\t\t\\item Constraints $C\\in \\mathcal{C}$ on $Y$ which define a subset of $D_1 \\times D_2 \\times ... \\times D_k$ \n\t\\end{itemize}\n\t\\item Given a CSP $\\left\\{\\mathcal{C}; y_1\\in D_1, ..., y_k\\in D_k \\right\\}$, $\\left(d_1, ..., d_k\\right)\\in D_1\\times ...\\times D_k$ is a solution iff for all $C\\in \\mathcal{C}: \\left(d_1, ..., d_k\\right)\\in C$\n\t\\item Most CSPs have only binary constraints (constraints over two variables) which can be modeled by a constraint graph (nodes are variables, edges/arcs are constraints)\n\\end{itemize}\n\\subsubsection{Backtrack search}\n\\begin{itemize}\n\t\\item Finding a solution for CSPs is a more general version of DPLL (PL is actually a special case CSP)\n\t\\item In the initial state, we have an empty set of assignments. We then assign a value to an unassigned variable that does not conflict with the current assignment. Test if CSP is fulfilled or unsatisfied for this assignment. If neither of both, choose the next variable etc.\n\t\\item The depth of the search tree is the number of variables, and the number of children/splits in the tree are given by the domain size. In the worst case, we have a search space of $|D|^{\\# vars}$\n\t\\item We can improve backtracking efficiency by:\n\t\\begin{enumerate}\n\t\t\\item \\textbf{Smart variable picking}: various heuristics for choosing the next variable\n\t\t\\begin{itemize}\n\t\t\t\\item \\textit{Most constrained variable}: choose the variable with the fewest legal values (therefore also called \\textit{Minimum remaining values} (MRV))\n\t\t\t\\item \\textit{Most constraining variable}: the variable with the most constraints on. Is mostly used as tie-breaker strategy for MRV\n\t\t\\end{itemize}\n\t\t\\item \\textbf{Smart value picking}: what possible value of the domain to pick for the variable\n\t\t\\begin{itemize}\n\t\t\t\\item \\textit{Least constraining value}: given a variable, choose the value that constraints the least, i.e. the one that rules out the fewest values in the remaining values  \n\t\t\\end{itemize}\n\t\t\\item \\textbf{Spotting failure early} to backtrack early: finding unsatisfiable constraints in a branch\n\t\t\\begin{itemize}\n\t\t\t\\item \\textit{Forward checking}: by keeping track of remaining legal values for unassigned variables, we can stop the search in this branch if one variable has an empty set/no legal values to be assigned. This is done by propagating information from assigned to unassigned variables.\n\t\t\t\\item \\textit{Arc consistency}: simplest form of constraint propagation (see Section~\\ref{sec:constraint_propagation}) that makes each arc/edge consistent. Hence, $x\\to y$ is consistent iff for every value $x$ in $D_x$ there exists a value $y$ in $D_y$ for which the constraint is satisfied (if not, remove contradicting values of $D_x$). Note that if $x$ has been changed, we have to recheck all its neighbors. Arc consistency can detect failures earlier than forward checking\n\t\t\t\\item More complex/sophisticated methods are discussed under constraint propagation in Section~\\ref{sec:constraint_propagation}\n\t\t\\end{itemize}\n\t\\end{enumerate}   \n\\end{itemize}\n\\subsection{Constraint propagation}\n\\label{sec:constraint_propagation}\n\\begin{itemize}\n\t\\item A constraint itself can restrict the search space without splitting\n\t\\item Constraint propagation is similar to unit propagation and subsumes resolution. It therefore checks the CSP for local consistency (a CSP is locally consistent if we can extend it by another assignment while being satisfied)\n\t\\item Example: the CSP $\\langle x < y ; x \\in \\left[50..200\\right], y \\in \\left[0..100\\right] \\rangle$ can be simplified to \n\t\n\t$\\langle x < y ; x \\in \\left[50..99\\right], y \\in \\left[51..100\\right] \\rangle$ without losing any possible solutions\n\t\\item This might be an iterative problem when multiple constraints interact with each other ($x$ has effect on $y$, $y$ on $z$, and so on)\n\t\\item Thus, we have to decide when to stop performing local consistency checking. There are various heuristics/methods:\n\t\\begin{itemize}\n\t\t\\item \\textit{(Directional) Arc consistency} is one of the most simplest approaches. Note that general arc consistency applies propagation to both sides $x\\to y$ and $y\\to x$, while we can also limit it by a directed approach (only $x\\to y$ or $y\\to x$). Problem: only looks at direct constraints and can for example not spot that the following CSP is inconsistent:\n\t\t\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.3\\textwidth]{figures/kr_csp_arc_const_example.pdf}\n\t\t\\end{figure}\n\t\t\\item \\textit{Path consistency}: extends arc consistency to picking two variables. Formally, for all $i$ and $j$:\n\t\t$$\\forall x,y: (x,y)\\in P_{i,j}, x\\in P_i, y\\in P_j \\to \\left(\\exists z: z \\in P_k \\wedge (x,z) \\in P_{i,k} \\wedge (z,y) \\in P_{k,j}\\right)$$\n\t\ti.e., any consistent assignment to two variables can be extended to a third one. Can be enforced by iterating the following assignment until nothing changes anymore:\n\t\t$$P_{i,j} := P_{i,j} \\cap \\left\\{(x,y)\\hspace{1mm}|\\hspace{1mm}\\exists z: (x,z) \\in P_{i,k} \\wedge (z,y) \\in P_{k,j}\\hspace{2mm}\\text{for all } k\\right\\}$$\n\t\tPath consistency subsumes arc consistency and detects inconsistency in the previous example. Note that path consistency removes pairs of values, and thus makes constraints explicitly\n\t\t\\item \\textit{$k$-Consistency}: generalization of arc and path consistency to arbitrary $k$. For each satisfying value assignment to $k-1$ variables, there exists an extension of this assignment to a $k$-th variable such that this extended assignment satisfies all constraints among these $k$ variables.\n\t\t\\item \\textit{Strong $k$-consistency}: A CSP is strongly $k$-consistent iff it is $j$-consistent for all $j\\leq k$. If a CSP of size $n$ $n$-consistent is, then we can construct a model in polynomial time. But note that checking this already solves the CSP and the computational costs exponentially grow with $k$.\n\t\t\\item \\textcolor{red}{Question: why does PC subsumes AC, but not $k$-consistency $k-1$-consistency?}\n\t\t\\item \\textit{Hyper-arc consistency}: extension of arc consistency to more than binary constraints (same for those). Defined by: for every constraint $C$ and every variable $x$ with domain $D_x$, each value for $x$ from $D_x$ participates in a solution to $C$.\n\t\\end{itemize}\n\\end{itemize}"
  },
  {
    "path": "Knowledge_Representation/kr_dl.tex",
    "content": "\\section{Description Logic}\n\\begin{itemize}\n\t\\item Description Logic is the logic for ontologies\n\t\\item Is more expressive than propositional logic, but less than first-order logic (DL is subsumed by FOL as classes are unary predicates and relations binary)\n\t\\item We limit our discussion to the $\\mathcal{ALC}$ description logic (\\textit{Attributive Concept Language with Complements})\n\\end{itemize}\n\\subsection{Ontologies}\n\\begin{itemize}\n\t\\item Organizing knowledge in a way that is useful for people\n\t\\item Fundamental elements of ontologies\n\t\\begin{itemize}\n\t\t\\item \\textit{Class}/\\textit{Type}/\\textit{Concept}: name + set of properties that describe certain set of individuals\n\t\t\\item \\textit{Instances}: members of the set defined by the class\n\t\t\\item \\textit{Property}/\\textit{Relation}: assert facts about the instances/relation between classes\n\t\\end{itemize}\n\t\\item The backbone of every ontology is a type-/class-hierarchy where multi-parent inheritance is possible\n\t\\item Axioms that need to be formalized in a logic for ontologies:\n\t\\begin{itemize}\n\t\t\\item Two classes are equivalent iff they have the same individuals and same definition\n\t\t\\item The intersection and union of classes\n\t\t\\item A class can be specified by either its definition or enumeration of all members\n\t\t\\item Restriction of some or all values from the specified class\n\t\\end{itemize}\n\t\\item Property types/possible properties of relations:\n\t\\begin{itemize}\n\t\t\\item \\textit{Symmetry}: if $a$ to $b$ by relation $r$, then $b$ to $a$ by $r$\n\t\t\\item \\textit{Asymmetry}: if $a$ to $b$ by relation $r$, then there can't be $b$ to $a$ by $r$\n\t\t\\item \\textit{Transitivity}: if $a$ to $b$ and $b$ to $c$ by relation $r$, then $a$ to $c$ by $r$\n\t\t\\item \\textit{Functionality}: if $a$ to $b$ and $a$ to $c$ by relation $r$, then $b$ and $c$ must be the same\n\t\t\\item \\textit{Inverse functionality}: if $a$ to $b$ and $c$ to $b$ by relation $r$, then $a$ and $c$ must be the same\n\t\t\\item \\textit{Reflexivity}: $a$ to $a$ by relation $r$ always holds\n\t\t\\item \\textit{Ir-reflexivity}: $a$ to $a$ by relation $r$ can never hold\n\t\t\\item \\textit{Inverse property}: if $a$ to $b$ by relation $r$, then $b$ to $a$ by relation $q$\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Fundamentals of Description Logic}\n\\begin{itemize}\n\t\\item A logic is defined by\n\t\\begin{itemize}\n\t\t\\item A language (``syntax'')\n\t\t\\item The meaning of expressions (``semantics'')\n\t\t\\item Inference (``deduction'')\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Syntax}\n\\begin{itemize}\n\t\\item The vocabulary of DL includes concept/class/type and role names\n\t\\item Furthermore, we have the universal concept $\\top$ (everything) and the bottom concept $\\bot$ (nothing)\n\t\\item More complex types can be set together by union $\\sqcup$, intersection $\\sqcap$ and complement $\\lnot$\n\t\\item Restrictions are encoded by $\\exists r.C$ and $\\forall r.C$. See semantics for meaning/explanation, and visualization in Figure~\\ref{fig:kr_dl_restriction_operator_visualized}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/kr_dl_restriction_operator_visualized.png}\n\t\t\\label{fig:kr_dl_restriction_operator_visualized}\n\t\t\\caption{Visualization of restriction operators $\\exists r.C$ and $\\forall r.C$}\n\t\\end{figure}\n\t\\item Examples:\n\t\\begin{itemize}\n\t\t\\item $\\texttt{Boys} \\sqcup \\texttt{Girls}$\n\t\t\\item $\\texttt{Girls} \\sqcap \\exists \\texttt{owns}.\\texttt{Car}$\n\t\\end{itemize}\n\t\\item Important for identifying concepts and roles in text. Example:\n\t\n\t``\\textit{Any artwork is created by an artist. A sculpture is an artwork. A painting is an artwork that is not a sculpture. A painter is someone who painted a painting. A sculptor is someone who sculptured an artwork and only create sculptures. If an artwork is created by an artist, he has either painted or sculptured it.}''\n\t\n\tThe solution would be the concepts $\\left\\{\\texttt{Artwork}, \\texttt{Artist}, \\texttt{Sculptor}, \\texttt{Painter}, \\texttt{Painting}, \\texttt{Sculpture}\\right\\}$, and the roles $\\left\\{\\texttt{created}, \\texttt{created\\_by}, \\texttt{painted}, \\texttt{sculptured}\\right\\}$\n\t\n\t``\\textit{An artwork that is not a sculpture}'': $\\texttt{Artwork}\\sqcap \\lnot\\texttt{Sculpture}$\n\t\n\t``\\textit{Some who painted a painting}'': $\\exists \\texttt{painted}.\\texttt{Painting}$\n\t\n\t``\\textit{Someone who sculptured an artwork and only created sculptures}'': $\\exists \\texttt{sculptured}.\\texttt{Artwork}\\sqcap \\forall \\texttt{created}.\\texttt{Sculpture}$\n\\end{itemize}\n\\subsubsection{Semantics}\n\\begin{itemize}\n\t\\item Mathematical meaning/interpretation of syntax\n\t\\item The interpretation function $\\mathcal{I}=\\left(\\Delta^{\\mathcal{I}}, \\cdot^{\\mathcal{I}} \\right)$ where $\\Delta^{\\mathcal{I}}$ is a non-empty domain of individuals, and $\\cdot^{\\mathcal{I}}$ is an interpretation function that maps\n\t\\begin{itemize}\n\t\t\\item $A^{\\mathcal{I}} \\subseteq \\Delta^{\\mathcal{I}}$, i.e. concepts to subsets of $\\Delta^{\\mathcal{I}}$\n\t\t\\item $r^{\\mathcal{I}} \\subseteq \\Delta^{\\mathcal{I}}\\times \\Delta^{\\mathcal{I}}$, i.e. role names to subsets of $\\Delta^{\\mathcal{I}}\\times\\Delta^{\\mathcal{I}}$\n\t\\end{itemize}\n\t% \\item Question: why has $\\Delta^{\\mathcal{I}}$ need to be non-empty? Is therefore a model without any individuals not allowed?\n\t\\item $\\Delta^{\\mathcal{I}}$ has to be not empty as otherwise no model exists. \n\t\\item Thus, a concept/class/type represents a set of individuals: $\\mathcal{I}(\\texttt{Painter}) = \\left\\{\\texttt{rembrandt}, \\texttt{vanGogh}\\right\\}$\n\t\\item Interpretation of a role/relation: $\\mathcal{I}(\\texttt{hasPainted}) = \\left\\{(\\texttt{rembrandt}, \\texttt{nightwatch}),(\\texttt{daVinci}, \\texttt{MonaLisa}) \\right\\}$ \n\t\\item $\\cdot^{\\mathcal{I}}$ is inductively extended over complex concept descriptions\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\top^{\\mathcal{I}} & = \\Delta^{\\mathcal{I}}\\\\\n\t\t\t\\bot^{\\mathcal{I}} & = \\emptyset\\\\\n\t\t\t(\\lnot C)^{\\mathcal{I}} & = \\Delta^{\\mathcal{I}}\\setminus C^{\\mathcal{I}}\\\\\n\t\t\t(C\\sqcap D)^{\\mathcal{I}} & = C^{\\mathcal{I}} \\cap D^{\\mathcal{I}}\\\\\n\t\t\t(C\\sqcup D)^{\\mathcal{I}} & = C^{\\mathcal{I}} \\cup D^{\\mathcal{I}}\\\\\n\t\t\t(\\exists r.C)^{\\mathcal{I}} & = \\left\\{x\\in \\Delta^{\\mathcal{I}}\\hspace{1mm}|\\hspace{1mm}\\exists y.(x,y)\\in r^{\\mathcal{I}} \\wedge y\\in C^{\\mathcal{I}} \\right\\}\\\\\n\t\t\t(\\forall r.C)^{\\mathcal{I}} & = \\left\\{x\\in \\Delta^{\\mathcal{I}}\\hspace{1mm}|\\hspace{1mm}\\forall y.(x,y)\\in r^{\\mathcal{I}} \\to y\\in C^{\\mathcal{I}} \\right\\}\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/kr_dl_interpretation_example.png}\n\t\t\\caption{Example for interpretation function $\\mathcal{I}$}\n\t\\end{figure}\n\t\\item Note that the quantifier $\\forall r.C$ also includes the empty set. To prevent this, use $\\forall r.C \\sqcap \\exists r.\\top$\n\t% kr_dl_interpretation_example.png\n\\end{itemize}\n\\subsection{Inference in Description Logic}\n\\begin{itemize}\n\t\\item A \\textbf{knowledge base} $\\mathcal{K} = (\\mathcal{T}, \\mathcal{A}) $ consists of:\n\t\\begin{itemize}\n\t\t\\item the terminology/theory $\\mathcal{T}$, that defines the general world model you can apply to any model. It summarizes the relations between concepts.\n\t\t\\begin{itemize}\n\t\t\t\\item The knowledge about relations between concepts is expressed by means of terminological axioms\n\t\t\t\\item \\textit{Concept inclusion}: $C\\sqsubseteq D$ models necessary conditions for object of type $C$ \n\t\t\t\n\t\t\tExample: $\\texttt{Elephant} \\sqsubseteq \\texttt{Animal} \\sqcap \\lnot\\texttt{Mouse}$\n\t\t\t\\item \\textit{Concept equivalence}: $C\\equiv D$ models necessary and sufficient conditions for object $C$. Can also be written as $C \\sqsubseteq D$ and $D \\sqsubseteq C$\n\t\t\t\n\t\t\tExample: $\\mathcal{T}$: ``\\textit{painter is a human that created a painting}'' $\\Rightarrow$ $\\texttt{Painter} \\equiv \\texttt{human}\\sqcap \\exists\\texttt{created}.\\texttt{Painting}$\n\t\t\\end{itemize}\n\t\t\\item the assertions $\\mathcal{A}$ that describes certain objects and the sets that are defined by $\\mathcal{T}$\n\t\t\\begin{itemize}\n\t\t\t\\item Knowledge about individuals in the domain expressed in terms of the vocabulary is specified by means of assertional axioms \n\t\t\t\\item \\textit{Concept assertion}: $a : C$ models that individual $a$ is of class $C$ \n\t\t\t\n\t\t\tExample: $\\texttt{dumbo}: \\texttt{Elephant}$\n\t\t\t\\item \\textit{Role assertions}: $(a, b) : r$ models that individual $a$ is related to individual $b$ by the role $r$\n\t\t\t\n\t\t\tExample: $(\\texttt{daVinci}, \\texttt{MonaLisa}) : \\texttt{painted}$\n\t\t\\end{itemize}\n\t\\end{itemize}\n\t\\item Definition of a \\textit{\\textbf{model}}\n\t\\begin{itemize}\n\t\t\\item An interpretation $\\mathcal{I}$ is a \\textit{model} of the $\\mathcal{T}$-box iff it satisfies every terminological axiom in $\\mathcal{T}$\n\t\t\\item An interpretation $\\mathcal{I}$ is a \\textit{model} of the $\\mathcal{A}$-box iff it satisfies every assertional axiom in $\\mathcal{A}$\n\t\t\n\t\tNote that this just requires $\\mathcal{I}$ to have the individuals and relations/assertions defined in $\\mathcal{A}$\n\t\t\\item Finally, an interpretation $\\mathcal{I}$ is a \\textit{model} of the knowledge base $\\mathcal{K} = (\\mathcal{T},\\mathcal{A})$-box iff $\\mathcal{I}$ is a model of both $\\mathcal{T}$ and $\\mathcal{A}$\n\t\t\\item A knowledge base is called \\textit{satisfiable}/\\textit{consistent} iff there exists a model for it.\n\t\\end{itemize}\n\t\\item In general, axioms of $\\mathcal{A}$- and $\\mathcal{T}$-box restrict the possible models\n\t\\item \\textbf{Reasoning tasks} for $\\mathcal{T}$-box\n\t\\begin{itemize}\n\t\t\\item \\textit{Concept satisfiability}: Concept $C$ is satisfiable w.r.t. $\\mathcal{T}$ iff there is a model $\\mathcal{I}$ of $\\mathcal{T}$: $C^{\\mathcal{I}} \\neq \\emptyset$\n\t\t\\item \\textit{Subsumption}: check if $\\mathcal{T} \\models C\\sqsubseteq D$. $C$ is subsumed by $D$ in $\\mathcal{T}$ iff $C^{\\mathcal{I}}\\subseteq D^{\\mathcal{I}}$ in every model $\\mathcal{I}$ of $\\mathcal{T}$\n\t\t\\item \\textit{Equivalence}: check if $\\mathcal{T} \\models C\\equiv D$. Concepts $C$ and $D$ are equivalent in $\\mathcal{T}$ iff $C^{\\mathcal{I}} = D^{\\mathcal{I}}$ in every model $\\mathcal{I}$ of $\\mathcal{T}$\n\t\t\\item All $\\mathcal{T}$-box problems can be reduced to concept satisfiability which is equivalent to showing that $C\\sqcap \\lnot D$ is unsatisfiable in $\\mathcal{T}$ (see section~\\ref{sec:tableau_algorithm} for Tableau algorithm to solve this)\n\t\\end{itemize}\n\t\\item \\textbf{Reasoning tasks} for $\\mathcal{A}$-box\n\t\\begin{itemize}\n\t\t\\item \\textit{$\\mathcal{A}$-box consistency}: $\\mathcal{A}$ is consistent w.r.t. $\\mathcal{T}$ iff there is a model of $\\mathcal{K}$. In such a case, $\\mathcal{K}$ is satisfiable.\n\t\t\\item \\textit{Instance checking}: check if $\\mathcal{K} \\models a:C$ (or respectively $\\mathcal{K} \\models (a,b) : r$). This holds iff for every model of $\\mathcal{K}$ is a model of $a:C$\n\t\t\\item \\textcolor{red}{Question: don't we also need the assertion that $\\mathcal{K}$ is consistent as otherwise we can prove anything for an unsatisfiable knowledge base?}\n\t\t\\item \\textit{Retrieval task}: given a concept $C$ and an $\\mathcal{A}$-box $\\mathcal{A}$, find all individuals $a$ such that $\\mathcal{K}\\models a:C$\n\t\t\\item \\textit{Realization task}: given an individual $a$ and a set of concepts, find the most specific concept $C$ such that $\\mathcal{K}\\models a:C$\n\t\t\\item All tasks are reducible to checking $\\mathcal{A}$-box consistency. We can check those by showing that $\\mathcal{A} \\cup \\left\\{a:\\lnot C\\right\\}$ is inconsistent.\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Tableau algorithm}\n\\label{sec:tableau_algorithm}\n\\begin{itemize}\n\t\\item All reasoning tasks in $\\mathcal{ALC}$ can be reduced to a single task of checking\n\t$\\mathcal{A}$-box consistency w.r.t. $\\mathcal{T}$-box.\n\t\\item Tableau shows (un-)satisfiability by contradiction. It searches through the tree of possible models and delivers a proof iff no model exists and therefore the input inconsistent is (or otherwise returns a valid model).\n\t\\item The general approach of Tableau is to extend the model until we find a model, or closed all branches\n\t\\item To reduce the number of tableau proves, we assume that all concepts appear in the Negation Normal Form (NNF). Conversion rules:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\lnot\\left(C \\cap D\\right) & \\Rightarrow \\left(\\lnot C \\cup \\lnot D\\right)\\\\\n\t\t\t\\lnot\\left(C \\cup D\\right) & \\Rightarrow \\left(\\lnot C \\cap \\lnot D\\right)\\\\\n\t\t\t\\lnot\\exists r.C & \\Rightarrow \\forall r.\\lnot C\\\\\n\t\t\t\\lnot\\forall r.C & \\Rightarrow \\exists r.\\lnot C\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item A \\textit{\\textbf{branch}} of a tableau is a set of $\\mathcal{A}$-box assertions. For any branch $S$, the following rules apply:\n%\t\\begin{figure}[ht!]\n%\t\t\\hspace{10mm}\n%\t\t\\includegraphics[width=0.6\\textwidth]{figures/kr_dl_tableau_rules.png}\n%\t\\end{figure}\n\t\\begin{itemize}\n\t\t\\item \\textbf{IF} $(a:C\\sqcap D)\\in S$ \\textbf{THEN} $S' := S \\cup \\left\\{a:C, a:D\\right\\}$\n\t\t\\item \\textbf{IF} $(a:C\\sqcup D)\\in S$ \\textbf{THEN} $S' := S \\cup \\left\\{a:C\\right\\}$ \\textbf{or} $S' := S \\cup \\left\\{a:D\\right\\}$\n\t\t\\item \\textbf{IF} $(a:\\exists r.C)\\in S$ \\textbf{THEN} $S' := S \\cup \\left\\{(a,b):r, b:C\\right\\}$ where $b$ is a ``fresh'' individual name in $S$\n\t\t\\item \\textbf{IF} $(a:\\forall r.C)\\in S$ \\textbf{and} $(a,b):r \\in S$ \\textbf{THEN} $S' := S \\cup \\left\\{b:C\\right\\}$\n\t\t\\item \\textbf{IF} $\\left\\{a:C, a:\\lnot C\\right\\}\\subseteq S$ \\textbf{THEN} mark branch as CLOSED (unsatisfiable)\n\t\\end{itemize}\n\t\\item To check for satisfiability, we assume that there is an individual of that given concept. Try to show that this leads to an contradiction.\n\t\\item Example: \\textit{show that $\\exists r.A \\sqcap \\exists r.B$ is subsumed by $\\exists r.(A\\sqcap B)$}\n\t\\begin{enumerate}\n\t\t\\item Write expression as concept: $\\exists r.A \\sqcap \\exists r.B \\sqsubseteq \\exists r.(A\\sqcap B)$\n\t\t\\item Negate concept (proof by contradiction): $\\exists r.A \\sqcap \\exists r.B \\sqcap \\lnot \\exists r.(A\\sqcap B)$\n\t\t\\item Rewrite concept in NNF: $\\exists r.A \\sqcap \\exists r.B \\sqcap \\forall r.(\\lnot A\\sqcup \\lnot B)$\n\t\t\\item Assume $\\mathcal{A}$-box $\\mathcal{A}=\\left\\{a: \\exists r.A \\sqcap \\exists r.B \\sqcap \\forall r.(\\lnot A\\sqcup \\lnot B)\\right\\}$ and search for a model\n%\t\t\\begin{itemize}\n%\t\t\t\\item $a: \\exists r.A$, $a: \\exists r.B$, $a: \\forall r.(\\lnot A\\sqcup \\lnot B)$\n%\t\t\t\\item $(a,b):r$, $b:A$, $(a,c):r$, $c:B$\n%\t\t\t\\item $b:(\\lnot A\\sqcup \\lnot B)$, $c:(\\lnot A\\sqcup \\lnot B)$\n%\t\t\t\n%\t\t\\end{itemize}\n% kr_dl_tableau_proof_example.png\n\t\t\\begin{figure}[ht!]\n\t\t\t\\hspace{30mm}\n\t\t\t\\includegraphics[width=0.5\\textwidth]{figures/kr_dl_tableau_proof_example.png}\n\t\t\\end{figure}\n\t\t\\item Step 13 shows the valid model where $\\Delta^{\\mathcal{I}}=\\left\\{a,b,c\\right\\}$, $A^{\\mathcal{I}}=\\left\\{b\\right\\}$, $B^{\\mathcal{I}}=\\left\\{c\\right\\}$, $r^{\\mathcal{I}}=\\left\\{(a,b), (a,c)\\right\\}$ $\\Rightarrow$ the negated concept is satisfiable and thus proves the hypothesis to be wrong\n\t\t\n\t\tIf we would not have been able to construct a model, the hypothesis would be correct \n\t\\end{enumerate}\n\t\\item Note that the tableau algorithm is sound (if algorithm finds a proof, then statement is correct) and complete (if statement is correct, the algorithm finds a proof).\n\\end{itemize}\n\\subsubsection{Reasoning with non-empty $\\mathcal{T}$-box}\n\\begin{itemize}\n\t\\item The tableau algorithm can be extended in order to support terminology axioms of the $\\mathcal{T}$-box\n\t\\item Input preprocessing\n\t\\begin{itemize}\n\t\t\\item Replace every $C\\equiv D \\in \\mathcal{T}$ with $C\\sqsubseteq D$ and $D\\sqsubseteq C$\n\t\t\\item Replace every $C\\sqsubseteq D \\in \\mathcal{T}$ with $\\top \\equiv NNF(\\lnot C \\sqcup D)$\n\t\t\\item Add all concepts/formula of $\\mathcal{T}$ to the root $S_0$ of the tableau\n\t\\end{itemize}\n\t\\item Extend the tableau rules by the following:\n\t\\begin{itemize}\n\t\t\\item \\textbf{IF} $(\\top \\equiv C) \\in S$ \\textbf{and} an individual $a$ occurs in $S$ \\textbf{THEN} $S' := S\\cup\\left\\{a:C\\right\\}$\n\t\\end{itemize}\n\t\\item Example:\n\t\\begin{figure}[ht!]\n\t\t\\hspace{10mm}\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/kr_dl_tableau_nonempty_tbox_example.png}\n\t\\end{figure}\n\t\\item In case of reasoning with non-empty $\\mathcal{T}$-box, the tableau algorithm is not guaranteed to terminate (rules in the form of $\\top\\equiv ...\\exists r.C...$ can lead to an infinite loop)\n\t\\item Solution: Detect cycles and prevent further application of the $\\Rightarrow_{\\exists}$ rule. This is achieved by a special blocking rule:\n\t\\begin{itemize}\n\t\t\\item \\textbf{IF} $b$ is a (possibly indirect) successor of $a$ in $S$ \\textbf{and} it is the case that: $$\\left\\{C\\hspace{1mm}|\\hspace{1mm} b:C \\in S\\right\\} \\subset \\left\\{D\\hspace{1mm}|\\hspace{1mm} a:D \\in S\\right\\} $$\n\t\t\n\t\t\\textbf{THEN} mark $b$ as BLOCKED by $a$ in $S$ and do not apply $\\Rightarrow_{\\exists}$ rule to $b$.\n\t\\end{itemize}\n\t\\item Example:\n\t\\begin{figure}[ht!]\n\t\t\\hspace{10mm}\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/kr_dl_tableau_blocking_example.png}\n\t\\end{figure}\n\t\\item Note that we can only apply the blocking rule if no other rules apply anymore on that branch.\n\\end{itemize}"
  },
  {
    "path": "Knowledge_Representation/kr_intro.tex",
    "content": "\\section{Introduction to KR}\n\\begin{itemize}\n\t\\item There are two main lines of development in AI: \\textit{symbolic} and \\textit{statistical} representation\n\t\\item Both approaches go along with different benefits/weaknesses\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/kr_intro_symbolic_vs_statistical.png}\n\t\t\\caption{Weaknesses of symbolic and statistical representation in AI}\n\t\\end{figure}\n\t\\begin{itemize}\n\t\t\\item \\textit{\\textbf{Construction}}: effort that is needed to create such a system\n\t\t\\begin{itemize}\n\t\t\t\\item A symbolic AI requires a knowledge base on which it bases its reasoning. This knowledge base is mostly created by a human which can take a lot of time. For example, the SNOMED database was created in more than 40 years and contains now about $300,000$ definitions. The construction of knowledge bases is summarized in the research area \\textit{Knowledge Engineering}.\n\t\t\t\\item In the connectionist/statistical approach, the model learns from data so that we need to provide a (huge) dataset. Depending on the task and the required labels, it can also take a lot of human effort until we have the required data size.\n\t\t\\end{itemize}\n\t\t\\item \\textit{\\textbf{Scalability}}: effect of data size/amount on the systems\n\t\t\\begin{itemize}\n\t\t\t\\item The more data we have for an symbolic AI, the easier it is to run into a problem. Huge knowledge bases tend to be not sound anymore (consistent) as a small mistake at one point can lead to wrong reasoning for any problem (if knowledge base is unsatisfiable, then all given problems are unsatisfiable), So we need to put extra effort in ensuring the soundness of the knowledge base.\n\t\t\t\\item Connectionist approaches learn from the statistics of a dataset which gets more accurate by increasing the amount of data. Small errors/noise are thereby smoothed out. However, this also means that statistical representations are inaccurate if there is only a small amount of data. \n\t\t\\end{itemize}\n\t\t\\item \\textit{\\textbf{Explainable}}: understanding how the system came to its decision\n\t\t\\begin{itemize}\n\t\t\t\\item Symbolic AI is dedicated to creating explainable systems as they only applies facts/statements/rules of the knowledge base that was hand-crafted and thus understandable for a human. The reasoning of such an AI system is explainable by the used and newly derived rules.\n\t\t\t\\item In contrast, connectionist approaches are less explainable. As they capture the data distribution in a very high dimensional space (e.g. neural networks with millions of parameters), it more serves as a black box. Errors that are produced by small, carefully selected input noises are harder to understand and to prevent. \n\t\t\\end{itemize}\n\t\t\\item \\textit{\\textbf{Generalization}}: performance across unseen domains\n\t\t\\begin{itemize}\n\t\t\t\\item Symbolic AI relies on the given knowledge base. If we try to reason about a domain that is unknown in the knowledge base, we don't get any answer (or rather that the reasoning ended without a result). For example, if we have a system based on SNOMED and try to show that if it is raining outside, I probably get wet the AI terminates without an solution because it is not provided by the needed rules/facts.\n\t\t\t\\item Commonly, statistical AI systems are already limited to their specific domain. In the task of classification, we usually have a fixed set of classes from which the system has to choose one. If we show a new image that does not belong to any of those, it will try to find the most similar class of what the system has so far (borders in high-dimensional space).\n\t\t\\end{itemize}\n\t\\end{itemize}\n\t\n\\end{itemize}"
  },
  {
    "path": "Knowledge_Representation/kr_qr.tex",
    "content": "\\section{Qualitative Reasoning}\n\\begin{itemize}\n\t\\item Learning by making (qualitative) representations, combine meanings and reason with them\n\t\\item Differs with numerical reasoning by not having exact values (we can express that something is positive, but we don't say it has the value $3.21$) \n\\end{itemize}\n\\subsection{General vocabulary}\n\\begin{itemize}\n\t\\item In QR, we express systems by the use of entities (like \\textit{population}), and describe them by their quantities (e.g. \\textit{size})\n\t\\item We can define relations and inequalities between those. The full vocabulary is visualized in Figure~\\ref{fig:kr_qr_vocabulary}\n\\end{itemize}\n\\begin{figure}[ht!]\n\t\\centering\n\t\\includegraphics[width=0.45\\textwidth]{figures/kr_qr_vocabulary.png}\n\t\\caption{Overview of the vocabulary used in QR}\n\t\\label{fig:kr_qr_vocabulary}\n\\end{figure}\n\\subsubsection{Quantities}\n\\begin{itemize}\n\t\\item Quantities are described by a magnitude and a derivative (can also include higher order derivatives)\n\t\\item The quantities are represented by a discrete scales which are build up by\n\t\\begin{itemize}\n\t\t\\item \\textit{Intervals} which defines a range of values (e.g. $+$ for positive values)\n\t\t\\item \\textit{Landmarks} which reflect key points of the system, and are defined as single number (e.g. $0$ or $\\max$)\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Influence and Proportional}\n\\begin{itemize}\n\t\\item \\textbf{Proportional}: If \\texttt{Q1} increases, then \\texttt{Q2} increases (P+)/decreases (P$-$):\n\t$\\partial \\texttt{Q1}>0\\implies \\partial \\texttt{Q2}>0$\n\t\\begin{itemize}\n\t\t\\item Mathematically, we can define the relation by:\n\t\t$$\\texttt{Q2}\\propto_{Q+}\\texttt{Q1} \\hspace{2mm}\\equiv\\hspace{2mm} \\exists f\\hspace{1mm}|\\hspace{1mm}\\texttt{Q2} = f(...,\\texttt{Q1},...) \\wedge f \\text{ is increasing monotonic in }\\texttt{Q1}$$\n\t\t\\item This means that if \\texttt{Q2} is \\textit{qualitatively} proportional to \\texttt{Q1}, we can express \\texttt{Q2} by a function based on \\texttt{Q1} such that it increases if \\texttt{Q1} increases (and everything else stays the same)\n\t\t\\item Monotonic functions provide an abstraction that cover a wide range of more mathematical expressions which can be narrowed by further constraints\n\t\\end{itemize}\n\t\\item \\textbf{Influence}: If \\texttt{Q1} is greater than 0, then \\texttt{Q2} increases (I+)/decreases (I$-$):\n\t$\\texttt{Q1} > 0\\implies \\partial \\texttt{Q2}>0$\n\t\\begin{itemize}\n\t\t\\item Expressed in mathematical terms, the influence relation is stated as:\n\t\t$$I^{+}(\\texttt{Q2},\\texttt{Q1})\\hspace{1mm} \\equiv\\hspace{1mm} d\\texttt{Q2}/dt=...+B+...$$\n\t\t\\item Note that this function definition is much more specific than for proportional. Hence it enables us knowledge of relative rates if multiple influences are given. Example: if $I^{+}(\\texttt{Q2},\\texttt{Q1})\\wedge I^{-}(\\texttt{Q2},\\texttt{Q3})$, then we know that if $\\texttt{Q1}>\\texttt{Q3}$, $\\texttt{Q2}$ increases.\n\t\t\\item \\textit{Warning}: for reasoning with influences, we often require the second-order derivative (if \\texttt{Q1} has a derivative and doesn't change its qualitative magnitude, we can't model the change in the derivative of \\texttt{Q2}). This is especially important if we have a positive and negative influence which both are in a interval with their magnitude.\n\t\\end{itemize}\n\t\\item Example in formulas:\n\t\\begin{itemize}\n\t\t\\item The change of the size of a closed population can be specified by:\n\t\t$N_{t+1} = N_{t} + B_{t} - D_{t}$\n\t\twhere $B$ is the number of births, and $D$ the number of deaths between $t$ and $t+1$.\n\t\t\\item Then, we have a positive influence of $B$ to $N$, and a negative influence of $D$ to $N$: $I^{+}(N,B)$, $I^{-}(N,D)$\n\t\t\\item Further, we specify that the birth and death rate proportionally depend on the population size: $B=f_B(...,N,...)$, $D=f_D(...,N...)$ $\\Rightarrow$ $P^{+}(D,N)$, $P^{+}(B,N)$\n\t\\end{itemize}\n\t\\item Both relations can lead to ambiguity if one influence/proportion is positive, and another is negative. Without given any other facts, all outcomes are possible new states\n\\end{itemize}\n\\subsubsection{Inequalities}\n\\begin{itemize}\n\t\\item We can define inequalities between quantities (magnitudes), landmarks and derivatives\n\t\\item Inequalities state a rather uncertain knowledge, or assumptions that are valid for at least the initial state\n\t \\item For example, we can state that $A$ is larger than $B$, but this might change during the simulation\n\t \\item Also, we can reason with inequalities by propagating knowledge (if $A>B$ and $B>C$, then $A>C$). This can for example lead to constraints in the value space of another quantity\n\\end{itemize}\n\\subsubsection{Value constraint}\n\\begin{itemize}\n\t\\item The value constraint binds landmarks/intervals of two quantities together.\n\t\\item If we have a value constraint from $A$ to $B$ to both 0, this means that if $A=0$ then $B=0$ (directed relation!)\n\\end{itemize}\n\\subsection{States and Transitions}\n\\begin{itemize}\n\t\\item A qualitative state is a period of time in which the qualitative behavior of the system doesn't change (changes include magnitudes, derivatives, inequalities, etc.)\n\t\\item Thus, states are \\textit{unique} sets of inequality quantity expressions\n\t\\item Transitions are changes of at least one quantity (but all changes must take place to enter the next state if multiple quantities change)\n\\end{itemize}\n\\begin{figure}[ht!]\n\t\\centering\n\t\\includegraphics[width=0.5\\textwidth]{figures/kr_qr_state_graph_example.png}\n\t\\caption{Example state graph for heating water in a cattle}\n\t\\label{fig:kr_qr_state_graph_example}\n\\end{figure}\n\\subsubsection{Compositional modeling}\n\\begin{itemize}\n\t\\item To improve re-usability of QR definitions, we can split the model into several parts\n\t\\item We define a hierarchy of generic entities (e.g. population) on which we will operate\n\t\\item The library of model fragments contain structures that defines certain entities/quantities and their relation (if a population exists, then...). These can be conditioned on assumptions (e.g. closed or open population)\n\t\\item In the scenarios, we define certain instances of our entity hierarchy (e.g. ``\\textit{Green Frogs}'' as instance of entity population). Based on them, we can state initial values (e.g. population size is max), and assumptions we make (e.g. closed population)\n\t\\item We then combine both the scenario and the model fragment to create the behavior graph\n\\end{itemize}\n\\begin{figure}[ht!]\n\t\\centering\n\t\\includegraphics[width=0.5\\textwidth]{figures/kr_qr_compositional_modeling.png}\n\t\\caption{Compositional modeling}\n\t\\label{fig:kr_qr_compositional_modeling}\n\\end{figure}\n\\subsubsection{Finding transitions}\n\\begin{itemize}\n\t\\item There are three common types of termination \n\t\\begin{itemize}\n\t\t\\item \\textbf{Value termination}: if a quantity has a derivative in a certain direction, then it can move to the next quantity point in this direction\n\t\t\\item \\textbf{Inequality termination}: Given a inequality, if one side is changing due to derivatives, then the truth value might change (from $\\texttt{Q2}>\\texttt{Q1}$ to $\\texttt{Q2}=\\texttt{Q1}$, remember continuous changes!)\n\t\t\\item \\textbf{Exogenous termination}: An external effect/input that leads to a change. Can for example be controlling the derivative of a quantity. The behavior can be random, sinusoidal, random, etc.\n\t\\end{itemize}\n\t\\item Another important concept for deciding on transitions is \\textbf{Epsilon ordering} dealing with value termination\n\t\\begin{itemize}\n\t\t\\item We distinguish between \\textit{immediate} and \\textit{non-immediate} transitions\n\t\t\\item Immediate transitions are if a derivative is unequals 0 and the magnitude is currently at a landmark/point. Then, we will immediately move to the next state with the magnitude being changed\n\t\t\\item Non-immediate transitions are when the derivative is unequals zero, but the magnitude is in an interval. Then it can change at some point, but does not have to be\n\t\\end{itemize}\n\t\\item Another concept we have to consider is value correspondence\n\t\\begin{itemize}\n\t\t\\item If two values are restraint by a value correspondence, then this can limit the possible transitions we can have\n\t\t\\item For example, in Figure~\\ref{fig:kr_qr_value_correspondence_termination}, we can either apply \\texttt{T1} or \\texttt{T2}, but both can't happen at the same time because then the value correspondence is not fulfilled\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.4\\textwidth]{figures/kr_qr_value_correspondence_termination.png}\n\t\t\t\\caption{Example for transitions that exclude each other because of a value constraint}\n\t\t\t\\label{fig:kr_qr_value_correspondence_termination}\n\t\t\\end{figure}\n\t\\end{itemize}\n\\end{itemize}"
  },
  {
    "path": "Knowledge_Representation/kr_sat.tex",
    "content": "\\section{Satisfiability solvers}\n\\subsection{Propositional Logic}\n\\begin{itemize}\n\t\\item In Knowledge Representation, we have three sets of rules/formula:\n\t\\begin{itemize}\n\t\t\\item \\textit{Knowledge base}: statements which are known to be true/need to be fulfilled by all models (\\textit{axioms})\n\t\t\\item \\textit{Premises}: statements that are only true for certain states/inputs (\\textit{implications} at partial truth value assignment)\n\t\t\\item \\textit{Conclusion}: statements for which we want to check if there exists an model for (\\textit{conjecture}). Derived by knowledge base and premises\n\t\\end{itemize} \n\t\\item Propositional logic is based on simple statements (\\textit{literals}) that can be combined to complex statements (\\textit{formula})\n\t\\begin{itemize}\n\t\t\\item \\textit{Conjunction}: $A\\wedge B$\n\t\t\\item \\textit{Disjunction}: $A\\vee B$\n\t\t\\item \\textit{Negation}: $\\lnot A$\n\t\t\\item \\textit{Implication}: $A\\to B$ ($\\equiv\\lnot A \\vee B\\equiv \\lnot (A \\wedge \\lnot B)$)\n\t\\end{itemize}\n\t\\item Truth values are assigned to literals by an interpretation function $I$\n\t\\item \\textbf{Clausal normal form}\n\t\\begin{itemize}\n\t\t\\item Every formula can be rewritten in CNF (conjunction of disjunctions)\n\t\t\\item Example: $(A\\vee B\\vee C\\vee D)\\wedge(E\\vee F)\\wedge(\\lnot A \\vee F \\vee D)\\wedge ...$\n\t\t\\item To rewrite a formula to CNF, we have to remove implications ($A\\to B \\equiv\\lnot A \\vee B$), move negations in front of the literals ($\\lnot (A\\vee B) \\equiv \\lnot A \\wedge \\lnot B$) and move conjunctions outside ($A \\vee (B\\wedge C) \\equiv (A\\vee B)\\wedge (A \\vee C)$) \n\t\\end{itemize}\n\t\\item Propositional logic is a weak language i.e. it is less expressive (no instances, no functions on terms, ...)\n\t\\item We can express first-order logic in propositional logic by instantiating all quantifiers and all possible input arguments to predicates\n\t\\begin{itemize}\n\t\t\\item Only possible for finite domains. Exponential explosion of number of instances\n\t\t\\item Example: Domain $\\left\\{A, B, C\\right\\}$, $\\forall x P(x) \\vee Q(x,x)$ $\\implies$ $(P\\_A \\vee Q\\_A\\_A)\\wedge (P\\_B \\vee Q\\_B\\_B) \\wedge (P\\_C \\vee Q\\_C\\_C)$\n\t\t\\item $\\exists x \\forall y Q(x, y) \\implies (Q\\_A\\_A \\wedge Q\\_A\\_B \\wedge Q\\_A\\_C) \\vee (Q\\_B\\_A\\wedge...)\\vee ...$\n\t\\end{itemize}\n\t\\item \\textit{Tautology}: formula that is always true (e.g. $A\\vee \\lnot A$), also called valid sentence. For $n$ symbols, we have $2^n$ models.\n\t\\item \\textit{Contradiction}: formula that is always false (e.g. $A\\wedge \\lnot A$), also called inconsistent sentence\n\t\\item \\textit{Satisfiable sentence}: formula that can be made true in at least one world (i.e. not inconsistent)\n\t\\item A \\textit{model} is a ``possible world'' (truth assignment) in which the knowledge base is true (including conjecture if given)\n\t\\item $P\\models Q$ means that any model of $P$ also fulfills $Q$\n\\end{itemize}\n\\subsection{Davis Putman algorithm}\n\\begin{itemize}\n\t\\item Algorithm for proving if there exists an model for a given knowledge base, \\textbf{and} returns it as a result in that case\n\t\\item In general, this problem is NP-complete\n\t\\begin{itemize}\n\t\t\\item Any other NP problem can be reduced to SAT\n\t\t\\item Exponential time $\\mathcal{O}(2^n)$ with number of literals, but algorithms are optimized to be fast for most cases (only worst case is exponential)\n\t\\end{itemize}\n\t\\item The DPLL algorithm is summarized in Figure~\\ref{fig:kr_sat_dpll_algorithm} and described in more detail here:\n\t\\begin{enumerate}\n\t\t\\item[Step 1] \\textbf{Simplification}: iteratively remove unnecessary clauses and derive literal implications\n\t\t\\begin{itemize}\n\t\t\t\\item \\textit{Tautology}: remove tautologies like $P\\vee \\lnot P$ from knowledge base (once in the beginning)\n\t\t\t\\item \\textit{Pure literals}: set predicates that solely occur in their positive \\textit{or} negative form to the corresponding truth value\n\t\t\t\\item \\textit{Unit clauses}: set literals for which the knowledge base contain a unit clause to true (or the predicate to false respectively)\n\t\t\\end{itemize}\n\t\t\\item[Step 2] \\textbf{Split}: pick a predicate and assume a truth value\n\t\t\\begin{itemize}\n\t\t\t\\item Heuristics of which literal to pick next can improve the efficiency of DPLL a lot\n\t\t\t\\item \\textbf{DLCS} (\\textit{Dynamic Largest Combined Sum}): Pick $v$ with the largest count of positive and negative occurrences: $CP(v) + CN(v)$. If $CP(v) > CN(v)$, choose $v=1$, else $v=0$\n\t\t\t\\item \\textbf{DLIS} (\\textit{Dynamic Largest Individual Sum}): Pick $v$ with either largest $CP$ or $CN$. Same truth assignment as for \\textit{DLCS}.\n\t\t\t\\item \\textbf{Jeroslow-Wang}: weight of literal depends on the length of clauses it occurs in. Thereby, we prefer small clauses. The score of a literal is $J(v)=\\sum_{c\\in \\mathcal{C}_v} 2^{-|c|}$, and we pick the highest value. One-sided JW looks at $v$ and $\\lnot v$ independently, whereas the two-sided approach looks at sum of $v$ and $\\lnot v$, and just picks the truth value based on $J(v)\\geq J(\\lnot v)\\Rightarrow v=1$, else $v=0$.\n\t\t\t\\item \\textbf{MOMs} (\\textit{Maximum Occurrences in Clauses of Minimum Size}): similar to JW, but we only look at the smallest clauses in the knowledge base. The number of occurrences in those is indicates by the function $f$. We choose the literal that maximizes $[f(v)+f(\\lnot v)]*2^{k}+f(v)\\cdot f(\\lnot v)$. $k$ is a tuning parameter for the trade-off between the balanced distribution of $v$ and $\\lnot v$ and their individual ones.\n\t\t\\end{itemize}\n\t\\end{enumerate}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.6\\textwidth]{figures/kr_sat_dpll_algorithm.png}\n\t\t\\caption{DPLL pseudo-code algorithm}\n\t\t\\label{fig:kr_sat_dpll_algorithm}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{Clause Learning}\n\\begin{itemize}\n\t\\item \\textit{Non-chronological backtracking}: If we encounter a conflict at the end of a search branch, we try to find the root of the problem, and then backtrack to the that point.\n\t\\item The implications (decisions/assignments) that lead to a conflict can be modeled as an acyclic graph where each node represents a literal assignment and each edge represents the reason for that assignment (see Figure~\\ref{fig:kr_sat_conflict_clause_figure})\n\t\\item We can then partition the graph such that one side contains at least all decision variables (called \\textit{reason side}), and the other the conflict literal (called \\textit{conflict side}). \n\t\\item The conflict clause is determined by the negations of the literals associated with the cut between both sides. \n\t\\item Different cuts of the implication graph distinguish learning schemes from one another as they imply different conflict clauses and hence\n\tthe information gained from them.\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.3\\textwidth]{figures/kr_sat_conflict_clause_figure.pdf}\n\t\t\\caption{Conflict clause implication graph}\n\t\t\\label{fig:kr_sat_conflict_clause_figure}\n\t\\end{figure}\n\t\\item Conflict clause learning is a selective application of resolution (probably more useful than random resolution). General resolution rule:\n\t$$\\frac{\\hspace{2mm}A\\vee \\lnot B\\hspace{6mm}C\\vee B\\hspace{2mm}}{A\\vee C}$$\n\\end{itemize}\n\\subsection{Stochastic solver}\n\\begin{itemize}\n\t\\item Properties of a SAT solver\n\t\\begin{itemize}\n\t\t\\item \\textit{Decidability} = completeness. \n\t\t\\begin{itemize}\n\t\t\t\\item This means that given enough runtime, the SAT solver guarantees to find an assignment or returns that there is no solution.\n\t\t\t\\item Although DPLL is complete in theory, it is not in practice as we have limited runtime to wait for an answer\n\t\t\t\\item Note that \\textit{undecidability} just indicates that it is not \\textit{always} guaranteed to get an answer (may be harmless in practice)\n\t\t\\end{itemize}\n\t\t\\item \\textit{Complexity}: exponential maximum runtime $\\mathcal{O}(n^2)$\n\t\t\\begin{itemize}\n\t\t\t\\item However, mean instead of worst-case runtime is more important in practice\n\t\t\t\\item Also, $\\mathcal{O}$-notation does not consider constants whereas with limited runtime, it might be important\n\t\t\t\\begin{figure}[ht!]\n\t\t\t\t\\centering\n\t\t\t\t\\includegraphics[width=0.2\\textwidth]{figures/kr_sat_prob_worst_case.png}\n\t\t\t\t\\caption{Comparison of worst case runtime $\\mathcal{O}(n)$ and $\\mathcal{O}(n^2)$ with different constants}\n\t\t\t\t\\label{fig:kr_sat_prob_worst_case}\n\t\t\t\\end{figure}\n\t\t\\end{itemize}\n\t\\end{itemize}\n\t\\item A good measurement for complexity in case of SAT solvers has been shown to be the ratio of clauses to variables (see Figure~\\ref{fig:kr_sat_prob_hardness})\n\t\\item Problems with a low ratio are easy to solve as they have (in average) many solutions\n\t\\item Problems with a high ratio tend to have no solution, and are easy to show that there exists a conflict (see Figure~\\ref{fig:kr_sat_prob_hardness_2})\n\t\\item In between, around $4.26$, are apparently the hardest randomly generated problems as there might a solution (or only a few) or not. This points is also referred to as \\textit{phase transition}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\begin{subfigure}[b]{0.23\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=\\textwidth]{figures/kr_sat_prob_hardness.png}\n\t\t\t\\caption{Problem complexity}\n\t\t\\end{subfigure}\n\t\t\\hspace{10mm}\n\t\t\\begin{subfigure}[b]{0.4\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=\\textwidth]{figures/kr_sat_prob_hardness_2.png}\n\t\t\t\\caption{Proportion of satisfiable problems}\n\t\t\t\\label{fig:kr_sat_prob_hardness_2}\n\t\t\\end{subfigure}\n\t\t\\caption{Hardness in SAT problems}\n\t\t\\label{fig:kr_sat_prob_hardness}\n\t\\end{figure}\n\t\\item Stochastic solvers have been shown to perform quite well on such randomly generated SAT instances, but might perform poorly compared to DPLL on highly structured problems\n\t\\item It is not intended to find all solution neither that there exists none. But is for example useful for MAXSAT (see later)\n\t\\item Stochastic SAT solvers perform local search \n\t\\begin{enumerate}\n\t\t\\item Make a guess (smart or random) about values of the variables\n\t\t\\item Evaluate how many clauses are broken\n\t\t\\item Try flipping a variable to make things better for a certain number of iterations (various heuristics in which variables to flip next)\n\t\\end{enumerate}\n\t\\item This search is repeated $N$ times until we either find a solution or terminate without an answer\n\t\\item \\textbf{GSAT}: Local greedy search/algorithm of picking the next variable to flip which increases the number of satisfied clauses the most (ties are broken randomly, note that flips will break also some clauses).\n\t\\item Full algorithm shown in Figure~\\ref{fig:kr_sat_prob_GSAT}\n\t\\item Note that this algorithm tends to get stuck in local minimum (flipping a single variable does not increase score). Thus, we perform random restarts to start new. That's also why GSAT spends most time on plateaus where score is not improved\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/kr_sat_prob_GSAT.png}\n\t\t\\caption{Pseudo-code of GSAT}\n\t\t\\label{fig:kr_sat_prob_GSAT}\n\t\\end{figure}\n\\end{itemize}\n\\subsection{MAXSAT}\n\\begin{itemize}\n\t\\item MAXSAT is Proportion \\textit{optimization} extension of SAT that asks what is the maximum number of clauses that can be simultaneously satisfied\n\t\\item Example: $\\lnot A\\wedge (A \\vee B) \\wedge (\\lnot B)$ is a contradiction. But the truth assignment $\\pi = \\left\\{\\lnot A, \\lnot B\\right\\}$ maximizes number of satisfied clauses\n\t\\item Some clauses might be more important to be satisfied than others $\\Rightarrow$ adding a cost/positive weight to every clause $C$ that will be incurred if $C$ is falsified\n\t\\item A cost of $\\infty$ indicates \\textit{hard} clauses that are mandatory to satisfy, whereas \\textit{soft} clauses have a finite cost\n\t\\item We try to minimize the sum of the costs of all unsatisfied clauses\n\t\\item There are different variations of MAXSAT:\n\t\\begin{itemize}\n\t\t\\item (Standard) MAXSAT: no hard clauses and all have a weight of 1 (solution maximizes the number of satisfied clauses)\n\t\t\\item \\textit{Weighted} MAXSAT: no hard clauses, but soft clauses with any finite, positive weight\n\t\t\\item \\textit{Partial} MAXSAT: hard clauses are allowed, but all soft clauses have weight 1\n\t\t\\item \\textit{Weighted Partial} MAXSAT: both hard and soft clauses are allowed. Subsumes all previously mentioned variations.\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{SAT planning}\n\\begin{itemize}\n\t\\item Real-world planning problems can be translated to a SAT problem, and the plan can be extracted from the truth assignments\\\\\n\t\\begin{minipage}{.65\\textwidth}\n\t\t\\item Formal definition of the planning problem\n\t\\begin{itemize}\n\t\t\\item States $S=\\left\\{...,s_i,...\\right\\}$ (in the example $S=\\left\\{s_0, s_1, s_2, s_3, s_4, s_5\\right\\}$)\n\t\t\\item Actions $A=\\left\\{..., a_i,...\\right\\}$ (in the example $A=\\left\\{\\texttt{move1}, \\texttt{move2}, \\texttt{put}, \\texttt{take}, \\texttt{load}, \\texttt{unload}\\right\\}$)\n\t\t\\item State-transition function $\\gamma:S\\times A\\to S$ (note that we only consider deterministic transitions, otherwise $2^S$ output)\n\t\t\\item Planning problem $P=(\\Sigma, s_0, s_G)$ where $\\Sigma=(S,A,\\gamma)$, and $s_0$ initial state and $s_G$ (set of) goal states\n\t\t\\item Classical plan is a sequence of actions: $\\pi = \\langle a_0, a_1, ..., a_{n-1}\\rangle$\n\t\t\\item Policy is a partial function from $S$ to $A$\n\t\\end{itemize}\n\t\\end{minipage}\n\t\\hspace{10mm}\n\t\\begin{minipage}{.3\\textwidth}\n\t\t\t\\includegraphics[width=0.8\\textwidth]{figures/kr_sat_planning_example.png}\n\t\t\t\\label{fig:kr_sat_planning_example}    \n\t\\end{minipage}\n\t\\item Translation for plan $P$ and a fixed plan length $n$:\n\t\\begin{itemize}\n\t\t\\item If $\\pi=\\langle a_0, a_1, ..., a_{n-1}\\rangle$ is a solution to the planning problem, we know that the traversed states are $s_0, s_1 = \\gamma(s_0, a_0), s_2 = \\gamma(s_1, a_1), ..., s_n = \\gamma(s_{n-1}, a_{n-1})$ (where $a_i$ is the $i$-th step of $\\pi$, and $s_i$ the states in which the agent is at step $i$)\n\t\t\\item We denote all possible literals with $L$\n\t\t\\item Formula describing the initial state: $\\bigwedge\\left\\{l_0\\hspace{1mm}|\\hspace{1mm}l \\in s_0\\right\\} \\wedge \\bigwedge\\left\\{\\lnot l_0\\hspace{1mm}|\\hspace{1mm}l \\in L-s_0\\right\\}$\n\t\t\\item Formula describing the goal state: $\\bigwedge\\left\\{l_n\\hspace{1mm}|\\hspace{1mm}l \\in g^+\\right\\} \\wedge \\bigwedge\\left\\{\\lnot l_n\\hspace{1mm}|\\hspace{1mm}l \\in g^{-}\\right\\}$ ($g^{+}$ valid goal states, $g^{-}$ invalid goal states)\n\t\t\\item Formula describing the state-transitions. For every $a_i$ add $\\bigwedge\\left\\{p_i\\hspace{1mm}|\\hspace{1mm}p\\in \\text{Precond}(a)\\right\\} \\wedge \\bigwedge\\left\\{e_{i+1}\\hspace{1mm}|\\hspace{1mm}e\\in \\text{Effects}(a)\\right\\}$\n\t\t\\item Complete exclusion axiom: only one action per step ($\\lnot a_{1,i} \\vee a_{2,i}$,...)\n\t\t\\item Frame axioms that describe what \\textit{doesn't} change between $i$ and $i+1$: $\\left(\\lnot l_i \\wedge l_{i+1} \\Rightarrow \\bigvee_{a\\in A} \\left\\{ a_i | l \\in \\text{Effects}^+ (a) \\right\\} \\right) \\wedge \\left(l_i \\wedge \\lnot l_{i+1} \\Rightarrow \\bigvee_{a\\in A} \\left\\{ a_i | l \\in \\text{Effects}^- (a) \\right\\} \\right)$ ($\\text{Effects}^+ (a)$: set of literals that change their truth value to true if $a$ is performed, and $\\text{Effects}^- (a)$ those that are changed to false)\n\t\\end{itemize}\n\t\\item We apply a SAT solver on this knowledge base. If we find a solution, then $P$ has a solution of length $n$\n\t\\item To find solutions of shortest length, we loop over different values for $n$. If we don't find any solution for $n=0$, we encode the problem for $n=1$, and so on.\n\t\\item In practice, SAT solvers for planning take too much time and memory, but can be combined with other techniques like planning-graph expansion (SatPlan).\n\\end{itemize}\n\\subsection{Applications of SAT}\n\\begin{itemize}\n\t\\item Model checking in terms of hardware and software verification (can state $S$ be ever reached? Is state $T$ always reached after $S$?)\n\t\\item Classical planning\n\t\\item Combinatorial design (existence of mathematical structures)\n\t\\item Solving subproblems in domains like scheduling, test pattern generation, multi-agent systems, ...\n\\end{itemize}"
  },
  {
    "path": "Knowledge_Representation/kr_summary.tex",
    "content": "\\documentclass[a4paper]{article} \n\\addtolength{\\hoffset}{-2.25cm}\n\\addtolength{\\textwidth}{4.5cm}\n\\addtolength{\\voffset}{-3.25cm}\n\\addtolength{\\textheight}{5cm}\n\\setlength{\\parskip}{0pt}\n\\setlength{\\parindent}{0in}\n\n\\usepackage{blindtext} % Package to generate dummy text\n\\usepackage{charter} % Use the Charter font\n\\usepackage[utf8]{inputenc} % Use UTF-8 encoding\n\\usepackage{microtype} % Slightly tweak font spacing for aesthetics\n\\usepackage[english]{babel} % Language hyphenation and typographical rules\n\\usepackage{amsthm, amsmath, amssymb} % Mathematical typesetting\n\\usepackage{float} % Improved interface for floating objects\n\\usepackage[final, colorlinks = true, \nlinkcolor = black, \ncitecolor = black]{hyperref} % For hyperlinks in the PDF\n\\usepackage{graphicx, multicol} % Enhanced support for graphics\n\\usepackage{xcolor} % Driver-independent color extensions\n\\usepackage{marvosym, wasysym} % More symbols\n\\usepackage{rotating} % Rotation tools\n\\usepackage{subcaption}\n\\usepackage{wrapfig}\n\\usepackage{censor} % Facilities for controlling restricted text\n\\newcommand{\\note}[1]{\\marginpar{\\scriptsize \\textcolor{red}{#1}}} % Enables comments in red on margin\n\\usepackage{bm}\n\\usepackage{blkarray}\n\\usepackage{enumitem}\n\\usepackage{pgfplots}\n\\usepackage{tikz}\n\\definecolor{colkeyword}{rgb}{0,0.4,0}\n\\definecolor{colname}{rgb}{0.4,0.4,0}\n\\definecolor{coltype}{rgb}{0.4,0,0.4}\n\\definecolor{coloperators}{rgb}{0,0,1.0}\n\\definecolor{colscopes}{rgb}{0.4,0,0}\n\n\\setcounter{tocdepth}{2}\n% Title Page\n\\title{Summary Knowledge Representation}\n\\author{Phillip Lippe}\n\n\n\\begin{document}\n\\maketitle\n\\tableofcontents\n\\newpage\n\n\\input{kr_intro.tex}\n\\input{kr_sat.tex}\n\\input{kr_dl.tex}\n\\input{kr_csp.tex}\n\\input{kr_qr.tex}\n\\appendix\n% \\newpage\n% \\input{kr_appendix.tex}\n\\end{document}"
  },
  {
    "path": "LICENSE",
    "content": "MIT License\n\nCopyright (c) 2022 Phillip Lippe\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
  },
  {
    "path": "ML4QS/mlqs_clustering.tex",
    "content": "\\section{Clustering}\n\\begin{itemize}\n\t\\item Using the features engineered before to cluster instances\n\t\\item Two different learning setups\n\t\\begin{itemize}\n\t\t\\item \\textit{Per instance}: cluster data points/instances of quantified selfs. If multiple quantified selfs are available, we concatenate the datasets.\n\t\t\\item \\textit{Per person}: cluster different quantified selfs. Here, we consider all recorded instances of a person as a single data point, and we compare datasets/persons and cluster them. \n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Distance Metrics}\n\\begin{itemize}\n\t\\item We need different distance metrics per scenario\n\t\\item We have to distinguish between feature-level and dataset-level\n\\end{itemize}\n\\subsubsection{Feature-level distance metrics}\n\\begin{itemize}\n\t\\item For \\textit{numerical} features, we can use the minkowski distance $\\left(\\sum_k \\left|x_i^k - x_j^k\\right|^q\\right)^{1/q}$ which subsumes the Euclidean and Manhatten ($q=1$). However, we need to consider the scaling of the features (assumed to be equal)\n\t\\item For \\textit{categorical} features, we can use the Gower's similarity\n\t\\begin{itemize}\n\t\t\\item For binary attributes (called \\textit{dichotomous}) $s(x_i^k, x_j^k)=1$ if $x_i^k$ and $x_j^k$ are present (i.e. are 1), else 0\n\t\t\\item For categorical, we have $s(x_i^k, x_j^k)=\\mathbbm{1}(x_i^k = x_j^k)$\n\t\t\\item For numerical values in a range $R$, the Gower's similarity is $s(x_i^k, x_j^k)=1 - \\frac{|x_i^k - x_j^k|}{R}$\n\t\t\\item Similarity over multiple attributes is the mean of them\n\t\t\\item Note that this is a similarity and not a distance (correlated by $\\text{Similarity}\\sim1/\\text{Distance}$)\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Dataset-level distance metrics}\n\\begin{itemize}\n\t\\item We have to distinguish between datasets with and without temporal ordering\n\t\\item \\textbf{Non-temporal personal level distance metrics}: three different approaches possible\n\t\\begin{enumerate}\n\t\t\\item Summarize values per attribute over the entire dataset into a single number, as e.g. take the mean, min, max, stddev, etc. On these, we can use the same distance metrics as before\n\t\t\\item Estimate parameters of a distribution that describes the dataset, such as a normal distribution with $\\mathcal{N}(\\mu, \\sigma^2)$. On the parameters $\\mu, \\sigma^2$ we can apply the same distance metrics as before\n\t\t\\item Compare the distributions of values for an attribute with a statistical test, such as the Kolmogorov Smirnov test. The distance metric would be $1-p$ where $p$ is the $p$-value returned by the test.\n\t\\end{enumerate}\n\t\\item \\textbf{Temporal personal level distance metrics}: again, three different approaches\n\t\\begin{enumerate}\n\t\t\\item \\textit{Feature-based}: extract features from the two time series, such as those from Section~\\ref{sec:chapter_4_feature_engineering} (time and frequency domain).\n\t\t\\item \\textit{Model-based}: we try to fit a model on the two time series, and use those parameters to compare them. For example, we could use dynamical systems or similar\n\t\t\\item \\textit{Raw-data based} uses a distance per point.\n\t\t\\begin{itemize}\n\t\t\t\\item For example, it can assume a equal number of points in both datasets, and just takes e.g. the Euclidean Distance per time point\n\t\t\t\\item Alternatively, we can also take a possible lag into account (shifted dataset). Then we compute the cross correlation coefficient $ccc(\\tau, x_{qs_i}^{l}, x_{qs_j}^{l})=\\sum_{k=-\\infty}^{\\infty} x_{k, qs_i}^{l} \\cdot x_{k+\\tau, qs_j}^{l}$. \n\t\t\t\\item Optimize $\\tau$ by $\\arg\\min_{\\tau} \\sum_{k=1}^{p} \\frac{1}{ccc\\left(\\tau, x_{qs_i}^{l}, x_{qs_j}^{l}\\right)}$. Note that we have a single $\\tau$ for all attributes\n\t\t\t\\item \\textbf{Dynamic Time Warping}: make best pairs of instances in the sequence to find minimum distance. Allows different frequencies of activities\n\t\t\t\\begin{itemize}\n\t\t\t\t\\item Two conditions for pairing: \\\\\n\t\t\t\t\\textit{Monoticity condition}: time order has to be preserved. We cannot go ``back'' in time\\\\\n\t\t\t\t\\textit{Boundary condition}: the first and last point must be aligned of the two time series\n\t\t\t\t\\item Algorithm similar to finding shortest path in a graph.\n\t\t\t\t\\begin{figure}[ht!]\n\t\t\t\t\t\\centering\n\t\t\t\t\t\\includegraphics[width=0.5\\textwidth]{figures/chapter_5_dynamic_time_warping.png}\n\t\t\t\t\t\\caption{Algorithm of dynamic time warping}\n\t\t\t\t\\end{figure}\n\t\t\t\t\\item The drawbacks of this methods are that it is computational expensive ($\\mathcal{O}(N\\cdot M)$), and that the distance metric might not always be the best fit (i.e. should it be allowed to align the first point of sequence 1 with the last point of sequence 2?)\n\t\t\t\t\\item \\comment{For this algorithm, it helps more to practice it several times instead of writing it down in all details.}\n\t\t\t\\end{itemize}\n\t\t\\end{itemize} \n\t\\end{enumerate}\n\\end{itemize}\n\\subsection{Clustering approaches}\n\\begin{itemize}\n\t\\item Overview of different clustering approaches\n\t\\item \\textbf{K-means}: define $k$ cluster means. A point is assigned to the cluster to which mean it is the closest. Means are updated by the assigned points.\n\t\\begin{itemize}\n\t\t\\item Using Silhoutte score to select best $k$/determine whether a clustering is good:\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\text{silhoutte} = \\frac{\\sum_{i=1}^{N}\\frac{b(x_i) - a(x_i)}{\\max(a(x_i), b(x_i)}}{N}\\hspace{3mm} & \\text{where}\\hspace{2mm} a(x_i)=\\frac{\\sum_{\\forall x_j \\in C_l} d(x_i, x_j)}{|C_l|} \\hspace{3mm} \\text{(} x_i\\in C_l\\text{)}\\\\\n\t\t\t\t& \\hspace{2mm}\\text{and} \\hspace{4mm} b(x_i) = \\min_{C_m \\neq C_l}\\frac{\\sum_{\\forall x_j \\in C_m} d(x_i, x_j)}{|C_m|}\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tIn text, the silhoutte score compares the average distance of a point with others within a cluster ($a(x_i)$) with the distance of points with the next closest cluster ($b(x_i)$). The larger the score, the better (between$[-1,1]$)\n\t\\end{itemize}\n\t\\item \\textbf{K-medoids}: very similar to $k$-means, but we use actual points as cluster centers instead of artificial ones\n\t\\begin{itemize}\n\t\t\\item Choose new cluster means as the point with the minimum distance to all other points in the cluster\n\t\t\\item More suitable if certain points in search space might not make sense\n\t\t\\item For example, k-medoids is known to work better for person-level clustering\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Hierarchical clustering}\n\\begin{itemize}\n\t\\item Perform clustering in an iterative approach\n\t\\item \\textbf{Divisive clustering}: start with one cluster with all points in it, and in each step, perform one split\n\t\\begin{itemize}\n\t\t\\item Define the dissimilarity of a point to all other points in a cluster as the average distance $$\\text{dissimilarity}(x_i, C)=\\frac{\\sum_{x_j\\neq x_i \\in C} \\text{distance}(x_i, x_j)}{|C|}$$\n\t\t\\item When creating a new cluster $C'$, we add the most dissimilar points (in order of dissimilarity) until a point is more dissimilar to the points in $C'$ than points in $C$. \n\t\t\\item If we have multiple clusters, we choose the cluster to split which has the greatest distance between any points in the cluster\n\t\\end{itemize}\n\t\\item \\textbf{Agglomerative clustering}: start with all points in separate clusters, and merge them step by step. Merge decision can be based on different criteria (equal to distance metric between clusters):\n\t\\begin{itemize}\n\t\t\\item \\textit{Single linkage}: merge the two clusters with the minimum distance between any two points\n\t\t$$d_{SL}(C_k, C_l) = \\min\\limits_{x_i \\in C_k, x_j \\in C_l} \\text{distance}(x_i, x_j)$$\n\t\t\\item \\textit{Complete linkage}: merge the two clusters where the maximum distance between any two points is minimal\n\t\t$$d_{SL}(C_k, C_l) = \\max\\limits_{x_i \\in C_k, x_j \\in C_l} \\text{distance}(x_i, x_j)$$\n\t\t\\item \\textit{Group average}: merge the two clusters with the average distance between all points is minimal\n\t\t$$d_{SL}(C_k, C_l) = \\frac{\\sum\\limits_{x_i \\in C_k, x_j \\in C_l} \\text{distance}(x_i, x_j)}{|C_k|\\cdot |C_l|}$$\n\t\t\\item \\textit{Ward's criterion}: merge the two clusters where the increase of standard deviation by the combined cluster is minimal\n\t\t$$d_{SL}(C_k, C_l) = \\sigma^2_{C_k \\cup C_l} - \\left(\\sigma^2_{C_l} + \\sigma^2_{C_k}\\right)$$\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Subspace clustering}\n\\begin{itemize}\n\t\\item The problem of standard clustering algorithms for a huge number of features is that the distance between two points get uninformative (small distance in all features compared to big difference in only one feature). Hence, the clusters get less meaningful as well\n\t\\item Better approach is therefore to look at subspaces in the feature space.\n\t\\item Pseudo algorithm:\n\t\\begin{enumerate}\n\t\t\\item For all features, define intervals of the feature space. This leads to units $u={u_1, ..., u_p}$ where $u_i(l)$ is the lower-bound for attribute $i$ in this unit, and $u_h(l)$ the upper bound respectively. Note that units do not require to cover all features, but can look at subspaces only (same to setting lower bound to $-\\infty$ and upper to $\\infty$). \n\t\t\\item Determine selectivity of a unit as proportion of points in them. We call a unit ``dense'' if it contains more points/higher proportion than a certain threshold (hyperparameter).\n\t\t\\item Units are connected to a cluster if they \n\t\t\\begin{itemize}\n\t\t\t\\item share a common face which is defined as having the lower bound equals to an upper bound of another unit (or other way round), and the same upper and lower bound for all other attributes. \n\t\t\t\\item or when they share a unit to which they both have a common face.\n\t\t\\end{itemize}\n\t\\end{enumerate}\n\t\\item In the end, we strive to find dense units using a combination of attributes to form clusters\n\t\\item To reduce number of attributes, we can start to make units with a single attribute, and add more attributes iteratively based on some \\textit{fancy} algorithm. After having found the units, we can create the clusters\n\\end{itemize}"
  },
  {
    "path": "ML4QS/mlqs_feature_engineering.tex",
    "content": "\\section{Feature Engineering}\n\\label{sec:chapter_4_feature_engineering}\n\\begin{itemize}\n\t\\item Create useful features from temporal data\n\\end{itemize}\n\\subsection{Time Domain}\n\\begin{itemize}\n\t\\item Summarize values of a certain attributes in a window size of $\\lambda$ steps before. If we would take the time steps $t$ and $t-1$ into account, our value for $\\lambda$ is $1$\n\t\\item Note that we cannot compute any values for the time steps $t=1,..,\\lambda$.\n\\end{itemize}\n\\subsubsection{Numerical}\n\\begin{itemize}\n\t\\item Aggregate values by mean, min, max, stddev, etc. (including current time step)\n\t\\item We could also use coefficient between first and last value interpolation (gradient of attribute)\n\\end{itemize}\n\\subsubsection{Categorical}\n\\begin{itemize}\n\t\\item Generate patterns of occurrences of categorical values. We distinguish between successive \\texttt{(b)} and co-occurring \\texttt{(c)} actions/classes. Example patterns:\n\t\\begin{itemize}\n\t\t\\item Activity level = high \\texttt{(c)} Activity = running\n\t\t\\item Activity = running \\texttt{(b)} Activity = running\n\t\\end{itemize}\n\t\\item If we have a window size of $\\lambda$, we just see if there is any time step within $t-\\lambda, ..., t-1, t$ where the activities are co-occuring \\texttt{(c)}, or there is one activity happening (arbitrary number of time steps) before another \\texttt{(b)}.\n\t\\item We can find important patterns by determining the support of such. This is important as the number of patterns exponentially increases with the number of categories/attributes, and only frequent patterns are interesting.\n\t\\item The support of a pattern is defined as the proportion of the processed time steps at which this pattern would occur.\n\t\\item The algorithm for finding such patterns starts with single attribute patterns, and extends only those which have sufficient support. In the end, we add all patterns (including the single-attribute) with enough support.\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/chapter_4_pattern_identification.png}\n\t\t\\caption{Pattern identification algorithm}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{Mixed data}\n\\begin{itemize}\n\t\\item We can also combine numerical and categorical attributes for features\n\t\\item Hereby, we create categories from numerical data by looking at them \\textbf{qualitatively} (greetings from QR). For example, we can define ranges like low, medium and high, or look at the trend/gradients as \\textit{increasing} or \\textit{decreasing}\n\t\\item Afterwards, we can apply the categorical approach on those\n\\end{itemize}\n\\subsection{Frequency Domain}\n\\begin{itemize}\n\t\\item Apply Fourier transformation on data within a window of $\\lambda$ (plus the current time point $t$) to extract periodicity of the data\n\t\\item Assume a base frequency of $f_0 = \\frac{2\\pi}{\\lambda+1}$ (or $f_0 = \\frac{N_{\\text{sec}}}{\\lambda+1}$ in seconds) which is the lowest frequency with a complete sinusoid in it. \n\t\\item We look at all the frequencies $\\left\\{0\\cdot f_0, 1\\cdot f_0, ..., \\lambda \\cdot f_0\\right\\}$ and determine the corresponding amplitudes\n\t\\item Our features can be:\n\t\\begin{itemize}\n\t\t\\item Frequency with highest amplitude\n\t\t\\item Frequency-weighted signal average $\\frac{\\sum_{k=0}^{\\lambda} a_{t-\\lambda}^{t}(k) \\cdot f(k)}{\\sum_{k=0}^{\\lambda} a_{t-\\lambda}^{t}(k)}$\n\t\t\\item \\textit{Power Spectrum Entropy}: Amount of information in the signal\n\t\t$$x\\_pse = - \\sum_{k=0}^{\\lambda} p_{t-\\lambda}^{t}(k) \\ln p_{t-\\lambda}^{t}(k), \\hspace{3mm}\\text{with}\\hspace{2mm} p_{t-\\lambda}^{t}(k) = \\frac{|a_{t-\\lambda}^{t}(k)|^2}{\\sum_{i=0}^{\\lambda} |a_{t-\\lambda}^{t}(i)|^2}$$\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Unstructured data}\n\\begin{itemize}\n\t\\item How to handle non-temporal/unstructured data like text, audio, images, etc.\n\t\\item Here we focus on text. The standard pipeline contains for steps:\n\t\\begin{itemize}\n\t\t\\item \\textit{Tokenization}: split sentence into smallest parts \n\t\t\\item \\textit{Lower case}: put all words to lower case to have no difference in such\n\t\t\\item \\textit{Stemming}: reduce words to their stem to remove all small variations (like tense, etc.)\n\t\t\\item \\textit{Stop word removal}: remove known, uninformative stop words \n\t\\end{itemize}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/chapter_4_text_pipeline.png}\n\t\t\\caption{Pipeline of text processing}\n\t\\end{figure}\n\t\\item Three approaches in general\n\t\\begin{itemize}\n\t\t\\item \\textbf{Bag of words}: count occurrences of n-gram within text. These counts are the features for the text\n\t\t\\item \\textbf{TF-IDF}: BoW does not take ``uniqueness'' of word into account. Thus, TF-IDF takes occurrences of a word in a text and in the whole corpus into account\n\t\t\\item \\textbf{Topic modeling}: assume that any text is a combination of $k$ topics. Perform LDA to get these topics, and topic distribution of text is its features.\n\t\\end{itemize}\n\\end{itemize}"
  },
  {
    "path": "ML4QS/mlqs_intro.tex",
    "content": "\\section{Introduction}\n\\label{sec:chapter_1_2_introduction}\n\\subsection{Definitions}\n\\begin{itemize}\n\t\\item The quantified self can be defined as:\n\t\\blockquote{\\textit{The quantified self is any individual engaged in the self-tracking of any kind of biological, physical, behavioral, or environmental information. \\\\The self-tracking is driven by a certain goal of the individual with a desire to act upon the collected information.}}\n\t\\item \\textbf{Augemberg} (2012): there are various types of measurements, including:\n\t\\begin{itemize}\n\t\t\\item Physical activities (movement by accelerometer, steps, etc.)\n\t\t\\item Diet (calories consumed, fat, protein, etc.)\n\t\t\\item Psychological states (mood, emotions, depression, etc.)\n\t\t\\item Mental and cognitive traits (IQ, reaction, memory, etc.)\n\t\t\\item Environmental (location, weather, etc.)\n\t\t\\item Situational (time of the day, context, etc.)\n\t\t\\item Social variables (influence, role in a group, etc.)\n\t\\end{itemize} \n\t\\item \\textbf{Choe} (2014): distinguish quantified selves into three categories based on their goal.\n\t\\begin{itemize}\n\t\t\\item Improved health (cure or manage a condition,\n\t\texecute a treatment plan, achieve a goal)\n\t\t\\item Improve other aspects of life (maximize work performance, be mindful)\n\t\t\\item Find new life experiences (have fun, learn new things)\t\n\t\\end{itemize}\n\t\\item \\textbf{Gimpel} (2013): identified five (non-exclusive) factors for quantified self motivation\n\t\\begin{itemize}\n\t\t\\item Self-healing (to become healthier)\n\t\t\\item Self-discipline (to experience rewarding aspects of it)\n\t\t\\item Self-design (to control and optimize ``yourself'', as e.g. on sport)\n\t\t\\item Self-association (to be associated with the movement of QS)\n\t\t\\item Self-entertainment (to experience entertainment value)\n\t\\end{itemize}\n\t\\item Machine Learning (automatically identifying patterns from data) is slightly different in the setting of Quantified Self because\n\t\\begin{itemize}\n\t\t\\item we have to deal with sensory noise\n\t\t\\item there might be missing measurements\n\t\t\\item we have temporal data (feature engineering) with irregular time points\n\t\t\\item there can be an interaction with a user (advice for training/mood improvements, etc.), but we cannot try out every possibility\n\t\t\\item Learn across multiple datasets/users\n\t\\end{itemize}\n\t\\item \\comment{Most of the above definitions need to be memorized for the exam}\n\\end{itemize}\n\\subsection{Basic Terminology and Notation}\n\\begin{itemize}\n\t\\item Measurement $=$ one value for an attribute recorded at a specific time point\n\t\\item Time series $=$ series of measurements in temporal order\n\t\\item Further notation: \n\t\\begin{itemize}\n\t\t\\item For matrix $\\bm{X}$, the columns are the different measurements like accelerometer, and rows are the time points (if dataset is temporally ordered, otherwise random list) \n\t\\end{itemize}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/intro_notation.png}\n\t\t\\caption{Overview of notation used in this course.}\n\t\\end{figure}\n\\end{itemize}\n\\subsection{Basic overview of Sensory Data}\n\\begin{itemize}\n\t\\item Different sensors available on mobile devices, such as:\n\t\\begin{itemize}\n\t\t\\item \\textit{Accelerometer}: Measures the changes in forces upon the phone in the $x$-$y$-$z$ plane\n\t\t\\item \\textit{Gyroscope}: Orientation of the phone compared to\n\t\tthe earth's surface\n\t\t\\item \\textit{Magnetometer}: Measures $x$-$y$-$z$ orientation compared to the earth's magnetic field\n\t\\end{itemize}\n\t\\item Transforming raw data of time series require selecting a step size $\\Delta t$, and combine sensory data over this interval. See Section~\\ref{sec:chapter_4_feature_engineering} for techniques\n\t\\item A large $\\Delta t$ gives (maybe too) coarse-grained data, but we have in the end a smaller dataset and lower standard deviation. The opposite is gained by a smaller $\\Delta t$ (fine-grained data, but large dataset and high stddev)\n\\end{itemize}"
  },
  {
    "path": "ML4QS/mlqs_modeling_with_time.tex",
    "content": "\\section{Predictive Modeling with Notion of Time}\n\\subsection{Time Series}\n\\begin{itemize}\n\t\\item Understanding the periodicity and trends in data in the time domain. Can be used for forecasting or control (e.g. how can we influence the trend)\n\t\\item Is build up by three components:\n\t\\begin{itemize}\n\t\t\\item \\textit{Seasonality/Periodicity}: any periodic/repeating pattern over any frequency (e.g. seconds, hours or days)\n\t\t\\item \\textit{Trend}: how the mean evolves over time\n\t\t\\item \\textit{Irregular variations}: noise, everything left after we remove periodicity and trend\n\t\\end{itemize}\n\t\\item \\textbf{Stationarity}: assumption/requirement for many algorithms applied on time series\n\t\\begin{itemize}\n\t\t\\item In general, stationarity means that the statistical properties of a process generating a time series do not change over time.\n\t\t\\item We call a time series stationary if trends and periodicity are removed (mean is constant), and the variance of the remaining irregular variations is constant over time\n\t\t\\item Additionally, the lagged auto correlation should be constant over time/lags $\\lambda$ and close to $0$:\n\t\t$$r_{\\lambda} = \\frac{\\sum\\limits_{t=1}^{N-\\lambda} (x_t - \\bar{x}) (x_{t+\\lambda} - \\bar{x})}{\\sum\\limits_{t=1}^{N}(x_t - \\bar{x})^2}$$\n\t\tIt can provides clues of underlying pattern in the data, which should not be the case for stationary ones.\n\t\t\\item If a time series is not stationary, we can mostly transform it to one by removing trend, periodicity, and try to stabilize the variance\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Filtering and Smoothing}\n\\begin{itemize}\n\t\\item To determine the trend of a signal, one of the simplest approaches is filtering and/or smoothing\n\t\\item Simplest filtering is for a window size of $\\pm q$ (creates a new, filtered time series):\n\t$$z_t = \\sum\\limits_{r=-q}^{q} a_r x_{t+r}$$\n\t\\item \\textbf{Differencing}\n\t\\begin{itemize}\n\t\t\\item For removing a trend, the most effective technique is differencing (or gradient filter): $z_t = x_t - x_{t-1} = \\nabla x_t$. As we expect the trend to be low-frequent, the gradient is therefore small. % Periodicity is not influenced by this filter (differentiating $\\sin$ just shifts it in time). % Not 100% true because if we have \\sin(0.1*x), we get 0.1 * \\cos(0.1 * x). Hence, 10 times smaller\n\t\t\\item We can also apply this operator multiple times, leading to a $d$-th order differencing ($\\nabla^d x_t$). For $d=2$, we get $z_t = \\nabla^2 x_t = x_t - 2x_{t-1}+x_{t-2}$. A $d$-th order differencing can remove trends than can be approximated by a polynomial of order $d$ or lower.\n\t\t\\item Drawback of differencing is that the variance of the remaining time series increases. Can be improved by using a better approximation of trend than $x_{t-1}$, as for example a exponential filtered signal $e_{t} = \\sum\\limits_{r=-q}^{0} \\frac{\\alpha(1 - \\alpha)^{|r|}}{2 - \\alpha} x_{t+r}$ (with e.g. $\\alpha=0.05$), and use that for the differencing: $z_t = x_t - e_{t}$\n\t\t\\item Still, we have to be careful as we might remove low-frequent periodicity as well ($\\partial \\sin(0.1\\cdot x)/\\partial x = 0.1 \\cdot \\cos(0.1 \\cdot x)$ $\\Rightarrow$ dampen signal by factor of 10). \n\t\t\\item If we would want to remove periodicity, we could apply the differencing operator not on two adjacent points, but two points that are moved by 1 period as the difference between those is expected to be zero (note that we need to know/estimate the frequency for that)\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{ARIMA}\n\\begin{itemize}\n\t\\item ``Autoregressive Integrated Moving Average Model''\n\t\\item We try to estimate a model that describes the empirical data well and can be used to forecast/predict new values\n\t\\item For this, we learn/determine a mapping of time point $t$ to probability distribution $P_t$\n\t\\item In ARIMA, we assume $P_t$ to be modeled by a combination of a autoregressive process (AR), and a moving average (MA) over white noise $W_t$:\n\t$$P_t = \\underbrace{\\phi_1 P_{t-1} + ... \\phi_p P_{t-p} + W_t}_{\\text{Autoregressive process}} + \\underbrace{\\theta_1 W_{t-1} + ... \\theta_q W_{t-q}}_{\\text{Moving Average}}$$\n\tNote that an AR can be expressed by an infinite MA, and the other way round. But to reduce the number of parameters, both concepts are used here.\n\t\\item To remove the drifts of mean (trend), we apply differencing of order $d$ on $P_t$ ($V_t = \\nabla^d P_t$). \n\t\\item Optimization of parameters\n\t\\begin{itemize}\n\t\t\\item For $p$ we can look at the autocorrelation between $x_t$ and $x_{t-p}$ to find patterns\n\t\t\\item For other parameters, gridsearch with objective function as the fit to the data we have\n\t\\end{itemize}\n\t\\item Note that ARIMA does not take seasonality/periodicity into account. Hence, we either have to add it externally, remove it beforehand or model it as well (ARIMAX)\n\\end{itemize}\n\\subsection{Recurrent Neural Networks}\n\\begin{itemize}\n\t\\item Unfolding for gradient calculation\n\t\\item \\textbf{Echo State Networks}: ``cheap'' RNNs without backprop through time\n\t\\begin{itemize}\n\t\t\\item Three weight matrices:\n\t\t\\begin{itemize}\n\t\t\t\\item $\\bm{W}^{\\textbf{in}}$: are the weights from the input layer to the memory (or here also called \\textit{reservoir}). \n\t\t\t\\item $\\bm{W}$: are the weights over time steps (or internally in the reservoir).\n\t\t\t\\item $\\bm{W}^{\\textbf{out}}$: specify the weights from the reservoir to the output\n\t\t\\end{itemize}\n\t\t\\item $\\bm{W}^{\\textbf{in}}$ and $\\bm{W}$ are randomly initialized and \\textbf{fixed} during training ,while $\\bm{W}^{\\textbf{out}}$ is learned (either by SGD or pseudo inverse)\n\t\t\\item Initialization of $\\bm{W}$ need to satisfy the \\textit{Echo State property} which state that the effect of a previous state should gradually decrease over time (prevent exploding values)\n\t\t\\item We can ensure this by randomly initializing a matrix, dividing by its spectral radius and (optionally) scale it down even further.\n\t\t\\item There are different initialization heuristics to optimize this process, but all underly the \\textit{No Free Lunch} theorem (optimizing for one use case will make it worse for others)\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Dynamical Systems}\n\\begin{itemize}\n\t\\item Build domain knowledge-based models that cover temporal relationships between attributes by the meas of differential equations. Furthermore, they assume a numerical state. Very simple model for velocity:\n\t$$y_{vel}(t) = y_{vel}(t-1) + \\gamma_1 \\cdot y_{acc}(t)$$\n\t\\item Models still contain parameters (as $\\gamma_1$ above) that can/need to be tuned\n\t\\item \\textbf{Parameter optimization}: three main approaches\n\t\\begin{itemize}\n\t\t\\item \\textit{Simulated Annealing}: similar to an EA with a population size of 1. \n\t\t\\begin{itemize}\n\t\t\t\\item We have a single solution which we randomly initialize first. At each iteration, we take a random step, and compare the difference in score. \n\t\t\t\\item If the new point is better than the old one, we replace it. Otherwise, we replace it with a probability based on the distance between the scores, and the number of steps we have already taken. \n\t\t\t\\item Note that the probability decreases with number of steps. Thus, we switch from exploration in the first iterations to exploitation in the last.\n\t\t\\end{itemize}\n\t\t\\item \\textit{Genetic Algorithms}: \n\t\t\\begin{itemize}\n\t\t\t\\item We represent parameter values as a bit string (arbitrary number of bits per parameter, all together concatenated into one genotype), and initialize a couple of them in our population \n\t\t\t\\item At each iteration, we choose a set of parents from our population (based on fitness value), and perform crossover as well as mutation on the children\n\t\t\t\\item Perform survivor selection, or just completely replace the old generation by the new one\n\t\t\\end{itemize}\n\t\t\\item \\textit{NSGA-II}: multi-criteria optimization GA\n\t\t\\begin{itemize}\n\t\t\t\\item ``Non-Dominated Sorting Genetic Algorithm''\n\t\t\t\\item Used when multiple targets need to be optimized, and there is no fixed tradeoff between both \n\t\t\t\\item Therefore, we find Pareto fronts in our population (individuals that are not Pareto dominated by any other individual in our population)\n\t\t\t\\item We create several Pareto fronts by iteratively creating one for our population, and then remove all individuals on it from the population, and start again\n\t\t\t\\item Interested in a wide spread of individuals/coverage of Pareto front $\\Rightarrow$ weight individuals on the Pareto front by the distance to other points (points on the border set to infinity because they are the best for a certain objective). We use this weight for survivor selection where we iteratively add individuals until we have enough for a new population. Note that we of course prioritize the individuals that are on a earlier Pareto front\n\t\t\\end{itemize}\n\t\\end{itemize}\n\\end{itemize}"
  },
  {
    "path": "ML4QS/mlqs_modeling_without_time.tex",
    "content": "\\section{Predictive Modeling without Notion of Time}\n\\begin{itemize}\n\t\\item Any predictor that does not explicitly take time into account (except the temporal features from time/frequency domain etc.)\n\t\\item Different learning setups possible\n\t\\begin{itemize}\n\t\t\\item \\textit{Individual}: train and test on a single user\n\t\t\\item \\textit{Population - unknown user}: train on a set of users, test on a different set of users\n\t\t\\item \\textit{Population - unseen data}: train on a set of users, test on the same users but different data\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Preventing overfitting}\n\\begin{itemize}\n\t\\item One big issue in the context of QS is that algorithms can easily overfit. This can be due to the big amount of features, the noise contained in them, and the usually small datasets we have\n\t\\item \\textbf{Feature selection}\n\t\\begin{itemize}\n\t\t\\item To prevent the models to overfit on not useful features, we can reduce the number of features to the essential ones\n\t\t\\item \\textit{Forward selection}: start with empty set, and iteratively add most predictive feature. At every iteration, we need to run a model on the previously added features plus any of the other features left. Stop when accuracy does not improve significantly anymore\n\t\t\\item \\textit{Backward selection}: start with all features, and iteratively remove the least predictive feature. Similar to forward selection, just doing the whole algorithm reversed.\n\t\\end{itemize}\n\t\\item \\textbf{Regularization}: add a term to the error function to punish more for more complex models. Examples include L1/L2 regularization for NN, points per leaf for decision trees, etc.\n\\end{itemize}"
  },
  {
    "path": "ML4QS/mlqs_reinforcement_learning.tex",
    "content": "\\section{Reinforcement Learning}\n\\begin{itemize}\n\t\\item RL for ML4QS to learn from interactions with user and influencing him\n\t\\item General overview of how to integrate RL in ML4QS is shown in Figure~\\ref{fig:chapter_9_RL_loop}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/chapter_9_RL_loop.png}\n\t\t\\caption{Reinforcement Learning in the loop for ML4QS}\n\t\t\\label{fig:chapter_9_RL_loop}\n\t\\end{figure}\n\t\\item \\textbf{Markov Property}: $\\mathbb{P}\\left\\{R_{t+1}=r, S_{t+1}=s|S_0, A_0, R_0, ..., S_t, A_t, R_t\\right\\} = \\mathbb{P}\\left\\{R_{t+1}=r, S_{t+1}=s|S_t, A_t\\right\\}$\\\\\n\tThe conditional probabilities of future state and rewards solely depend on the last state $S_t, A_t$. \n\t\\item For every problem that satisfies this property, we can easily create a Markov Decision Process with a finite set of states as in Figure~\\ref{fig:chapter_9_MDP}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.35\\textwidth]{figures/chapter_9_MDP.png}\n\t\t\\caption{Markov Decision Process for simple example}\n\t\t\\label{fig:chapter_9_MDP}\n\t\\end{figure}\n\t\\item \\textbf{SARSA}:\n\t\\begin{itemize}\n\t\t\\item On-policy optimization, update $Q$-values by:\n\t\t$$Q(S_t, A_t) \\leftarrow Q(S_t, A_t) + \\alpha(R_{t+1} + \\gamma Q(S_{t+1}, A_{t+1}) - Q(S_t, A_t))$$\n\t\t\\item Popular policies are $\\epsilon$-greedy or softmax over q-values for different actions\n\t\\end{itemize}\n\t\\item \\textbf{Q-Learning}:\n\t\\begin{itemize}\n\t\t\\item Off-policy optimization, update by taking the maximum Q-value over next state:\n\t\t$$Q(S_t, A_t) \\leftarrow Q(S_t, A_t) + \\alpha(R_{t+1} + \\gamma \\max\\limits_{A'(S_{t+1})} Q(S_{t+1}, A') - Q(S_t, A_t))$$\n\t\\end{itemize}\n\t\\item \\textbf{Eligibility traces}: update frequently seen states in a single run more\n\t\\begin{itemize}\n\t\t\\item If we have seen a state and action combination more frequently in our history, then we want to increase the weight of the update because it is more eligible (i.e. more responsible for the outcome). We can determine the eligibility by:\n\t\t$$Z_t(s, a) = \\begin{cases}\n\t\t\\gamma \\lambda Z_{t-1} + 1 & \\text{if } s=S_t\\wedge a=A_t\\\\\n\t\t \\gamma \\lambda Z_{t-1} & \\text{otherwise}\n\t\t\\end{cases}$$\n\t\t\\item In our learning algorithms, we can incorporate this by increasing the weight of the update, as e.g. in Q-Learning:\n\t\t$$Q(S_t, A_t) \\leftarrow Q(S_t, A_t) + \\alpha(R_{t+1} + \\gamma \\max\\limits_{A'(S_{t+1})} Q(S_{t+1}, A') - Q(S_t, A_t))\\cdot Z_t(s,a)$$\n\t\\end{itemize}\n\t\\item Usually, the Q-values are stored in a table. If the number of states and actions are very large, this is not feasible. Alternative is to learn a function/model, that takes as input the state and action, and predicts the Q-value.\n\t\\item For continuous state spaces, we can discretize it by e.g. the \\textbf{U-tree} algorithm\n\t\\begin{itemize}\n\t\t\\item Start with a single unit/leaf/discrete state where all continuous states are mapped to \n\t\t\\item Collect data by trial and error for a while and estimate the Q-values\n\t\t\\item On the collected data for each leaf, we test whether we can find splits for any attribute $X_i$ with a significant difference in Q-values\n\t\t\\item Choose $X_i$ and its split with the lowest $p$-value, and create new leafs. Continue until maximum number of leafs is reached\n\t\\end{itemize}\n\\end{itemize}"
  },
  {
    "path": "ML4QS/mlqs_sensory_noise.tex",
    "content": "\\section{Handling Sensory Noise}\n\\label{sec:chapter_3_sensory_noise}\n\\subsection{Outlier Detection}\n\\begin{itemize}\n\t\\item ``\\textit{An outlier is an observation point that is distant from other observations}''\n\t\\item Outliers can be caused by measurement errors, or variability of the data (e.g. very high heart rate due to pushing someone's limits)\n\t\\item Outliers can be detected by either \\textit{domain knowledge} (known in what range to expect value, e.g. heart rate should not be over 220), or without by filtering noise. We distinguish between two ways for doing so:\n\t\\begin{itemize}\n\t\t\\item \\textit{Distribution-based}: assume a certain distribution of the data, and remove all points with a likelihood lower than a certain threshold\n\t\t\\item \\textit{Distance-based}: focus on the distance between data points, and mark those as outliers which are far apart\n\t\\end{itemize}\n\t\\item After detecting the outliers, we can replace them with \\textit{unknown} values/value missing tag. \n\\end{itemize}\n\\subsubsection{Distribution-based outlier detection}\n\\begin{itemize}\n\t\\item \\textbf{Chauvenet's criterion}: assume a normal distribution for a single attribute\n\t\\begin{itemize}\n\t\t\\item We can fit a normal distribution by calculating the mean and stddev of the data\n\t\t\\item For each point, calculate the probability $P(X\\leq x_{i}^{j})=\\int_{-\\infty}^{x_{i}^{j}} \\frac{1}{\\sqrt{2\\pi}\\sigma}e^{-\\frac{(u-\\mu)^2}{2\\sigma^2}} \\partial u$\\\\ (instance $i$ from $j$th attribute)\n\t\t\\item A point is an outlier if:\n\t\t$$P(X\\leq x_{i}^{j}) < \\frac{1}{c\\cdot N} \\hspace{3mm}\\text{or}\\hspace{3mm} \\left(1-P(X\\leq x_{i}^{j})\\right) < \\frac{1}{c\\cdot N}$$\n\t\tThus, the probability of a point being an outlier decreases with the number of observations for this attribute (as the likelihood increases to observe rare values)\n\t\t\\item Mostly $c=2$ is chosen.\n\t\\end{itemize}\n\t\\item \\textbf{Mixture models}: assume the data to be described by $K$ normal distributions $p(x)=\\sum_{k=1}^{K}\\pi_k \\mathcal{N}\\left(x|\\mu_k, \\sigma_k\\right)$\n\t\\begin{itemize}\n\t\t\\item Use EM algorithm to optimize maximum-likelihood of the data\n\t\t\\item A point is considered as an outlier if it has a lower probability than a certain threshold\n\t\t\\item Both threshold and number of distributions $K$ is a hyperparameter to optimize\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Distance-based outlier detection}\n\\begin{itemize}\n\t\\item Outlier detection based on a distance metric $d(x_{i}^{j}, x_{k}^{j})$ (as e.g. Euclidean distance) which can also be across multiple attributes\n\t\\item \\textbf{Simple distance-based approach}:\n\t\\begin{itemize}\n\t\t\\item Call two points close if they are within a distance $d_{\\text{min}}$ (hyperparameter)\n\t\t\\item A point $x_{i}^{j}$ is considered as an outlier if:\n\t\t$$\\frac{\\sum_{n=1}^{N} \\mathbbm{1}\\left(d(x_{i}^{j}, x_{n}^{j}) > d_{\\text{min}}\\right)}{N} > f_{\\text{min}}$$\n\t\tHence, we look if the number of points within the range of $d_{\\text{min}}$ are at least $1-f_{\\text{min}}$.\n\t\t\\item Example values of hyperparameters: $d_{\\text{min}}=0.1, f_{\\text{min}}=0.99$\n\t\t\\item Not working if we have multi-modal distribution (does not take local densities into account)\n\t\\end{itemize}\n\t\\item \\textbf{Local outlier factor}: use local densities to determine outliers.\n\t\\begin{itemize}\n\t\t\\item Define $k_{\\text{dist}}$ for point $x_{i}$ as the maximum distance in set of its $k$ closest neighbors. \n\t\t\\item The reachability distance of $x_{i}$ \\textbf{from} $x$ is defined as:\n\t\t$$k_{\\text{reach\\_dist}}(x_i, x) = \\max\\left(k_{\\text{dist}}(x), d(x, x_i)\\right)$$\n\t\tNote that this distance is \\textit{not} symmetric as it uses the $k$th nearest neighbors of a point.\\\\\n\t\tFurthermore, this operation is only done to reduce the influence of very close-by points. \n\t\t\\item The local reachability distance of a point is defined by:\n\t\t$$k_{\\text{lrd}}\\left(x_{i}\\right) = \\frac{\\left|k_{\\text{distnh}}\\left(x_{i}\\right)\\right| }{\\sum\\limits_{x\\in k_{\\text{distnh}}\\left(x_{i}\\right)} k_{\\text{reach\\_dist}}\\left(x_i, x\\right)}$$\n\t\tHence, it is high if a point is very close to others.\n\t\t\\item Outlier if neighbor points have much higher local reachability points than actual point:\n\t\t$$k_{\\text{lof}}\\left(x_{i}\\right) = \\frac{\\sum\\limits_{x\\in k_{\\text{distnh}}\\left(x_{i}\\right)} k_{\\text{lrd}}\\left(x\\right)}{\\left|k_{\\text{distnh}}\\left(x_{i}\\right)\\right| \\cdot k_{\\text{lrd}}\\left(x_{i}\\right)}$$\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Missing value imputation}\n\\begin{itemize}\n\t\\item Due to outliers or measuring errors, we might have missing values in our dataset\n\t\\item We can use simple methods like replace it by the mean or median of the other observed data, or also use more advanced methods that take the values of the other attributes at this observation into account, or a local time window.\n\t\\begin{itemize}\n\t\t\\item Example for the latter: \\textbf{interpolation} $x_{i}^{j} = x_{i-k}^{j} + k \\cdot \\frac{x_{i+l}^{j} - x_{i-k}^{j}}{l+k}$\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Kalman Filter}\n\\begin{itemize}\n\t\\item Combine outlier detection and imputation into a single model\n\t\\item Therefore, we keep a latent state $s_t$, for which $x_t$ are the observations in this states (Kalman filter relates $x_t$ and $s_t$)\n\t\\item The next value of a state is defined as: $$s_t = F_t s_{t-1} + B_t u_t + w_t$$ where $u_t$ is a control input state (as e.g. sending a message), $w_t$ is white noise, and $F_t$ and $B_t$ are learned matrices\n\t\\item The measurements associated with $s_t$ can be predicted by: $$x_t = H_t s_t + v_t$$ where $v_t$ is again white noise.\n\t\\item We can predict the next state (without noise) by $\\hat{s}_{t|t-1} = F_t \\hat{s}_{t-1|t-1} + B_t u_t$\n\t\\item The error at time $t$ compared to the observations $x_t$ is then $e_t = x_t - H^T \\hat{s}_{t|t-1}$\n\t\\begin{itemize}\n\t\t\\item If we observe (after training/modeling) a high error, we can assume a value to be an outlier, and replace it with the prediction of the Kalman Filter.\n\t\\end{itemize}\n\t\\item Given this error, we can update our prediction accordingly: $\\hat{s}_{t|t} = \\hat{s}_{t|t-1} + K_t e_t$ where $K_t$ takes the expected prediction error into account (based on the white noise $w_t$ and $v_t$)\n\t% \\item Next, we can estimate our prediction error of $\\hat{s}_{t|t-1}$ by $P_{t|t-1}=\\mathbb{E}\\left[\\left(s_t - \\hat{s}_{t|t-1}\\right)\\left(s_t - \\hat{s}_{t|t-1}\\right)^T\\right]$\n\\end{itemize}\n\\subsection{Transforming the Data}\n\\begin{itemize}\n\t\\item Transform data to extract most useful data, and get rid of remaining noise\n\t\\item Different approaches can be used\n\t\\item \\textbf{Lowpass filter}: filter out high-frequent noise\n\t\\begin{itemize}\n\t\t\\item We assume that our signal has a certain periodicity, but we are only interested in certain parts of the frequency band (noise is mostly very high-frequent)\n\t\t\\item The low-pass filter can remove those by weighting each periodicity by its frequency:\n\t\t$$|G(f)|^2 = \\frac{1}{1+\\left(f/f_c\\right)^{2n}}$$\n\t\twith $|G(f)|$ as the magnitude, $f_c$ is the cutoff frequency (magnitude halved), and $n$ the order of the filter\n\t\\end{itemize}\n\t\\item \\textbf{Principal Component Analysis}: find components that explain most of the variance in the data\n\t\\begin{itemize}\n\t\t\\item Select number of components based on the explained variance. Other, low-variance components are removed to reduce noise \n\t\t\\item Problem: we loose insight in the data because the components are not easily interpretable anymore\n\t\\end{itemize}\n\\end{itemize}"
  },
  {
    "path": "ML4QS/mlqs_summary.tex",
    "content": "\\documentclass[a4paper]{article} \n\\addtolength{\\hoffset}{-2.25cm}\n\\addtolength{\\textwidth}{4.5cm}\n\\addtolength{\\voffset}{-3.25cm}\n\\addtolength{\\textheight}{5cm}\n\\setlength{\\parskip}{0pt}\n\\setlength{\\parindent}{0in}\n\n\\usepackage{blindtext} % Package to generate dummy text\n\\usepackage{charter} % Use the Charter font\n\\usepackage[utf8]{inputenc} % Use UTF-8 encoding\n\\usepackage{microtype} % Slightly tweak font spacing for aesthetics\n\\usepackage[english]{babel} % Language hyphenation and typographical rules\n\\usepackage{amsthm, amsmath, amssymb, amsfonts} % Mathematical typesetting\n\\usepackage{float} % Improved interface for floating objects\n\\usepackage[final, colorlinks = true, \nlinkcolor = black, \ncitecolor = black]{hyperref} % For hyperlinks in the PDF\n\\usepackage{graphicx, multicol} % Enhanced support for graphics\n\\usepackage{xcolor} % Driver-independent color extensions\n\\usepackage{marvosym, wasysym} % More symbols\n\\usepackage{rotating} % Rotation tools\n\\usepackage{subcaption}\n\\usepackage{wrapfig}\n\\usepackage{censor} % Facilities for controlling restricted text\n\\newcommand{\\note}[1]{\\marginpar{\\scriptsize \\textcolor{red}{#1}}} % Enables comments in red on margin\n\\usepackage{bm}\n\\usepackage{blkarray}\n\\usepackage{enumitem}\n\\usepackage{pgfplots}\n\\usepackage{tikz}\n\\usepackage{bbm}\n\\usepackage[autostyle]{csquotes} \n\n\\newcommand{\\pd}[2]{\\frac{\\partial #1}{\\partial #2}}\n\\newcommand{\\loss}[0]{\\mathcal{L}}\n\\newcommand{\\chain}[3]{\\frac{\\partial #1}{\\partial #2}\\frac{\\partial #2}{\\partial #3}}\n\\newcommand{\\eq}[1]{\\begin{equation*}\\begin{split}#1\\end{split}\\end{equation*}}\n\\newcommand{\\TODO}[1]{\\textbf{\\textcolor{red}{#1}}}\n\\newcommand{\\comment}[1]{\\textit{\\textcolor{blue}{Comment: #1}}}\n\n\\definecolor{green}{RGB}{0,160,0}\n\\definecolor{blue}{RGB}{0,0,160}\n\\definecolor{red}{RGB}{160,0,0}\n\\definecolor{orange}{RGB}{200,160,0}\n\\definecolor{purple}{RGB}{170,0,200}\n\\definecolor{cyan}{RGB}{0,200,200}\n\\definecolor{lightred}{RGB}{200,50,50}\n\n\\setcounter{tocdepth}{2}\n% Title Page\n\\title{Summary Machine Learning for the Quantified Self}\n\\author{Phillip Lippe}\n\n\n\\begin{document}\n\\maketitle\n\\tableofcontents\n\\newpage\n\n\\input{mlqs_intro.tex}\n\\input{mlqs_sensory_noise.tex}\n\\input{mlqs_feature_engineering.tex}\n\\input{mlqs_clustering.tex}\n\\input{mlqs_supervised_learning.tex}\n\\input{mlqs_modeling_without_time.tex}\n\\input{mlqs_modeling_with_time.tex}\n\\input{mlqs_reinforcement_learning.tex}\n%\\appendix\n%\\newpage\n%\\input{ml4qs_appendix.tex}\n\n\\end{document}"
  },
  {
    "path": "ML4QS/mlqs_supervised_learning.tex",
    "content": "\\section{Supervised Learning}\n\\begin{itemize}\n\t\\item The perspective on supervised learning in this course is summarized in Figure~\\ref{fig:chapter_6_supervised_learning}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/chapter_6_supervised_learning_overview.png}\n\t\t\\caption{Overview of supervised learning framework}\n\t\t\\label{fig:chapter_6_supervised_learning}\n\t\\end{figure}\n\t\\item Discussion of error measuring\n\t\\begin{itemize}\n\t\t\\item \\textit{Risk} $E(h,f)$ describes the distance between our hypothesis $h$ and the target function $f$\n\t\t\\item \\textit{Loss} is the point-wise risk $e(h(x),f(x))$\n\t\t\\item Given the evidence $p(x)$, we can determine the risk by $E(h,f)=\\int e(h(x), f(x)) p(x) dx$\n\t\t\\item However, this integral can (usually) not be computed, and only approximated by Monte-Carlo integration. \n\t\t\\item For definitions of $e$, we can use metrics like F1 or accuracy (classification), or MSE etc. (regression)\n\t\t\\item The in-sample error is the average loss over all training points $E_{in}(h)=\\frac{1}{N}\\sum_{(x,y) \\in \\mathcal{O}_{\\text{train}}} e(y, h(x))$\n\t\t\\item The out-of-sample error accordingly for points not in the training set:\\\\\n\t\t$E_{out}(h)=\\int_{\\mathcal{X}\\setminus \\mathcal{O}_{\\text{train}}} e(h(x), f(x)) p(x) dx$\n\t\\end{itemize}\n\t\\item We select the model with the lowest in-sample error, but need to be careful with overfitting\n\\end{itemize}\n\\subsection{PAC Learnability and VC dimensionality}\n\\begin{itemize}\n\t\\item ``Probably approximately correct learning''\n\t% \\item A hypothesis set is PAC learnable if a learning algorithm exists that can minimize the generalization error to $|E_{out}(\\hat{f}) - E_{in}(\\hat{f})| < \\epsilon$ with a probability of $1-\\delta$\n\t\\item A hypothesis set is PAC learnable when it can be shown that given any value of $\\delta$, $\\epsilon$ there is an $N$ (number of samples) where with probability $1-\\delta$ the difference between the in-sample and out-of-sample error is less than $\\epsilon$. \n\t\\item \\textit{Probably}: $1-\\delta$, \\textit{Approximate correct}: $|E_{out}(\\hat{f}) - E_{in}(\\hat{f})| < \\epsilon$ \n\t\\item For a finite set of $M$ hypotheses, we determine it by:\n\t$$E_{out}(\\hat{f}) \\leq E_{in}(\\hat{f}) + \\sqrt{\\frac{1}{2N}\\log \\frac{2M}{\\delta}}$$\n\tHence, every finite set of hypotheses is PAC learnable, and we can calculate the expected error given number of samples $N$, hypothesis set size $M$, and probability $\\delta$\n\t\\item For infinite set of hypotheses, we can look at VC dimensionality\n\t\\begin{itemize}\n\t\t\\item We say that a set of input vectors $X$ is shattered by a hypothesis set $\\mathcal{H}$ if it can represent all possible labeling\n\t\t\\item The VC dimension of $\\mathcal{H}$ is an $X$ with the highest cardinality $D$. Note that not all possible point sets of cardinality $D$ must be shattered by $\\mathcal{H}$. It is sufficient if it is true for at least one. \n\t\t\\item Example for a perceptron:\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.3\\textwidth]{figures/chapter_6_VC_dimensions.png}\n\t\t\\end{figure}\n\t\n\t\tThe VC dimension of a perceptron in 2 dimensions is $3$, because there exists no set of 4 points that can be shattered (i.e. for which we can learn any labeling)\n\t\t\\item If the hypothesis set can represent any labeling for an arbitrary large set $X$, it has a VC dimension of $\\infty$\n\t\t\\item Important finding: all hypothesis set with a finite VC-dimension are PAC learnable\n\t\t\\item We can \n\t\\end{itemize}\n\t\\item Some implications from this study\n\t\\begin{itemize}\n\t\t\\item Given a few training samples, it is easy to get a low in-sample error. But with increasing number of samples, the out-of-sample error decreases\n\t\t\\item In addition, for a fixed $N$, we can study the influence of more complex hypotheses and find the best compromise\n\t\t\\item  \n\t\\end{itemize}\n\\end{itemize}"
  },
  {
    "path": "Machine_Learning_1/ml_appendix.tex",
    "content": "\\section{Appendix: Foundations}\n\\subsection{Important functions}\n\\subsubsection{Rectified Linear Unit}\nProperties of the ReLU function:\n\\begin{itemize}\n\t\\item $\\text{ReLU}(x)=\\max(x,0)$\n\t\\item $\\text{ReLU}'(x)=\\begin{cases}\n\t1 & \\text{ if } x>0\\\\\n\t0 & \\text{ if } x<0\\\\\n\t\\text{undef} & \\text{ if } x=0\\\\\n\t\\end{cases}$ (last case usually set to 0)\n\t\\item Variations: \n\t\\begin{itemize}\n\t\t\\item Leaky ReLU: $f(x)=\\begin{cases}\n\t\tx & \\text{ if } x>0\\\\\n\t\t0.01x & \\text{ otherwise }\n\t\t\\end{cases}$\n\t\t\\item ELU: $f(x)=\\begin{cases}\n\t\tx & \\text{ if } x>0\\\\\n\t\t\\alpha (e^x-1) & \\text{ otherwise }\n\t\t\\end{cases}$\n\t\t\\item Self-normalizing ELU (carefully selected $\\alpha$ and scaling, so that activations stay close to mean 0, variance 1)\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Sigmoid}\nProperties of the sigmoid function:\n\\begin{itemize}\n\t\\item $\\sigma(x)=\\frac{1}{1+e^{-x}}$\n\t\\item $\\sigma(-x) = 1 - \\sigma(x)$\n\t\\item $\\sigma'(x) = \\sigma(x)\\left(1 - \\sigma(x)\\right)$\n\t\\item Output range: $[0,1]$\n\\end{itemize}\n\\begin{figure}[ht]\n\t\\centering\n\t\\begin{subfigure}[b]{0.3\\textwidth}\n\t\t\\centering\n\t\t\\scalebox{0.5}{%\n\t\t\t\\begin{tikzpicture}\n\t\t\t\\begin{axis}[\n\t\t\ttitle = {Sigmoid},\n\t\t\taxis lines = left,\n\t\t\txlabel = {input $x$},\n\t\t\tylabel = {output $y$},\n\t\t\txmin = -5,\n\t\t\txmax = 5,\n\t\t\tymin = 0,\n\t\t\tymax = 1,\n\t\t\tymajorgrids=true,\n\t\t\txmajorgrids=true,\n\t\t\tgrid style=dashed\n\t\t\t]\n\t\t\t%Here the blue parabloa is defined\n\t\t\t\\addplot [\n\t\t\tdomain=-5:5, \n\t\t\tsamples=1000, \n\t\t\tcolor=black!60!green,\n\t\t\tline width = 0.5mm\n\t\t\t]\n\t\t\t{1/(1+e^(-x))};\n\t\t\t\n\t\t\t\\end{axis}\n\t\t\t\\end{tikzpicture}\n\t\t}\n\t\t\\caption{$\\sigma(x)$}\n\t\t\\label{img:activation_function_sigmoid}\n\t\\end{subfigure}\n\t\\begin{subfigure}[b]{0.3\\textwidth}\n\t\t\\centering\n\t\t\\scalebox{0.5}{%\n\t\t\t\\begin{tikzpicture}\n\t\t\t\\begin{axis}[\n\t\t\ttitle = {Hyperbolic tangent},\n\t\t\taxis lines = left,\n\t\t\txlabel = {input $x$},\n\t\t\tylabel = {output $y$},\n\t\t\txmin = -4,\n\t\t\txmax = 4,\n\t\t\tymin = -1,\n\t\t\tymax = 1,\n\t\t\tymajorgrids=true,\n\t\t\txmajorgrids=true,\n\t\t\tgrid style=dashed\n\t\t\t]\n\t\t\t%Here the blue parabloa is defined\n\t\t\t\\addplot [\n\t\t\tdomain=-4:4, \n\t\t\tsamples=1000, \n\t\t\tcolor=blue,\n\t\t\tline width = 0.5mm\n\t\t\t]\n\t\t\t{tanh(x)};\n\t\t\t\n\t\t\t\\end{axis}\n\t\t\t\\end{tikzpicture}\n\t\t}\n\t\t\\caption{$\\tanh(x)$}\n\t\t\\label{img:activation_function_tanh}\n\t\\end{subfigure}\n\t\\begin{subfigure}[b]{0.3\\textwidth}\n\t\t\\centering\n\t\t\\scalebox{0.5}{%\n\t\t\t\\begin{tikzpicture}\n\t\t\t\\begin{axis}[\n\t\t\ttitle = {Rectified linear unit},\n\t\t\taxis lines = left,\n\t\t\txlabel = {input $x$},\n\t\t\tylabel = {output $y$},\n\t\t\txmin = -2,\n\t\t\txmax = 2,\n\t\t\tymin = 0,\n\t\t\tymax = 2,\n\t\t\tymajorgrids=true,\n\t\t\txmajorgrids=true,\n\t\t\tgrid style=dashed\n\t\t\t]\n\t\t\t%Here the blue parabloa is defined\n\t\t\t\\addplot [\n\t\t\tdomain=-2:2, \n\t\t\tsamples=1000, \n\t\t\tcolor=orange,\n\t\t\tline width=0.5mm\n\t\t\t]\n\t\t\t{(x > 0)*x};\n\t\t\t\n\t\t\t\\end{axis}\n\t\t\t\\end{tikzpicture}\n\t\t}\n\t\t\\caption{$\\text{ReLU}(x)$}\n\t\t\\label{img:activation_function_relu}\n\t\\end{subfigure}\n\t\n\t\\caption[Comparison of activation functions]{(a) The sigmoid function maps the inputs to a range of 0 to 1 while having high gradients near to $y=0$ to bring the output more to either 0 or 1. (b) The hyperbolic tangent is similar to the sigmoid function but has a output range of -1 to 1. (c) A rectified linear unit (ReLU) is 0 for all input lower than 0. All other values are processed linearly so that they do not change.}\n\\end{figure}\n\\subsubsection{Hyperbolic tan}\nProperties of the hyperbolic tan:\n\\begin{itemize}\n\t\\item $\\tanh(x) = \\frac{e^{x} - e^{-x}}{e^{x} + e^{-x}}$\n\t\\item $\\tanh(-x) = -\\tanh(x)$\n\t\\item $\\tanh'(x) = 1 - \\tanh(x)^2$\n\t\\item Output range: $[-1,1]$\n\\end{itemize}\n\\subsubsection{Softmax}\nProperties:\n\\begin{itemize}\n\t\\item $\\text{softmax}(x_k) = \\frac{\\exp(x_k)}{\\sum_{i=1}^{N}\\exp(x_i)}$\n\t\\item If $x_k \\gg x_j$, the softmax for all $j\\neq k$ is approx. 0, whereas for $k$ it is 1\n\t\\item Maps vector from $\\bm{y} \\in \\mathbb{R}^D$ to probability distribution $\\bm{y}' \\in [0,1]^D$ with $\\sum_{i=1}^{D}y'_{i} = 1$ $\\Rightarrow$ useful for multi-class classification\n\t\\item Invariant to bias: $\\frac{\\exp(x_k + c)}{\\sum_{i=1}^{N}\\exp(x_i + c)} = \\frac{\\exp(x_k)}{\\sum_{i=1}^{N}\\exp(x_i)}= \\text{softmax}(x_k)$\n\\end{itemize}\n\\subsection{Matrix operations}\n\\subsubsection{Properties of transposed and inverse matrices}\n\\textbf{Transpose}\n\\begin{itemize}\n\t\\item $(AB)^T=B^T A^T$\n\t\\item $\\det\\left(A^{T}\\right) = \\det\\left(A\\right)$\n\\end{itemize}\n\\textbf{Inverse}\n\\begin{itemize}\n\t\\item $(AB)^{-1} = B^{-1} A^{-1}$\n\t\\item $\\det\\left(A^{-1}\\right) = \\det\\left(A\\right)^{-1}$\n\\end{itemize}\n\\textbf{Combination}\n\\begin{itemize}\n\t\\item $\\left(A^{-1}\\right)^T = \\left(A^{T}\\right)^{-1}$\n\\end{itemize}\n\\subsubsection{Derivations}\n\\subsubsection{Hand-in 1: 1.3d}\nDerivation of multivariate Gaussian by matrix\n\\subsection{Lagrange Multiplier}\n\\begin{itemize}\n\t\\item Finding stationary points of a function with subject to one or more constraints\n\t\\item \\textbf{Equality constraint}\n\t\\begin{itemize}\n\t\t\\item Maximize $f(\\bm{x})$ with respect to constraint $g(\\bm{x})=0$\n\t\t\\item At a constrained maximum, we know that $\\nabla f(\\bm{x}) = -\\lambda \\nabla g(\\bm{x})$\n\t\t\\item The Lagrangian function is therefore $$ L(\\bm{x}, \\lambda) = f(\\bm{x}) + \\lambda g(\\bm{x})$$\n\t\t\\item We solve it by maximizing regarding to $\\bm{x}$ and $\\lambda$: $$\\max_{\\bm{x}} \\max_{\\lambda} L(\\bm{x}, \\lambda)$$\n\t\t\\item Note that the sign of the constraint is irrelevant. A minus sign leads to the same result as $g(x)$ must be zero at this point\n\t\t\\item We find solutions by setting the derivate of both primal and dual variables to 0:\n\t\t$$\\frac{\\partial }{\\partial \\bm{x}} L(\\bm{x}, \\lambda) = 0, \\text{\\hspace{3mm}} \\frac{\\partial }{\\partial \\lambda} L(\\bm{x}, \\lambda) = 0$$\n\t\\end{itemize}\n\t\\item \\textbf{Inequality constraint}\n\t\\begin{itemize}\n\t\t\\item Maximize $f(\\bm{x})$ with respect to constraint $g(\\bm{x})\\geq0$ (introduce Lagrangian multiplier $\\mu$)\n\t\t\\item Two kinds of solutions:\n\t\t\\begin{itemize}\n\t\t\t\\item If the optimum of $f(\\bm{x})$ lies already in the region of $g(\\bm{x})\\geq0$, then we have an inactive constraint $\\Rightarrow$ $\\mu=0$\n\t\t\t\\item Otherwise, the optimum is on the boundary so that $g(\\bm{x})=$ and $\\mu> 0$\n\t\t\\end{itemize}\n\t\t\\item Thus, our primal Lagrangian is defined as:\n\t\t$$L(\\bm{x}, \\mu) = f(\\bm{x}) + \\mu g(\\bm{x})$$\n\t\t\\item We now maximize regarding $\\bm{x}$, but \\textit{minimize} for the Lagrangian multiplier as we prefer $f(x)$ being inside the constraint area:\n\t\t$$\\max_{\\bm{x}} \\min_{\\mu} L(\\bm{x}, \\mu)$$\n\t\t\\item Note that the sign is here important. When we minimize $f(x)$, we can keep the max-min conditions for the Lagrangian but then have to switch the sign in front of the constraint!\n\t\t\\item Also, deriving by $\\mu$ does not guarantee a valid solution anymore as we have the following KKT conditions for \\textit{every} Lagrangian multiplier:\n\t\t$$\\mu \\geq 0 \\hspace{5mm} g(\\bm{x})\\geq 0 \\hspace{5mm} \\mu g(\\bm{x}) = 0$$\n\t\t\\item We obtain the dual Lagrangian by optimizing with respect to only the primal variables $\\bm{x}$, and replacing those in the primal Lagrangian:\n\t\t$$\\tilde{L}(\\mu) = \\max_{\\bm{x}} L(\\bm{x},\\mu)$$\n\t\t\\item Next, minimize with respect to the dual parameters $\\mu$ by considering the constraint $\\mu=0$\n\t\\end{itemize}\n\t\\item \\textbf{Combined constraints}\n\t\\begin{itemize}\n\t\t\\item If we have multiple constraints (can be pure (in-)equalities or mixed), we just add them all to our Lagrangian function\n\t\t\\item Solve with respect to all constraints\n\t\\end{itemize}\n\\end{itemize}"
  },
  {
    "path": "Machine_Learning_1/ml_basic_probability.tex",
    "content": "\\section{Probability Theory}\n\\subsection{Multivariate Gaussian}\n$$\\mathcal{N}\\left(\\bm{x}|\\bm{\\mu}, \\bm{\\Sigma}\\right) = \\frac{1}{\\left(2\\pi\\right)^{D/2} \\cdot |\\bm{\\Sigma}|^{1/2}}\\cdot \\exp\\left(-\\frac{1}{2}\\left(\\bm{x}-\\bm{\\mu}\\right)^T\\bm{\\Sigma}^{-1}\\left(\\bm{x}-\\bm{\\mu}\\right)\\right)$$\n\n$$\\frac{\\partial}{\\partial \\bm{\\mu}}\\mathcal{N}\\left(\\bm{x}|\\bm{\\mu}, \\bm{\\Sigma}\\right)  = \\mathcal{N}\\left(\\bm{x}|\\bm{\\mu}, \\bm{\\Sigma}\\right) \\left(\\bm{x}-\\bm{\\mu}_k\\right)^T\\bm{\\Sigma}^{-1}$$\n\n% $$\\frac{\\partial}{\\partial \\bm{\\Sigma}}\\mathcal{N}\\left(\\bm{x}|\\bm{\\mu}, \\bm{\\Sigma}\\right)  = ...$$\n\n\\subsection{Rules of probability}\n\\begin{table}[ht]\n\t\\centering\n\t\\begin{tabular}{c|cc}\n\t\t& \\textbf{Discrete} & \\textbf{Continuous}\\\\\n\t\t\\hline\n\t\t\\textbf{Additivity} & $p(X\\in A) = \\sum\\limits_{x\\in A}p(x)$ & $p\\left(x\\in (a,b)\\right) = \\int\\limits_{a}^{b}p(x)dx$\\\\[10pt]\n\t\t\\textbf{Positivity} & $0 \\leq p(x)\\leq 1$ & $0 \\leq p(x) \\not\\leq 1$\\\\[10pt]\n\t\t\\textbf{Normalization} & $\\sum_{x} p(x) = 1$ & $\\int_{\\chi} p(x)dx = 1$\\\\[10pt]\n\t\t\\textbf{Sum Rule} & $p(x) = \\sum\\limits_{y\\in\\mathcal{Y}} p(x,y)$ & $p(x)=\\int\\limits p(x,y)dy$\\\\[10pt]\n\t\t\\textbf{Product Rule} & $p(x,y) = p(x|y)p(y)$ & $p(x,y) = p(x|y)p(y)$\n\t\\end{tabular}\n\\end{table}\n\\subsection{Bayes Rule}\n$$\\underbrace{p(x|y)}_{\\text{posterior}} = \\frac{\\overbrace{p(y|x)}^{\\text{likelihood}} \\overbrace{p(x)}^{\\text{prior}}}{\\underbrace{p(y)}_{\\text{evidence}}} = \\frac{p(y|x) p(x)}{\\int p(y|x) p(x)dx} \\text{\\hspace{5mm}or\\hspace{5mm}} \\frac{p(y|x) p(x)}{\\sum p(y|x) p(x)}$$"
  },
  {
    "path": "Machine_Learning_1/ml_combining_models.tex",
    "content": "\\section{Combining models}\n\\begin{itemize}\n\t\\item Improve performance by combining different models\n\t\\item For example, we can train $L$ different models and take their average as prediction (called committee)\n\t\\item Alternatively, we can also make the choice of which model we should use for an input $\\bm{x}$ dependent on $\\bm{x}$. This example includes Mixtures of experts\n\t\\item \\textbf{Bayesian model averaging vs. model combination methods}\n\t\\begin{itemize}\n\t\t\\item In Bayesian model averaging, the entire dataset is generated by a single model. We are just unsure which one it is. The likelihood of the data is thus:\n\t\t$$p(\\bm{X}) = \\sum_{h=1}^{H} p(\\bm{X}|h)p(h)$$\n\t\t\\item In contrast, model combination methods consider that different data points can be generated by different components. So, every data point has its own latent variable $\\bm{z}_n$. The likelihood is here given by:\n\t\t$$p(\\bm{X}) = \\prod_{n=1}^{N}\\sum_{\\bm{z}_n} p(\\bm{x}_n|\\bm{z}_n)p(\\bm{z}_n)$$\n\t\tExample methods include Gaussian mixture models and Mixture of experts.\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Committees}\n\\begin{itemize}\n\t\\item We can motivate the idea of committees by the bias-variance decomposition: when we average over models, we are able to reduce the variance of the model's predictions. Thus, by using complex models with low bias error, we can improve the performance by reducing the variance through averaging\n\t\\item Averaging is therefore only effective if models are complex enough to overfit\n\t\\item However, in practice, we have only one dataset on which we train $\\Rightarrow$ introduce variability between the models within the committee by various methods\n\\end{itemize}\n\\subsubsection{Bootstrap aggregation}\n\\begin{itemize}\n\t\\item Suppose we have a dataset $\\bm{X} = \\left[\\bm{x}_1, ..., \\bm{x}_N\\right]^T$\n\t\\item \\textbf{Bootstrapping dataset}: we create $B$ datasets by sampling $N$ datapoints \\textit{with replacement} from the original dataset $\\bm{X}$. So, in $\\bm{X}_b$, some points will occur more than once and others might be absent\n\t\\item For doing regression with this method, we train $B$ models on their corresponding dataset, and use the average prediction for a new point:\n\t$$y(\\bm{x}) = \\frac{1}{B}\\sum\\limits_{b=1}^{B} y_b(\\bm{x})$$\n\t\\item This is called bootstrap aggregation or also \\textit{bagging}\n\t\\item The average error made by one of the models is $E_{\\text{AV}} = \\frac{1}{B}\\sum_{b=1}^{B} \\mathbb{E}_{\\bm{x}}\\left[\\epsilon_b(\\bm{x})^2\\right]$. In contrast, for the committee, we expect an error of:\n\t$$E_{\\text{COM}} = \\mathbb{E}_{\\bm{x}}\\left[\\left\\{\\frac{1}{B}\\sum\\limits_{b=1}^{B}\\epsilon_b(\\bm{x})\\right\\}^2\\right]$$\n\t\\item If all models would be independent (which they are not because of using very similar datasets), we would reduce the expected error by factor $B$. In practice, we can at least guarantee that $E_{\\text{COM}}\\leq E_{\\text{AV}}$\n\t\\item Still, bias error cannot be reduced by bagging!\n\\end{itemize}\n\\subsubsection{Feature bagging}\n\\begin{itemize}\n\t\\item Similar to bagging, but based on features: sample a subset of \\textit{features} of length $r<D$ for each learner. \n\t$$\\bm{x} = \\left[x_1, x_2, \\dots, x_D\\right]^T\\Rightarrow \\bm{\\tilde{x}} = \\left[x_1, x_3, x_5, x_{D-1}\\right]^T$$\n\t\\item Also called \\textit{random subspace method}\n\t\\item Works especially well if features are uncorrelated and/or if the number of features is much larger than the number of training points\n\t\\item Decision trees with bagging and random subspaces lead to random forests\n\\end{itemize}\n\\subsubsection{Boosting}\n\\begin{itemize}\n\t\\item Use a set of simple individual models (also called weak classifiers) which even can be only slightly better than random\n\t\\item In the following description, we concentrate on boosting for classification, but it can also be used for regression\n\t\\item \\textbf{AdaBoost}: adaptive boosting\n\t\\item Base classifiers are trained in a sequence where every model uses a weighted form of the dataset\n\t\\item The weight coefficients are associated to the performance of the previous models\n\t\\item In the end, a prediction is based on the (weighted) majority voting scheme:\n\t$$Y_M(\\bm{x}) = \\text{sign}\\left(\\sum\\limits_{m=1}^{M}\\alpha_m y_m(\\bm{x})\\right)$$\n\t\\item AdaBoost algorithm:\n\t\\begin{enumerate}\n\t\t\\item Initialize weights $w_n = 1/N$ for all $n=1, ...,N$\n\t\t\\item For all models $m=1,...,M$ sequentially:\n\t\t\\begin{enumerate}\n\t\t\t\\item Fit classifier $y_m(\\bm{x})$ to minimize $J_m = \\sum\\limits_{n=1}^{N} w_n^{(m)} \\bm{I}[y_m(\\bm{x}_n)\\neq t_n]$\n\t\t\t\\item Compute weighted error rate $\\epsilon_m = \\frac{\\sum_{n=1}^{N}w_n^{(m)}\\bm{I}[y_m(\\bm{x}_n)\\neq t_n]}{\\sum_{n=1}^{N}w_n^{(m)}}$ and $\\alpha_m = \\ln\\left(\\frac{1-\\epsilon_m}{\\epsilon_m}\\right)$\n\t\t\t\\item Update weights $w_n^{(m+1)} = w_n^{(m)}\\exp\\left\\{\\alpha_m \\bm{I}\\left[y_m(\\bm{x}_n)\\neq t_n\\right]\\right\\}$\n\t\t\\end{enumerate}\n\t\t\\item Make predictions $Y_M(\\bm{x}) = \\text{sign}\\left(\\sum\\limits_{m=1}^{M}\\alpha_m y_m(\\bm{x})\\right)$\n\t\\end{enumerate}\n\t\\item Note that the weight in the prediction ($\\alpha_m$) is based on the average error it has on the weighted training dataset (greater weights for more accurate models)\n\t\\item When taking a huge number of basis models (large $M$), we can easily overfit\n\t\\item Interpretation/Derivation of AdaBoost: minimizing exponential error function sequentially ($E_m = \\sum_{n=1}^{N}\\exp\\left(-t_n f_m(\\bm{x}_n)\\right)$)\n\t\\item \\textbf{Advantages}: simple boosting algorithm\n\t\\item \\textbf{Disadvantages}: very sensitive to outliers ($t_n y_m(\\bm{x})$ very large and exponentially increasing weight), no probabilistic interpretation\n\\end{itemize}\n\\subsection{Decision trees}\n\\begin{itemize}\n\t\\item Split input space into rectangles which are aligned along the axes (parallel to axes)\n\t\\item We use sequential binary decisions which can be summarized in a tree structure\n\t\\item Used for classification and regression\n\t\\item \\textbf{Advantages}: interpretable, combining with boosting strongly increases performance\n\t\\item \\textbf{Disadvantages}: Not state-of-the-art, large trees easily overfit but small trees underfit (can be prevented by training large trees and sequentially removing nodes that reduce the error the least)\n\t\\item Tree building process is recursively by minimizing the squared error (for regression). At each iteration, we add the feature boundary that reduces the error the most\n\t\\item Stop criteria can be for example min. number of data points in region, depth/height,... or decrease of loss is lower than certain threshold\n\t\\item \\textit{Pruning}: give a penalty to trees with large number of leafs to prevent unnecessary overfitting \n\t\\item \\textbf{Random forests}: By combining bootstrapping and feature bagging, we ensure that the models uses different features to build the trees. Thus, the models are less correlated and probably result in better accuracies.\n\\end{itemize}"
  },
  {
    "path": "Machine_Learning_1/ml_kernel_methods.tex",
    "content": "\\section{Kernel methods}\n\\begin{itemize}\n\t\\item Standard parametric models have either fixed basis functions (like linear regression or linear classification models) or learnable basis functions like in neural networks. The training points are solely used to optimize the parameters $\\bm{w}$, and all further predictions are based on these optimal parameters\n\t\\item In contrast, kernel methods keep the training data points and also use them (or a subset) during prediction\n\t\\item The predictions are based on a linear combination of the kernel function evaluated on the training data points:\n\t$$y(\\bm{x}) = \\sum\\limits_{n=1}^{N} \\alpha_n k\\left(\\bm{x}, \\bm{x}'\\right)$$\n\t\\item For linear models with fixed basis functions, the kernel is:\n\t$$k(\\bm{x},\\bm{x}') = \\bm{\\phi}(\\bm{x})^T\\bm{\\phi}(\\bm{x}')$$\n\t\\item The kernel measures \\textit{similarity} between $\\bm{x}$ and $\\bm{x}'$ in features space defined by $\\phi(x)$. Thus, it is symmetric: $$k(\\bm{x},\\bm{x}')=k(\\bm{x}',\\bm{x})$$\n\\end{itemize}\n\\subsection{Kernelizing linear parametric models}\n\\begin{itemize}\n\t\\item Many linear parametric model can be re-casted into a ``dual representation'' by using the \\textbf{kernel trick}: \n\t\\begin{itemize}\n\t\t\\item If we have an algorithm formulated in such a way that the input vector $\\bm{x}$ enters only in the form of a scalar product, we can replace the scalar product with some other choice of kernel\n\t\\end{itemize}\n\t\\item For instance, the linear regression model is determined by minimizing the regularized sum-of-squares error function given by:\n\t$$J(\\bm{w}) = \\frac{1}{2}\\sum\\limits_{n=1}^{N} \\left\\{\\bm{w}^T \\bm{\\phi}\\left(\\bm{x}_n\\right) - t_n \\right\\} + \\frac{\\lambda}{2}\\bm{w}^T \\bm{w}$$\n\t\\item Solving the equation of the derivate being equals to 0, we obtain:\n\t$$\\bm{w} = \\left(\\bm{\\Phi}^T\\bm{\\Phi} +\\lambda \\bm{I}_M\\right)^{-1}\\bm{\\Phi}^T\\bm{t} = \\bm{\\Phi}^T\\left(\\bm{\\Phi}\\bm{\\Phi}^T +\\lambda \\bm{I}_M\\right)^{-1}\\bm{t} $$\n\t\\item Here, we can replace the inner product $\\bm{\\Phi}\\bm{\\Phi}^T$ by the gram matrix $\\bm{K}$ where $K_{ij} = \\bm{\\phi}(\\bm{x}_i)^T\\bm{\\phi}(\\bm{x}_j)$\n\t\\item By defining the dual variable $\\bm{\\alpha} = \\left(\\bm{K} +\\lambda \\bm{I}_M\\right)^{-1}\\bm{t}$, we get the following equations:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\bm{w} & =\\bm{\\Phi}^T \\bm{\\alpha} = \\sum\\limits_{n=1}^{N} \\alpha_n \\bm{\\phi}(\\bm{x}_n)\\\\\n\t\t\ty\\left(\\bm{x}'\\right) & = \\bm{w}^T \\bm{\\phi}(\\bm{x}') = \\sum\\limits_{n=1}^{N} \\alpha_n \\bm{\\phi}\\left(\\bm{x}_n\\right)^T \\bm{\\phi}\\left(\\bm{x}'\\right) = \\sum\\limits_{n=1}^{N} \\alpha_n k\\left(\\bm{x},\\bm{x}'\\right)\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Thus, we can express linear regression by a dual representation with kernel methods\n\t\\item \\textbf{Benefits} of kernel representation:\n\t\\begin{itemize}\n\t\t\\item We have no explicit parameters/features anymore, only implicit by the kernel function $k(\\bm{x},\\bm{x}')$\n\t\t\\item No need to handpick locations of basis functions \n\t\t\\item No increase in number of parameters when using kernel methods as those implicitly map inputs to a higher dimensional space\n\t\\end{itemize}\n\t\\item \\textbf{Disadvantages}/\\textbf{problems}:\n\t\\begin{itemize}\n\t\t\\item The computational cost to retrieve $\\bm{\\alpha}$ is $\\mathcal{O}(N^3)$ as $\\bm{K}\\in\\mathbb{R}^{N\\times N}$ compared to $\\mathcal{O}(M^3)$ for calculating $M$ on the standard way (the cost comes from the inverse)\n\t\t\\item During prediction, we need $\\mathcal{O}(N\\cdot M)$ to compute the output for a new point, but would only need $\\mathcal{O}(N)$ with the primal parameters $\\bm{w}$ $\\Rightarrow$ slow prediction for large datasets\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Constructing valid kernels}\n\\begin{itemize}\n\t\\item For a valid kernel, the gram matrix $\\bm{K}$ must be positive semi-definite for all possible choices of $\\left\\{x_n\\right\\}_{n=1}^{N}$\n\t\\item An equivalent constraint would be that $\\bm{z}^T \\bm{K} \\bm{z} \\geq 0$ for all $\\bm{z}\\in\\mathbb{R}^{N}$ or the eigenvalues must all be positive (note that $\\bm{K}$ can still contain negative elements)\n\t\\item We can construct a kernel from an explicit set of basis functions when we use the expression $k\\left(\\bm{x},\\bm{x}'\\right)=\\bm{\\phi}^T(\\bm{x})\\bm{\\phi}(\\bm{x})$\n\t\\item Further, we can construct new kernels by using other valid kernels and extend them by for example multiplying with a constant (no need to know all variations)\n\t\\item Given a valid kernel function, we can derive its corresponding feature vectors (which can be hard and possible infinite). Therefore, we need to express it in the form of $\\bm{\\phi}(\\bm{x})^T \\bm{\\phi}(\\bm{x}')$ where $\\bm{\\phi}$ must be the same function applied on different points\n\t\\item For example a polynomial kernel of $M=2$ can be rewritten as:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tk\\left(\\bm{x},\\bm{z}\\right) & = \\left(1+\\bm{x}^T\\bm{z}\\right)^2 = \\left(1 + x_1 z_1 + x_2 z_2\\right)^2\\\\\n\t\t\t& = 1 + 2x_1 z_1 + 2x_1 z_2 + (x_1 z_1)^2 + (x_2 z_2)^2 + 2 x_1 z_1 x_2 z_2\\\\\n\t\t\t& = \\left[1, \\sqrt{2}x_1, \\sqrt{2}x_2, x_1, x_2, \\sqrt{2}x_1 x_2\\right] \\cdot \\left[1, \\sqrt{2}z_1, \\sqrt{2}z_2, z_1, z_2, \\sqrt{2}z_1 z_2\\right]^T\\\\\n\t\t\t& = \\bm{\\phi}(\\bm{x})^T \\bm{\\phi}(\\bm{z})\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Here we see that from a two-dimensional vector, we scaled it up to a 6-dimensional feature vector just from our kernel\n\t\\item Some (popular) kernels:\n\t\\begin{itemize}\n\t\t\\item Generalized polynomial kernel $k\\left(\\bm{x}, \\bm{x}'\\right) = \\left(c + \\bm{x}^T \\bm{x}'\\right)^{M}$ (feature vector only contains polynomial to order $M$)\n\t\t\\item Gaussian kernel with infinite feature dimensionality: $k\\left(\\bm{x}, \\bm{x}'\\right) = \\exp\\left(-\\frac{1}{2l^2} ||\\bm{x}-\\bm{x}'||^2\\right)$\n\t\t\\item Radial basis functions of the form $k\\left(\\bm{x}, \\bm{x}'\\right) = k\\left(||\\bm{x}-\\bm{x}'||^2\\right)$\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Support Vector Machines}\n\\begin{itemize}\n\t\\item To overcome the slow prediction problem, support vector machines only uses a subset of the training points on which the kernel function needs to be evaluated (also called kernel methods with \\textit{sparse} solutions)\n\t\\item It is a convex optimization problem so that only one single optimum exists\n\t\\item No good probabilistic interpretation (see Gaussian Processes for that)\n\\end{itemize}\n\\subsubsection{Maximum Margin Classifier}\n\\begin{itemize}\n\t\\item Similar to discriminant functions in 3.3\n\t\\item For a linearly separable dataset, the maximum margin is defined as the distance between the decision boundary and the closest training point $\\Rightarrow$ most robust and stable for perturbations of the input\n\t\\item The distance of a point to the decision boundary is (as previously) defined by:\n\t$$r_n = \\frac{|y(\\bm{x}_n)|}{||w||} = \\frac{t_ny(\\bm{x}_n)}{||w||}\\text{\\hspace{2mm} if } \\bm{x}_n \\text{ correctly classified}$$ \n\t\\item The margin is defined as the minimum distance of decision boundary to any point:\n\t$$\\min_n \\frac{t_n \\left(\\bm{w}^T\\bm{x}_n + b\\right)}{||\\bm{w}||} $$\n\t\\item As we can easily increase the distance by increasing $\\bm{w}$ by a factor $\\kappa$ and still get the same minimum ($\\min_n \\frac{t_n \\left(\\kappa \\bm{w}^T\\bm{x}_n + \\kappa b\\right)}{||\\kappa \\bm{w}||}$), we restrict the choice by setting $t_n \\left(\\bm{w}^T\\bm{x}_n + b\\right) = 1$ for the closest point. \n\t\\item Thus, for all other points, the following constraint must hold: $t_n \\left(\\bm{w}^T\\bm{x}_n + b\\right) \\geq 1$\n\t\\item A maximum margin is found by maximizing $\\frac{1}{||\\bm{w}||}$ (as the upper part of the fraction is fixed to 1)\n\\end{itemize}\n\\subsubsection{Optimizing Maximum Margin}\n\\begin{itemize}\n\t\\item To maximize the margin, we try to minimize $\\frac{1}{2}||\\bm{w}||^2$ (has same optimum as $\\frac{1}{||\\bm{w}||}$ but is easier to optimize)\n\t\\item By that, we need to fulfill the constraint $t_n \\left(\\bm{w}^T\\bm{x}_n + b\\right) \\geq 1$ for all data points\n\t\\item To do that, we use Lagrange multiplier for inequalities\n\t\\begin{itemize}\n\t\t\\item Given the problem to maximize $f(\\bm{x})$ subject to $g(\\bm{x})\\geq 0$, it is equivalent to optimize:\n\t\t$$\\max_{\\bm{x}} \\min_{\\mu} L\\left(\\bm{x},\\mu\\right) = \\max_{\\bm{x}} \\min_{\\mu} f(\\bm{x}) + \\mu g(\\bm{x})$$\n\t\t\\item Note that if we want to minimize $f(\\bm{x})$, it is equivalent to maximizing $-f(\\bm{x})$: \n\t\t$$\\max_{\\bm{x}} \\min_{\\mu} L\\left(\\bm{x},\\mu\\right) = \\max_{\\bm{x}} \\min_{\\mu} -f(\\bm{x}) + \\mu g(\\bm{x}) \\Rightarrow \\min_{\\bm{x}} \\max_{\\mu} L\\left(\\bm{x},\\mu\\right) = \\min_{\\bm{x}} \\max_{\\mu} f(\\bm{x}) - \\mu g(\\bm{x})$$\n\t\t\\item We have the following (Karush-Kuhn-Tucker) conditions when optimizing this function:\n\t\t$$\\mu\\geq0, \\text{\\hspace{5mm}}g(\\bm{x})\\geq0, \\text{\\hspace{5mm}}\\mu\\cdot g(\\bm{x}) = 0$$\n\t\t\\item There are two kinds of solutions:\n\t\t\\begin{itemize}\n\t\t\t\\item If the stationary points lies in the region $g(\\bm{x})\\geq 0$, we have $\\nabla f(\\bm{x})=0$ and $\\mu=0$\n\t\t\t\\item Otherwise, if stationary points lies on the boundary we have $\\nabla f(\\bm{x})=-\\mu \\nabla g(\\bm{x})$\n\t\t\\end{itemize}\n\t\t\\item We can solve the optimization problem by first getting a solution for $\\tilde{L}(\\mu) = \\max_{\\bm{x}} L(\\bm{x},\\mu)$, and then optimizing it with respect to $\\mu$: $\\max_{\\mu}\\tilde{L}(\\mu)$\n\t\\end{itemize}\n\t\\item When we apply this for our maximum margin classifier, we get the following optimization objective with $N$ Lagrange multipliers $a_n$:\n\t$$L\\left(\\bm{w},b,\\bm{a}\\right) = \\frac{1}{2}||\\bm{w}||^2 - \\sum\\limits_{n=1}^{N} a_n \\left\\{t_n \\left(\\bm{w}^T \\bm{x} + b\\right) - 1\\right\\}$$\n\t\\item First, minimize with respect to the primal variables $\\bm{w}$ and $b$:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\frac{\\partial L}{\\partial \\bm{w}} & = \\bm{w}^T - \\sum\\limits_{n=1}^{N} a_n t_n \\bm{x}_n^T = 0\\to \\bm{w} = \\sum\\limits_{n=1}^{N}a_n t_n \\bm{x}_n^T\\\\\n\t\t\t\\frac{\\partial L}{\\partial b} & = - \\sum\\limits_{n=1}^{N} a_n t_n = 0\\to \\sum\\limits_{n=1}^{N} a_n t_n = 0\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Eliminating $\\bm{w}$ and $b$ gives the dual representation $\\tilde{L}(\\bm{a})$:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\tilde{L}(\\bm{a}) = \\sum\\limits_{n=1}^{N} a_n - \\frac{1}{2} \\sum\\limits_{n=1}^{N}\\sum\\limits_{m=1}^{N} a_n a_m t_n t_m \\underbrace{\\bm{x}_n^T \\bm{x}_m}_{k(\\bm{x}_n, \\bm{x}_m)}\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item For prediction, we use the previously derived result $\\bm{w} = \\sum_{n=1}^{N}a_n t_n \\bm{x}_n^T$ to convert it into a kernel:\n\t$$y(\\bm{x})=\\bm{w}^T \\bm{x} + b = \\sum_{n=1}^{N}a_n t_n k(\\bm{x}_n, \\bm{x})$$\n\t\\item For every data point, $a_n = 0$ or $t_n y(\\bm{x}) = 1$. For all points that have $a_n > 0$ influence the prediction, so called support vectors. They lie on maximum margin hyperplanes \n\t\\item The bias $b$ can be determined by solving $t_n y(\\bm{x}_n) = 1$ for a support vector $x_n$\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.3\\textwidth]{figures/svm_support_vectors.png}\n\t\t\\caption{Visualization of non-linear support vectors}\n\t\t\\label{img:svm_support_vectors}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{Soft Margin Classifier}\n\\begin{itemize}\n\t\\item So far we assumed that dataset is (non-linear) separable. However, sometimes distributions overlap \n\t\\item Thus, soft margin classifier allow data points to be on the \"wrong\" side of the margin but causing a certain penalty\n\t\\item We introduce \\textbf{slack variables} $\\xi_n\\geq 0$ for $n=1,...,N$\n\t\\item If a point is on the correct side of the margin, its slack variable is $\\xi_n = 0$\n\t\\item If it is one the wrong side of the margin, the slack variable is $\\xi_n = |t_n - y(\\bm{x}_n)|$\n\t\\item Hence, we also have a ``soft'' constraint/margin $t_n y(\\bm{x}_n)\\geq 1 - \\xi_n$\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.3\\textwidth]{figures/svm_soft_margin.png}\n\t\t\\caption{A soft margin classifier uses slack variables to penalize data points on the wrong side.}\n\t\t\\label{img:svm_soft_margin}\n\t\\end{figure}\n\t\\item The goal is now to maximize the margin while minimizing the penalty given by the slack variables:\n\t$$\\arg\\min_{\\bm{w},b,\\bm{\\xi}} \\frac{1}{2} ||\\bm{w}||^2 + C\\sum\\limits_{n=1}^{N} \\xi_n$$\n\t\\item Introducing the conditions $\\xi_n\\geq 0$ and $t_n y(\\bm{x}_n)\\geq 1 - \\xi_n$ into the minimization problem, we get the following Lagrangian:\n\t$$L\\left(\\bm{w}, b, \\bm{\\xi}, \\bm{a}, \\bm{\\mu}\\right) = \\frac{1}{2} ||\\bm{w}||^2 +C \\sum\\limits_{n=1}^{N} \\xi_n - \\sum\\limits_{n=1}^{N} a_n \\left\\{t_n \\left(\\bm{w}^T \\bm{x}_n + b\\right) - 1 + \\xi_n \\right\\} - \\sum\\limits_{n=1}^{N} \\mu_n \\xi_n $$\n\t\\item The KKT conditions for the dual variables are:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t& a_n \\geq 0,\\text{\\hspace{3mm}} t_n y(\\bm{x}_n) - 1 + \\xi_n \\geq 0,\\text{\\hspace{3mm}} a_n \\left\\{t_n \\left(\\bm{w}^T \\bm{x}_n + b\\right) - 1 + \\xi_n \\right\\} = 0\\\\\n\t\t\t& \\mu_n \\geq 0,\\text{\\hspace{3mm}} \\xi_n \\geq 0,\\text{\\hspace{3mm}} \\mu_n \\xi_n = 0\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Minimize w.r.t. primal variables $\\bm{w}, b, \\bm{\\xi}$ and use these conditions to eliminate $\\bm{w}, b, \\bm{\\xi}$ from the Lagrangian to obtain the \\textbf{dual representation}\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\frac{\\partial L}{\\partial \\bm{w}} & = \\bm{w}^T - \\sum\\limits_{n=1}^{N} a_n t_n \\bm{x}_n^T = 0 \\implies \\bm{w} = \\sum\\limits_{n=1}^{N} a_n t_n \\bm{x}_n\\\\\n\t\t\t\\frac{\\partial L}{\\partial b} & = - \\sum\\limits_{n=1}^{N} a_n t_n = 0 \\implies \\sum\\limits_{n=1}^{N} a_n t_n = 0\\\\\n\t\t\t\\frac{\\partial L}{\\partial \\xi_n} & = C - a_n - \\mu_n = 0 \\implies a_n  = C - \\mu_n\\\\\n\t\t\t\\Rightarrow \\tilde{L}(\\bm{a}) & = \\sum\\limits_{n=1}^{N} a_n - \\frac{1}{2}  \\sum\\limits_{n=1}^{N}  \\sum\\limits_{m=1}^{N} a_n a_m t_n t_m \\bm{x}_n^T \\bm{x}_m\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item The remaining constraints are $0\\leq a_n \\leq C$, and $\\sum\\limits_{n=1}^{N} a_n t_n = 0$, and we try to \\textit{maximize} $\\tilde{L}(\\bm{a})$\n\t\\item We can also express the dual representation with the kernel trick:\n\t$$\\tilde{L}(\\bm{a}) = \\sum\\limits_{n=1}^{N} a_n - \\frac{1}{2}  \\sum\\limits_{n=1}^{N}  \\sum\\limits_{m=1}^{N} a_n a_m t_n t_m k(\\bm{x}_n, \\bm{x}_m)$$\n\t\\item When we want to predict the class for a new test data point $\\bm{x}'$, we use:\n\t$$y(\\bm{x}') = \\sum\\limits_{n=1}^{N} a_n t_n k(\\bm{x}_n, \\bm{x}') + b$$\n\t\\item Points for different dual parameters:\n\t\\begin{itemize}\n\t\t\\item Only points with $a_n > 0$ are support vectors and contribute to the prediction\n\t\t\\item If $C > a_n > 0$, then $t_n y(\\bm{x}_n) = 1$ (points on the margin) as $\\mu_n > 0$ and hence $\\xi_n = 0$\n\t\t\\item If $a_n = C$, then $\\mu_n = 0$ and $\\xi_n \\geq 0$. When $\\xi_n \\leq 1$, the points is still correctly classified but within the margin. Otherwise, the point is misclassified\n\t\\end{itemize}\n\t\\item If $C\\to\\infty$, we recover the hard margin classifier again as we don't allow any outliers\n\t\\item If $C\\to 0$, the margin gets really large as we try to maximize the margin without caring about the misclassifications. Also, all points $a_n$ will become support vectors \n\\end{itemize}\n\\subsection{Gaussian Processes}\n\\subsubsection{Essentials of Gaussian distributions}\n\\begin{itemize}\n\t\\item \\textbf{Marginalization property}: if two random variables $x_1$ and $x_2$ are jointly Gaussian distributed, then marginalizing out one variables still leads to a Gaussian\n\t$$p\\left(\\left[\\begin{array}{c}\n\tx_1 \\\\ x_2\n\t\\end{array}\\right]\\right) = \\left(\\left[\\begin{array}{c}\n\tx_1\\\\x_2\n\t\\end{array}\\right]|\\left[\\begin{array}{c}\n\t\\mu_1\\\\\\mu_2\n\t\\end{array}\\right], \\left[\\begin{array}{cc}\n\t\\Sigma_{11} & \\Sigma_{12} \\\\ \\Sigma_{21} & \\Sigma_{22} \n\t\\end{array}\\right]\\right)\\implies\\hspace{1mm} p(x_1) = \\mathcal{N}(\\mu_1, \\Sigma_11),\\hspace{3mm} p(x_2) = \\mathcal{N}(\\mu_2, \\Sigma_22)$$\n\t\\item \\textbf{Conditional property}: if two random variables $x_1$ and $x_2$ are jointly Gaussian distributed, then conditioning one variables on the other still leads to a Gaussian\n\t$$p(x_1|x_2) = \\mathcal{N}(\\mu_{1|2}, \\Sigma_{1|2})$$\n\t\\item \\textbf{Sum property}: Summing two independent Gaussian random variables lead to a new Gaussian variable:\n\t$$x\\sim \\mathcal{N}(\\mu_1, \\Sigma_1) \\text{\\hspace{1mm}and\\hspace{1mm}} y\\sim \\mathcal{N}(\\mu_2, \\Sigma_2) \\implies x+y=z\\sim\\mathcal{N}(\\mu_1+\\mu_2, \\Sigma_1+\\Sigma_2) $$ \n\t\\item \\textbf{Correlation property}: If $\\bm{x}$ is an uncorrelated Gaussian random variable $\\mathcal{N}(\\bm{0}, \\bm{I})$ then $\\bm{y} = \\bm{\\mu} + \\bm{A}\\bm{x}$ is correlated by $\\bm{y}\\sim\\mathcal{N}(\\mu, \\bm{A}\\bm{A}^T)$\n\\end{itemize}\n\\subsubsection{Introduction to Gaussian Processes}\n\\begin{itemize}\n\t\\item In Bayesian linear regression, we assume that the target is distributed as $t=\\bm{\\phi}\\left(\\bm{x}\\right)^T \\bm{w} + \\epsilon$ where $\\epsilon\\sim\\mathcal{N}(0,\\beta^{-1})$. The posterior is also Gaussian distributed: $p(\\bm{w}|\\bm{X},\\bm{t}) = \\mathcal{N}(\\bm{w}|\\bm{m}_N, \\bm{S}_N)$.\n\t\\item When we predict for new points, we use the mean $\\mu_N = \\sum_{n=1}^{N}\\beta \\bm{\\phi}(\\bm{x})^T \\bm{S}_N^{-1} \\bm{\\phi}(\\bm{x}_n)t_n$ % and variance $\\sigma_N^2(\\bm{x})=\\beta^{-1} + \\bm{\\phi}(\\bm{x})^T \\bm{S}_N \\bm{\\phi}(\\bm{x})$\n\t\\item Here we see that we can express the mean by the kernel $k(\\bm{x}_n, \\bm{x}_m) = \\bm{\\phi}(\\bm{x}_n)^T \\bm{S}_N^{-1} \\bm{\\phi}(\\bm{x}_m)$ $\\Rightarrow$ increase expressiveness of Linear Bayesian regression by using more complex kernels\n\t\\item Definition of Gaussian Processes: A Gaussian process is a collection of random variables, any finite number of which is jointly Gaussian distributed\n\t\\item Gaussian Processes represent distributions over random functions!\n\t$$f(\\circ) \\sim \\mathcal{N}(m(\\circ), k(\\circ, \\circ))$$\n\t\\item The function \\textit{evaluated} at a specific point $\\bm{x}$ is a random variable, with $\\mathbb{E}[f(\\bm{x})] = m(\\bm{x})$ and $\\text{cov}(f(\\bm{x}), f(\\bm{x}')) = k(\\bm{x}, \\bm{x}')$ (covariance matrix is the gram matrix $K$)\n\t\\item Thus, for a finite set of points $\\left\\{\\bm{x}_1, ...,\\bm{x}_N\\right\\}$, the random variables $\\left\\{f(\\bm{x}_1), ...,f(\\bm{x}_N)\\right\\}$ are:\n\t$$p\\left(\\begin{bmatrix}\n\tf(\\bm{x}_1)\\\\\n\t\\vdots\\\\\n\tf(\\bm{x}_N)\\\\\n\t\\end{bmatrix}\\right) = \\mathcal{N}\\left(\\begin{bmatrix}\n\tm(\\bm{x}_1)\\\\\n\t\\vdots\\\\\n\tm(\\bm{x}_N)\\\\\n\t\\end{bmatrix}, \\begin{bmatrix}\n\tk(\\bm{x}_1, \\bm{x}_1) & \\cdots & k(\\bm{x}_1, \\bm{x}_N)\\\\\n\t\\vdots & \\ddots & \\vdots\\\\\n\tk(\\bm{x}_N, \\bm{x}_1)& \\cdots & k(\\bm{x}_N, \\bm{x}_N)\\\\\n\t\\end{bmatrix}\\right)$$\n\t\\item Each entry is the sampled function evaluated at point $\\bm{x}$. We can evaluate/sample functions by just using a fine-grained set of points\n\t\\item The kernel has a significant influence on how the functions might look like. When we consider the kernel $k(\\bm{x}_n, \\bm{x}_m) = \\theta_0 \\exp\\left(-\\frac{1}{2\\theta_1}||\\bm{x}_n - \\bm{x}_m||^2\\right) + \\theta_2 + \\theta_3 \\bm{x}_n^T \\bm{x}_m$, we see that:\n\t\\begin{itemize}\n\t\t\\item $\\theta_0$ influences the amplitude of the samples of $f$\n\t\t\\item $\\theta_1$ scale the length of correlation\n\t\t\\item $\\theta_2$ introduces a random bias for sampled $f$ (different bias for every sample)\n\t\t\\item $\\theta_3$ adds a linear component into the samples leading to a up-/down-ward trend\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Regression with Gaussian Processes}\n\\begin{itemize}\n\t\\item We have observed data which we model by $f(\\bm{x}_i) = y(\\bm{x}_i) + \\epsilon$ ($\\epsilon\\sim\\mathcal{N}(0,\\beta^{-1})$)\n\t\\item We can now model $y$ as GP: \n\t$$p\\left(\\begin{bmatrix}\n\ty(\\bm{x}_1)\\\\\n\t\\vdots\\\\\n\ty(\\bm{x}_N)\\\\\n\t\\end{bmatrix}\\right) = \\mathcal{N}\\left(\\bm{0}, \\begin{bmatrix}\n\tk(\\bm{x}_1, \\bm{x}_1) & \\cdots & k(\\bm{x}_1, \\bm{x}_N)\\\\\n\t\\vdots & \\ddots & \\vdots\\\\\n\tk(\\bm{x}_N, \\bm{x}_1)& \\cdots & k(\\bm{x}_N, \\bm{x}_N)\\\\\n\t\\end{bmatrix}\\right)$$\n\t\\item Then, $f(\\circ)$ is also a GP since $\\bm{f} = \\bm{y} + \\epsilon$ ($\\bm{f}\\sim \\mathcal{N}(\\bm{0}, K(\\bm{X}, \\bm{X}) + \\beta^{-1}\\bm{I})$)\n\t\\item For new test data points, we can predict them by using:\n\t$$p\\left(\\begin{bmatrix}\n\t\\bm{f}\\\\\n\t\\bm{f}^{*}\\\\\n\t\\end{bmatrix}\\right) = \\mathcal{N}\\left(\\bm{0}, \\begin{bmatrix}\n\tK(\\bm{X}, \\bm{X}) + \\beta^{-1}\\bm{I} & K(\\bm{X},\\bm{X}^{*})\\\\\n\tK(\\bm{X}^{*},\\bm{X}) & K(\\bm{X}^{*}, \\bm{X}^{*}) + \\beta^{-1}\\bm{I}\\\\\n\t\\end{bmatrix}\\right)$$\n\t\\item The more points we see the more certain our predictions gets\n\t\\item Kernel parameters can be chosen based on MLE on training observations\n\\end{itemize}"
  },
  {
    "path": "Machine_Learning_1/ml_linear_classification.tex",
    "content": "\\section{Linear classification}\n\\begin{itemize}\n\t\\item Input $\\bm{x}=\\left(x_1, x_2, ..., x_D\\right)^T$ with $\\bm{x}\\in\\mathbb{R}^D$.\n\t\\item Target $t\\in\\left\\{C_1, C_2, ..., C_K\\right\\}$ with $K$ classes (one-hot representation)\n\t\\item Goal: divide input space $\\mathbb{R}^D$ into $K$ decision regions $R_k$ with $k=1,...,K$\n\t\\item Boundaries of decision regions are called \\textit{decision boundaries/surfaces}\n\t\\begin{itemize}\n\t\t\\item Linear classification only considers \\textit{linear} decision boundaries $\\Rightarrow$ $D-1$ dimensional hyperplanes\n\t\t\\item A dataset is \\textit{linearly separable}, if its classes can be exactly separated by linear decision boundaries\n\t\\end{itemize}\n\t\\item First, we derive the optimal solution for decision boundaries in general (3.1 Decision Theory), and then look at different models for deriving such solutions (3.2-3.4) \n\\end{itemize}\n\\subsection{Decision Theory For Classification}\n\\begin{itemize}\n\t\\item For every observed datapoint: label/ground truth $t_n=C_j$, prediction $t_n=C_k$\n\t\\item Confusion matrix (row: GT class, columns: prediction region/class)\n\t$$\\begin{blockarray}{ccccc}\n\t& R_1 & R_2 & \\dots & R_K \\\\\n\t\\begin{block}{c(cccc)}\n\tC_1 & 6 & 1 & \\dots & 0 \\\\\n\tC_2 & 4 & 2 & \\dots & 3 \\\\\n\t\\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\\n\tC_K & 1 & 0 & \\dots & 5 \\\\\n\t\\end{block}\n\t\\end{blockarray}$$\n\t\\begin{itemize}\n\t\t\\item The elements on the diagonal represent the correctly classified examples\n\t\t\\item Try to minimize misclassified examples (off-diagonal elements), or the probability of a mistake: $p\\left(\\text{mistake}\\right) = 1 - \\sum\\limits_{k=1}^{K}p(\\bm{x}\\in R_k, C_k)$\n\t\\end{itemize}\n\t\\item Assign $x$ to class $C_k$ if $\\forall j\\neq k: p\\left(\\bm{x}, t=C_k\\right) > p\\left(\\bm{x}, t=C_j\\right)$ $\\Rightarrow$ $p\\left(C_k|\\bm{x}\\right) > p\\left(C_j|\\bm{x}\\right)$\n\t\\item Optimal decision boundary where $p\\left(C_k|\\bm{x}\\right) = p\\left(C_j|\\bm{x}\\right)$\n\t\\item Problem: \\textit{class imbalance} $\\Rightarrow$ possible solution: weighted loss for balancing the importance of each class\n\t\\item For imbalanced datasets, assign $x$ to $C_k$ if $\\sum\\limits_{j=1}^{K}L_{jk}p\\left(x,C_j\\right)$ is minimal \n\t\\begin{itemize}\n\t\t\\item $L_{jk}$ is misclassification weight matrix where $L_{ii}=0$\n\t\t\\item Example for dataset with 1\\% cancer patients: \n\t\t$$L = \\begin{blockarray}{ccc}\n\t\t\\text{pred. cancer} & \\text{pred. healthy} & \\\\\n\t\t\\begin{block}{(cc)c}\n\t\t0 & 1000 & \\text{true cancer} \\\\\n\t\t1 & 0 & \\text{true healthy} \\\\\n\t\t\\end{block}\n\t\t\\end{blockarray}$$\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Probabilistic generative models}\n\\begin{itemize}\n\t\\item Model the class conditional densities $p\\left(x|C_k\\right)$ \\textbf{and} the prior class probabilities $p(C_k)$ to compute posterior probabilities $p\\left(C_k|x\\right)$ (as we know from Decision Theory that at $p\\left(C_k|x\\right)=p\\left(C_j|x\\right)$ are the optimal decision boundaries)\n\t\\item For $K=2$, the posterior is: $p\\left(C_1|\\bm{x}\\right) = \\frac{p\\left(\\bm{x}|C_1\\right)p\\left(C_1\\right)}{p\\left(\\bm{x}\\right)} = \\frac{p\\left(\\bm{x}|C_1\\right)p\\left(C_1\\right)}{p\\left(\\bm{x}|C_1\\right)p\\left(C_1\\right) + p\\left(\\bm{x}|C_2\\right)p\\left(C_2\\right)}$\n\t\\item We can simplify the previous equation by using the sigmoid function:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tp\\left(C_1|\\bm{x}\\right) & = \\frac{1}{1 + \\frac{p\\left(\\bm{x}|C_2\\right)p\\left(C_2\\right)}{p\\left(\\bm{x}|C_1\\right)p\\left(C_1\\right)} } = \\frac{1}{1+\\exp\\left(-a\\right)} \\text{\\hspace{5mm}where\\hspace{5mm}} a=\\ln\\frac{\\sigma}{1-\\sigma} = \\ln \\frac{p\\left(\\bm{x}|C_2\\right)p\\left(C_2\\right)}{p\\left(\\bm{x}|C_1\\right)p\\left(C_1\\right)}\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item For general $K$: $p\\left(C_k|\\bm{x}\\right) = \\frac{p\\left(\\bm{x}|C_k\\right)p\\left(C_k\\right)}{\\sum_{j=1}^{K}p\\left(\\bm{x}|C_j\\right)p\\left(C_j\\right)} = \\frac{\\exp(a_k)}{\\sum_{j=1}^{K} \\exp(a_j)}$ with $a_k = \\ln \\left[ p\\left(\\bm{x}|C_k\\right)p\\left(C_k\\right)\\right]$ (softmax)\n\t\\item In the special case of $K=2$: $a=a_1-a_2$\n\\end{itemize}\n\\subsubsection{Continuous inputs}\n\\begin{itemize}\n\t\\item Assume that the class-conditional  densities are Gaussian:\n\t$$p\\left(\\bm{x}|C_k\\right) = \\mathcal{N}\\left(\\bm{x}|\\bm{\\mu}_k,\\bm{\\Sigma}_k\\right) = \\frac{1}{\\left(2\\pi\\right)^{D/2}}\\frac{1}{|\\bm{\\Sigma}_k|^{1/2}}\\exp\\left\\{\\frac{1}{2}\\left(\\bm{x}-\\bm{\\mu}_k\\right)^T\\bm{\\Sigma}^{-1}\\left(\\bm{x}-\\bm{\\mu}_k\\right)\\right\\}$$\n\t\\item We assume that all classes share the same covariance: $\\bm{\\Sigma}_k = \\bm{\\Sigma}$\\\\$\\Rightarrow$ We are able to apply \\textbf{linear discriminant analysis} (otherwise, decision boundaries would be quadratic)\n\t\\item Determining posterior for $K=2$:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\ta & = \\ln \\frac{p\\left(\\bm{x}|C_2\\right)p\\left(C_2\\right)}{p\\left(\\bm{x}|C_1\\right)p\\left(C_1\\right)} = \\ln \\mathcal{N}\\left(\\bm{x}|\\bm{\\mu}_1,\\bm{\\Sigma}_1\\right) - \\ln \\mathcal{N}\\left(\\bm{x}|\\bm{\\mu}_2,\\bm{\\Sigma}_2\\right) + \\ln \\frac{p\\left(C_1\\right)}{p\\left(C_2\\right)}\\\\\n\t\t\t& = \\left(\\bm{\\mu}_1 - \\bm{\\mu}_2\\right)^T\\bm{\\Sigma}^{-1}\\bm{x} - \\frac{1}{2}\\bm{\\mu}_1^T\\bm{\\Sigma}^{-1}\\bm{\\mu}_1 + \\frac{1}{2}\\bm{\\mu}_2^T\\bm{\\Sigma}^{-1}\\bm{\\mu}_2 + \\ln \\frac{p\\left(C_1\\right)}{p\\left(C_2\\right)}\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Thus, the posterior can be expressed by\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tp(C_1|\\bm{x})=\\sigma\\left(\\bm{w}^Tx+w_0\\right) \\text{\\hspace{3mm}where\\hspace{2mm}} & \\bm{w} = \\bm{\\Sigma}^{-1}\\left(\\bm{\\mu}_1 - \\bm{\\mu}_2\\right)\\\\\n\t\t\t& w_0 = -\\frac{1}{2}\\bm{\\mu}_1^T\\bm{\\Sigma}^{-1}\\bm{\\mu}_1 + \\frac{1}{2}\\bm{\\mu}_2^T\\bm{\\Sigma}^{-1}\\bm{\\mu}_2 + \\ln \\frac{p\\left(C_1\\right)}{p\\left(C_2\\right)}\n\t\t\\end{split}\n\t\\end{equation*} \n\t\\item For general $K$, we get $p\\left(C_k|\\bm{x}\\right) = \\frac{\\exp\\left(a_k\\left(\\bm{x}\\right)\\right)}{\\sum_{j=1}^{K}\\exp\\left(a_j\\left(\\bm{x}\\right)\\right)}$ with \n\t\\begin{equation*}\n\t\\begin{split}\n\ta_k\\left(\\bm{x}\\right) = \\ln \\left[p\\left(\\bm{x}|C_k\\right)\\cdot p\\left(C_k\\right)\\right]=\\bm{w}_k^Tx+w_{k0} \\text{\\hspace{3mm}where\\hspace{2mm}} & \\bm{w}_k = \\bm{\\Sigma}^{-1}\\bm{\\mu}_k\\\\\n\t& w_{k0} = -\\frac{1}{2}\\bm{\\mu}_k^T\\bm{\\Sigma}^{-1}\\bm{\\mu}_k + \\ln p\\left(C_k\\right)\n\t\\end{split}\n\t\\end{equation*} \n\t\\item Decision boundaries are at $p\\left(C_k|\\bm{x}\\right) = p\\left(C_j|\\bm{x}\\right)$ $\\Rightarrow$ $a_k = a_j\\Rightarrow$ Linear decision boundaries!\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.7\\textwidth]{figures/linear_classification_pgm.png}\n\t\t\\caption{Left: Gaussian class-conditional densities, Right: corresponding posterior with sigmoid}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{Maximum likelihood solution for $K=2$}\n\\begin{itemize}\n\t\\item Binary targets $t_n\\in\\left\\{0,1\\right\\}$ ($1$ for $C_1$, $0$ for $C_0$)\n\t\\item We use maximum likelihood to find optimal solution for $\\bm{\\mu}_k$, $\\bm{\\Sigma}$ and priors $p\\left(C_k\\right)$\n\t\\item For $K=2$, the priors are denoted by $p\\left(C_1\\right) = q$ and $p\\left(C_2\\right) = 1-q$\n\t\\item If $\\bm{x}_n$ has target $t_n=1$: $p\\left(\\bm{x}_n, C_1\\right) = p\\left(\\bm{x}_n|C_1\\right)p\\left(C_1\\right) =q\\mathcal{N}\\left(\\bm{x}_n|\\bm{\\mu}_1, \\bm{\\Sigma}\\right)$\n\t\\item If $\\bm{x}_n$ has target $t_n=0$: $p\\left(\\bm{x}_n, C_2\\right) = p\\left(\\bm{x}_n|C_2\\right)p\\left(C_2\\right) =(1-q)\\mathcal{N}\\left(\\bm{x}_n|\\bm{\\mu}_2, \\bm{\\Sigma}\\right)$\n\t\\item Combined likelihood: $p\\left(\\bm{t}, \\bm{X}|q,\\bm{\\mu}_1,\\bm{\\mu}_2,\\bm{\\Sigma}\\right) = \\prod\\limits_{n=1}^{N} \\left[q\\mathcal{N}\\left(\\bm{x}_n|\\bm{\\mu}_1, \\bm{\\Sigma}\\right)\\right]^{t_n}\\left[(1-q)\\mathcal{N}\\left(\\bm{x}_n|\\bm{\\mu}_2, \\bm{\\Sigma}\\right)\\right]^{1-t_n}$\n\t\\item Log-likelihood: $$\\ln p\\left(\\bm{t}, \\bm{X}|q,\\bm{\\mu}_1,\\bm{\\mu}_2,\\bm{\\Sigma}\\right) = \\sum\\limits_{n=1}^{N}t_n \\ln q + t_n \\ln \\mathcal{N}\\left(\\bm{x}_n|\\bm{\\mu}_1, \\bm{\\Sigma}\\right) + (1 - t_n) \\ln \\left(1 - q\\right) + \\left(1 - t_n\\right) \\ln \\mathcal{N}\\left(\\bm{x}_n|\\bm{\\mu}_2, \\bm{\\Sigma}\\right) $$\n\t\\item Estimate for $q$: $\\frac{\\partial}{\\partial q} \\ln p\\left(\\bm{t}, \\bm{X}|q,\\bm{\\mu}_1,\\bm{\\mu}_2,\\bm{\\Sigma}\\right) = \\sum\\limits_{n=1}^{N} \\frac{t_n}{q} - \\frac{1 - t_n}{1 - q} = \\sum\\limits_{n=1}^{N} \\frac{t_n - q}{q\\left(1 - q\\right)} \\Rightarrow q_{\\text{ML}} = \\frac{1}{N}\\sum\\limits_{n=1}^{N} t_n = \\frac{N_1}{N}$\\\\\n\t- Thus, the estimate of $p(C_1)$ is the proportion of samples that are assigned to class 1\n\t\\item Estimate for $\\bm{\\mu_1}$: $\\bm{\\mu}_{1,\\text{ML}} = \\frac{1}{N_1}\\sum\\limits_{n=1}^{N} t_n \\bm{x}_n$, $\\bm{\\mu}_{2,\\text{ML}} = \\frac{1}{N_2}\\sum\\limits_{n=1}^{N} \\left(1-t_n\\right) \\bm{x}_n$\\\\\n\t- Thus, the estimate of $\\bm{\\mu}_k$ is the mean of the samples assigned to class $k$\n\t\\item Estimate for $\\bm{\\Sigma}$: \n\t$$\\bm{\\Sigma}_{\\text{ML}} = \\frac{N_1}{N}\\underbrace{\\left[\\frac{1}{N_1} \\sum\\limits_{n=1}^{N} t_n\\left(\\bm{x} - \\bm{\\mu}_{1,\\text{ML}}\\right)\\left(\\bm{x} - \\bm{\\mu}_{1,\\text{ML}}\\right)^T\\right]}_{\\text{sample covariance of class 1}} + \\frac{N_2}{N}\\underbrace{\\left[\\frac{1}{N_2} \\sum\\limits_{n=1}^{N} (1-t_n)\\left(\\bm{x} - \\bm{\\mu}_{2,\\text{ML}}\\right)\\left(\\bm{x} - \\bm{\\mu}_{2,\\text{ML}}\\right)^T\\right]}_{\\text{sample covariance of class 2}}$$\n\t- Thus, the estimate of $\\bm{\\Sigma}$ is a weighted average (based on number of samples for each class) of the class' sample covariance\\\\\n\t- Note that this assumes a similar covariance matrix for every class cluster. If this is not the case, the estimation gives bad results (see Figure~\\ref{img:linear_discriminative_analysis_different_cov})\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.2\\textwidth]{figures/linear_discriminant_analysis_different_cov.png}\n\t\t\\caption{Example for three classes with different covariance matrices $\\bm{\\Sigma}_k^{-1}$. Linear discriminant analysis fails as it estimates a weighted sum of the sample covariance, and the distribution of green class significantly differs from the other two. The resulting estimate would tend to be a circle for each class instead of the drawn ellipses.}\n\t\t\\label{img:linear_discriminative_analysis_different_cov}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{Discrete inputs}\n\\begin{itemize}\n\t\\item In contrast to the previous subsections, we now assume that $\\bm{x}_n \\in\\left\\{0,1\\right\\}^D$ and is therefore discrete\n\t\\item As we know have no PDF anymore, we need $2^D - 1$ parameters per class to guarantee a perfect fit\n\t\\item However, if we use the Naive Bayes assumption (feature values are treated as independent given $C_k$), we reduce the number of features to $D$ per class (by using the Bernoulli distribution):\n\t$$p\\left(\\bm{x}|C_k\\right) = \\prod\\limits_{i=1}^{D}p\\left(x_i|C_k\\right) = \\prod\\limits_{i=1}^{D} \\pi_{ki}^{x_i}\\left(1 - \\pi_{ki}\\right)^{1-x_i}$$\n\t\\item We can apply this simplification to rewrite $a_k$:\n\t$$a_k = \\ln p\\left(x|C_k\\right) + \\ln p\\left(C_k\\right)  = \\sum\\limits_{i=1}^{D}\\left[x_i\\ln \\pi_{ki} + (1 - x_i)\\ln (1 - \\pi_{ki}) \\right] + \\ln p\\left(C_k\\right) = \\bm{x}^T\\bm{w} + w_0 $$\n\t where $w_i=\\ln \\frac{\\pi_{ki}}{1 - \\pi_{ki}}$ and $w_0 = \\ln p\\left(C_k\\right) + \\sum_{i=1}^{D}\\ln \\left(1-\\pi_{ki}\\right)$ \n\\end{itemize}\n\\subsection{Discriminant functions}\n\\begin{itemize}\n\t\\item Direct mapping of input to target (similar to regression)%$t=y(x,w)$\n\t\\item We use $y\\left(\\bm{x},\\bm{\\tilde{w}}\\right) = f\\left(\\bm{\\tilde{w}}^T\\bm{\\phi}\\right)$, where $f$ is the activation function and might be non-linear \n\t\\item The decision boundary is defined at a point where $y\\left(\\bm{x},\\bm{\\tilde{w}}\\right) = \\text{const}_1$. As $y$ represents the application of $f$, we can rewrite it as $\\bm{\\tilde{w}}^T\\bm{\\phi} = \\text{const}_2$\n\t\\item We first review the application of the case of two classes, and then try to find a solution for multiple classes\n\\end{itemize}\n\\subsubsection{Discriminant functions for two classes}\n\\begin{itemize}\n\t\\item For a two class problem, we set the decision boundary to 0 as we are still able to shift it by $w_0$\n\t\\item If $y\\left(\\bm{x},\\bm{\\tilde{w}}\\right)\\geq 0$, the input $\\bm{x}$ is assigned to class $C_1$, whereas if $y\\left(\\bm{x},\\bm{\\tilde{w}}\\right)< 0$, the class is $C_2$ $\\Rightarrow w_0$ is considered as the activation threshold\n\t\\item To determine how the weights $\\bm{\\tilde{w}}$ influence this classification, we assume two points $\\bm{x}_a$ and $\\bm{x}_b$ on the decision boundary $\\Rightarrow$ $y\\left(\\bm{x}_a\\right) = y\\left(\\bm{x}_b\\right) = 0 \\Rightarrow \\bm{w}^T(\\bm{x}_a - \\bm{x}_b) = 0$ (see Figure~\\ref{img:discriminant_function_two_classes})\n\t\\item Hence, $\\bm{w}$ is orthogonal to every vector lying within the decision surface, so that $\\bm{w}$ determines the orientation of the surface\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/discriminant_function_two_classes.png}\n\t\t\\caption{Illustration of the geometry of a linear discriminant function in two dimensions. The decision surface, shown in red, is perpendicular to $\\bm{w}$, and its displacement from the origin is controlled by the bias parameter $w_0$. Also, the signed orthogonal distance of a general point $\\bm{x}$ from the decision surface is given by $y(\\bm{x})/ \\bm{w}$ .}\n\t\t\\label{img:discriminant_function_two_classes}\n\t\\end{figure}\n\t\\item So, we can express every point by the summation of a point on the decision surface and the weights: $$\\bm{x} = \\bm{x}_{\\perp} + r\\frac{\\bm{w}}{||\\bm{w}||}$$\n\t\\item Applied in $y$, we get: $y\\left(\\bm{x}\\right) = \\bm{w}^T \\bm{x} + w_0 = \\bm{w}^T \\bm{x}_{\\perp} + w_0 + r\\frac{\\bm{w}^T\\bm{w}}{||\\bm{w}||} = r ||\\bm{w}|| \\Rightarrow $\n\t\\item So, the distance between a point $\\bm{x}$ and the decision surface is $r = \\frac{y\\left(\\bm{x}\\right)}{||\\bm{w}||}$\n\\end{itemize}\n\\subsubsection{Discriminant functions for multiple classes}\n\\begin{itemize}\n\t\\item K-class discriminant: $\\bm{y}_k (\\bm{x}) = \\bm{w}_k^T + w_{k0}$\n\t\\item Assign $\\bm{x}$ to $C_k$ if $y_k(\\bm{x})>y_j(\\bm{x})$ for all $j\\neq k$\n\t\\item Thus, the decision boundary between $\\mathcal{R}_k$ and $\\mathcal{R}_j$ is determined by: $y_k(\\bm{x})=y_j(\\bm{x})$\n\t\\item Note that decision regions of linear discriminant functions are convex (if two points are in $\\mathcal{R}_k$, then all points between those are also in the same region $\\mathcal{R}_k$)\n\\end{itemize}\n\\subsubsection{Least squares discriminant}\n\\begin{itemize}\n\t\\item Consider $\\bm{t}_n$ as one-hot vector. We try to learn a function $y_k(\\bm{x}, \\bm{\\tilde{w}}_k)$ for every class $k$ that maps $\\bm{x}$ to its corresponding value in the one-hot vector (basically regression task)\n\t\\item For shorter notation, we write $\\bm{y}(\\bm{x}) = \\bm{\\tilde{W}}\\bm{\\tilde{x}}$ to combine all classes and weights into a single operation\n\t\\item As before: assign $\\bm{x}$ to class $C_k$ if $k=\\arg\\max_j y_j (\\bm{x})$\n\t\\item The error function is the sum-of-squares: $$E_D(\\bm{\\tilde{W}}) = \\frac{1}{2} \\text{Tr}\\left[\\left(\\bm{\\tilde{X}}\\bm{\\tilde{W}}-\\bm{T}\\right)^T\\left(\\bm{\\tilde{X}}\\bm{\\tilde{W}}-\\bm{T}\\right)\\right] = \\frac{1}{2}\\sum\\limits_{n=1}^{N}\\sum\\limits_{k=1}^{K}\\sum\\limits_{d=1}^{D}\\left(\\tilde{X}_{nd}\\tilde{W}_{dk}-\\tilde{T}_{nd}\\right)$$\n\t\\item Minimizing this error leads to:  $\\bm{\\tilde{W}}_{\\text{LS}} = \\left(\\bm{\\tilde{X}}^T \\bm{\\tilde{X}}\\right)^{-1}\\bm{\\tilde{X}}^T \\bm{\\tilde{T}}$\n\t\\item But: there are many problems with least squared errors\n\t\\begin{itemize}\n\t\t\\item The decision boundaries are very sensitive to outliers (try to minimizes \\textit{mean} error to one-hot vector)\n\t\t\\item For $K>2$, some decision regions become very small or are even completely ignored (also called masking)\n\t\t\\item components of $\\bm{y}_{\\text{LS}}$ are not probabilities and can be outside the interval $\\left[0,1\\right]$\n\t\\end{itemize}\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.6\\textwidth]{figures/discriminant_function_least_squares_problem.png}\n\t\t\\caption{Illustration of the problems with least squared error discriminant}\n\t\t\\label{img:discriminant_function_least_squares_problem}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{Perceptron}\n\\begin{itemize}\n\t\\item For the perceptron, we use the step function as activation function:\n\t$$y\\left(\\bm{x}\\right) = f\\left(\\bm{w}^T\\bm{\\phi}(\\bm{x})\\right) \\text{\\hspace{5mm} where \\hspace{5mm}} f(a) = \\begin{cases}\n\t1 & \\text{ if } a\\geq 0\\\\\n\t-1 & \\text{ if } a <0\n\t\\end{cases}$$\n\t\\item Thus, assign $\\bm{x}$ to class $C_1$ if $\\bm{w}^T\\bm{\\phi}(\\bm{x})\\geq0$, otherwise $C_2$\n\t\\item The goal is now to find a $\\bm{w}$ such that $\\bm{w}^T\\bm{\\phi}(\\bm{x})t_n\\geq 0$ ($t_n \\in \\left\\{1,-1\\right\\}$)\n\t\\item We can define the error of a perceptron based on the set of misclassified examples $\\mathcal{M}$: \\\\$E_P(\\bm{w})=-\\sum\\limits_{n\\in\\mathcal{M}}\\bm{w}^T \\bm{\\phi}(\\bm{x}_n)t_n = \\sum\\limits_{n\\in\\mathcal{M}}E_n(\\bm{w})$\n\t\\item Use Stochastic Gradient Descent (SGD) for each misclassified $\\bm{x}_n$:\\\\ $$\\bm{w}^{\\tau+1} = \\bm{w}^{\\tau} - \\eta \\bigtriangledown^T E_n(\\bm{w}) = \\bm{w}^{\\tau} + \\eta \\bm{\\phi}(\\bm{x}_n) t_n$$\n\t\\item If $\\bm{X}$ is linearly separable, SGD will converge\n\t\\item However, there are some problems with the perceptron algorithm:\n\t\\begin{itemize}\n\t\t\\item Perceptron only works for 2 classes\n\t\t\\item There might be many optimal solutions, so that the exact outcome depends on initialization of $\\bm{w}$ and order of data that are used in SGD\n\t\t\\item If dataset is not linearly separable, the perceptron algorithm will not converge\n\t\t\\item Based on linear combination of fixed basis functions \n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Usage of Basis functions}\n\\begin{itemize}\n\t\\item If the data in the input space is not linearly separable, we can use basis functions (that might be non-linear) to transform them into a new space, where they can be linearly separated!\n\t\\item However, prior knowledge is required for this step as the general data distribution must be known and how to convert it into a linearly separable space. This step is especially hard/not possible for high dimensions\n\\end{itemize}\n\\subsection{Probabilistic discriminative models}\n\\begin{itemize}\n\t\\item Instead of specifying the class-conditional probabilities $p\\left(\\bm{x}|C_k\\right)$ and applying maximum likelihood to find the best parameters, we can try to explicitly model the posterior class probability $p\\left(C_k|\\bm{x}\\right)$ and find its distribution $\\Rightarrow$ posteriors are non-linear functions with a linear function of $\\bm{\\phi}$ as input: $p\\left(C_k|\\bm{\\phi}, \\bm{w}\\right) = f(\\bm{w}_k^T\\bm{\\phi})$\n\t\\item The implicit method of finding the parameters of a generalized model is by fitting $p\\left(\\bm{x}|C_k\\right)$ and $p\\left(C_k\\right)$ representing a generative model (generate synthetic data from $p(\\bm{x})$)\n\\end{itemize}\n\\subsubsection{Logistic regression for two classes}\n\\begin{itemize}\n\t\\item Logistic regression uses the sigmoid function to model the posterior:\\\\\n\t$$p\\left(C_1|\\bm{\\phi},\\bm{w}\\right) = y\\left(\\bm{\\phi}\\right) = \\sigma\\left(\\bm{w}^T\\bm{\\phi}\\right), p\\left(C_2|\\bm{\\phi},\\bm{w}\\right) = 1 - p\\left(C_1|\\bm{\\phi},\\bm{w}\\right)$$\n\t\\item For inference/classification, take the class with the higher probability ($>0.5$) $\\Rightarrow$ Decision boundaries: $\\bm{w}\\bm{\\phi}(\\bm{x}) = 0$\n\t\\item If $\\bm{w}\\in\\mathbb{R}^M$, we use $M$ number of parameters (compared to $M(M+5)/2 + 1$ for modeling a Gaussian multivariate distribution)\n\t\\item Use maximum likelihood to determine the parameters of the logistic regression model\n\t\\item Conditional likelihood: $p\\left(\\bm{t}|\\bm{X},\\bm{w}\\right) = \\prod\\limits_{n=1}^{N} p\\left(t_n | \\bm{x}_n,\\bm{w}\\right) = \\prod\\limits_{n=1}^{N} y_n^{t_n}\\left(1 - y_n\\right)^{1-t_n}$\n\t\\item Maximizing the likelihood is equal to minimizing the cross-entropy loss:\n\t$$E(\\bm{w}) = -\\ln p\\left(\\bm{t}|\\bm{X},\\bm{w}\\right) = -\\sum\\limits_{n=1}^{N} \\left[t_n \\ln y_n + (1 - t_n) \\ln (1 - y_n)\\right] =\\sum\\limits_{n=1}^{N}E_n(\\bm{w})$$\n\t\\item The loss $E(\\bm{w})$ is convex (has a single, \\textbf{unique minimum}), but no closed-form solution exists ($y_n = \\sigma(\\bm{w}^T \\phi_n)$ is nonlinear in $\\bm{w}$) $\\Rightarrow$ Use SGD/...\n\t\\item For taking the gradient, we can make use of the property of the sigmoid function: \n\t$$\\frac{\\partial E_n(\\bm{w})}{\\partial w_j} = \\frac{\\partial E_n(\\bm{w})}{\\partial y_n} \\frac{\\partial y_n}{\\partial w_j} = \\left[-\\frac{t_n}{y_n}+\\frac{1 - t_n}{1 - y_n}\\right] \\cdot \\left[\\sigma(\\bm{w}^T\\bm{\\phi}(\\bm{x}_n))\\left(1 - \\sigma(\\bm{w}^T\\bm{\\phi}(\\bm{x}_n))\\right) \\phi_j(\\bm{x}_n)\\right] = (y_n - t_n)\\phi_j(\\bm{x}_n)$$\n\t\\item Update rule (SGD): $\\bm{w}^{\\tau + 1} = \\bm{w}^{\\tau} - \\eta \\triangledown^{\\tau} E_n(\\bm{w})^{\\tau}= \\bm{w}^{\\tau} - \\eta (y_n - t_n)\\bm{\\phi}\\left(\\bm{x}_n\\right)$\n\t\\item If $\\eta$ too large: no convergence. If $\\eta$ too small: very slow convergence\n\t\\item Converged $\\bm{w}^*$ minimizes the loss $E(\\bm{w})$\n\\end{itemize}\n\\subsubsection{Iterative reweighted least squares}\n\\begin{itemize}\n\t\\item Also called the \\textit{Newton-Raphson iterative optimization scheme}\n\t\\item We use a \\textbf{quadratic approximation} instead of a linear at $E(\\bm{w}^{\\tau})$ as the difference between the difference of the loss function to a second order polynomial is quite small (find $E(\\bm{w}^{\\tau+1})$ that minimizes our quadratic approximation $\\Rightarrow$ no learning rate)\n\t\\item New update rule: $$\\bm{w}^{\\tau} = \\bm{w}^{\\tau-1} - \\bm{H}^{-1} \\triangledown E(\\bm{w}^{\\tau - 1})$$ where $\\bm{H}$ is the Hessian matrix whose elements comprise the second derivatives of $E(\\bm{w})$: $H_{ij}=\\frac{\\partial E(\\bm{w})}{\\partial w_i \\partial w_j}$ ($\\bm{H}$ is symmetric!)\n\t\\item Derived from the previous section, the gradient for all $N$ data-points is \n\t$$\\triangledown E(\\bm{w}) = \\sum\\limits_{n=1}^{N} (y_n - t_n)\\bm{\\phi}\\left(\\bm{x}_n\\right) = \\bm{\\Phi}^T\\left(\\bm{y} - \\bm{t}\\right)$$\n\t\\item The elements of the Hessian derive this by a second parameter again:\n\t$$H_{ij} = \\frac{\\partial E(\\bm{w})}{\\partial w_i \\partial w_j} = \\frac{\\partial}{\\partial w_i}\\sum\\limits_{n=1}^{N} (y_n - t_n)\\phi_j \\left(\\bm{x}_n\\right) = \\sum\\limits_{n=1}^{N} \\phi_j\\left(\\bm{x}_n\\right) \\frac{\\partial y_n}{\\partial w_i} = \\sum\\limits_{n=1}^{N} y_n (1 - y_n) \\phi_i(\\bm{x}_n)\\phi_j(\\bm{x}_n)$$\n\t\\item The overall Hessian matrix is therefore \n\t$$\\bm{H} = \\sum\\limits_{n=1}^{N} y_n (1 - y_n) \\bm{\\phi}(\\bm{x}_n)\\bm{\\phi}(\\bm{x}_n)^T = \\bm{\\Phi}^T \\bm{R}\\bm{\\Phi}$$\n\twhere $R_{nn} = y_n(1-y_n)$, and otherwise $R_{nm} = 0$ for $n\\neq m$\n\t\\item Applying this term in the update equation leads to:\n\t$$\\bm{w}^{(\\tau)} = \\bm{w}^{(\\tau - 1)} - \\left(\\bm{\\Phi}^T \\bm{R} \\bm{\\Phi}\\right)^{-1}\\bm{\\Phi}^T\\left(\\bm{y} - \\bm{t}\\right) = \\left(\\bm{\\Phi}^T \\bm{R} \\bm{\\Phi}\\right)^{-1} \\bm{\\Phi}^T \\bm{z} \\text{\\hspace{5mm}where\\hspace{5mm}} \\bm{z} = \\bm{\\Phi}\\bm{w}^{(\\tau-1)}-\\bm{R}^{-1}(\\bm{y}-\\bm{t})$$\n\t\\item Note the similarity to the maximum likelihood solution $\\bm{w}_{\\text{ML}} = \\left(\\bm{\\Phi}^T \\bm{\\Phi}\\right)^{-1} \\bm{\\Phi}^T\\bm{t}$\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/logistic_regression_sgd_iterat.png}\n\t\t\\caption{Illustration of SGD (green) and Newton Raphson (red). SGD always goes in the direction of the steepest gradient and is therefore slower than Newton-Raphson.}\n\t\t\\label{img:logistic_regression_sgd_iterat}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{Logistic regression for multiple classes}\n\\begin{itemize}\n\t\\item Our posterior distribution is now given by a softmax:\n\t$$p(C_k|\\bm{\\phi}, \\bm{w}_1, ..., \\bm{w}_K) = y_k\\left(\\bm{\\phi}\\right) = \\frac{\\exp\\left(a_k\\right)}{\\sum_{j=1}^{K}\\exp\\left(a_j\\right)} \\text{\\hspace{5mm}where\\hspace{5mm}} a_k=\\bm{w}_k^T\\bm{\\phi}(\\bm{x})$$\n\t\\item The derivation by a element $a_j$ is $\\frac{\\partial y_k}{\\partial a_j}=y_k\\left(\\mathbb{I}\\left(k=j\\right) - y_j\\right)$\n\t\\item We use again maximum likelihood to determine the optimal parameters\n\t\\item Conditional likelihood $p\\left(\\bm{T}|\\bm{X},\\bm{w}_1,...\\bm{w}_K\\right) = \\prod\\limits_{n=1}^{N}\\prod\\limits_{k=1}^{K}p(C_k|x_k, \\bm{w}_1, ..., \\bm{w}_K) = \\prod\\limits_{n=1}^{N}\\prod\\limits_{k=1}^{K} y_{k}\\left(\\bm{\\phi}_n\\right)^{t_{nk}}$\n\t\\item Taking the negative logarithm gives the \\textit{cross-entropy} loss function for the multiclass classification problem\n\t$$E\\left(\\bm{w}_1,...,\\bm{w}_K\\right) = - \\ln p\\left(\\bm{T}|\\bm{w}_1,...,\\bm{w}_K\\right) = -\\sum\\limits_{n=1}^{N}\\sum\\limits_{k=1}^{K} t_{nk} \\ln y_{nk}$$\n\t\\item To minimize the function by SGD or Newton Raphson, we need to take the derivate:\n\t$$\\triangledown_{\\bm{w}_j} E\\left(\\bm{w}_1, ...,\\bm{w}_K\\right) = \\sum\\limits_{n=1}^{N}\\left(y_{nj} - t_{nj}\\right) \\bm{\\phi}\\left(\\bm{x}_n\\right)$$\n\t\\item For Newton Raphson/Iterative reweighted least squares, we also need the Hessian matrix:\n\t$$\\frac{\\partial}{\\partial \\bm{w}_k}\\frac{\\partial}{\\partial \\bm{w}_j}E\\left(\\bm{w}_1,...,\\bm{w}_K\\right) = \\sum\\limits_{n=1}^{N} y_{nk}\\left(\\mathbb{I}\\left(k=j\\right) - y_{nj}\\right) \\bm{\\phi}_n \\bm{\\phi}_n^T$$\n\t\\item Decision boundaries at $\\left(\\bm{w}_k^*\\right)^T \\bm{\\phi}\\left(\\bm{x}'\\right) = \\left(\\bm{w}_j^*\\right)^T \\bm{\\phi}\\left(\\bm{x}'\\right)$\n\\end{itemize}"
  },
  {
    "path": "Machine_Learning_1/ml_linear_regression.tex",
    "content": "\\section{Linear Regression}\n\n\\subsection{Basic approaches}\n\\subsubsection{Maximum likelihood}\n\\begin{itemize}\n\t\\item Given a dataset $D=(\\bm{x}_1, \\bm{x}_2, ..., \\bm{x}_N)$ of $N$ independent observations\n\t\\item The likelihood of the dataset given the model parameters $\\bm{w}$ is specified as $p(D|\\bm{w})$\n\t\\item \\textit{Maximum likelihood estimation}: the most likely ``explanation'' of $D$ is $\\bm{w}_{\\text{ML}}$:\n\t$$\\bm{w}_{\\text{ML}} = \\arg\\max_{\\bm{w}} p(D|\\bm{w})$$\n\t\\item Using the i.i.d. assumption, we can state $p(D|\\bm{w}) = \\prod\\limits_{n=1}^{N} p(\\bm{x}_n|\\bm{w})$\n\t\\item For preventing numerical overflow and mostly simplifying the derivation, we can take the logarithm $\\log p(D|\\bm{w})$\n\t\\item Maximum where $\\frac{\\partial}{\\partial \\bm{w}}\\log p(D|\\bm{w}) = 0$\n\t\\item We can check whether our estimation is biased by comparing the expected result by the distribution parameters: $$\\mathbb{E}\\left[\\sigma_{ML}^{2}\\right] = \\mathbb{E}\\left[\\frac{1}{N}\\sum\\limits_{i=1}^{N}\\left(x_i - \\frac{1}{N}\\sum_{n=1}^{N} x_n\\right)^2\\right] = \\frac{N-1}{N} \\sigma^2 \\implies \\text{biased estimator}$$\n\\end{itemize}\n\\subsubsection{Maximum a posteriori}\n\\begin{itemize}\n\t\\item Choose the most probable model parameters $\\bm{w}$ given data $D$:\n\t$$\\bm{w}_{\\text{MAP}} = \\arg\\max_{\\bm{w}} p(\\bm{w}|D)$$\n\t\\item By applying the Bayes rule (and log), we get:\n\t$$\\bm{w}_{\\text{MAP}} = \\arg\\max_{\\bm{w}} \\log p(D|\\bm{w}) + \\log p(\\bm{w}) - \\log p(D)$$\n\t\\item We can drop the evidence as it is independent of $\\bm{w}$\n\\end{itemize}\n\\subsubsection{Bayesian approach}\n\\begin{itemize}\n\t\\item Frequentist approaches only consider point estimates without taking the uncertainty of the prediction into account\n\t\\item Given a prior belief over $\\bm{w}$, we are interested in the posterior distribution (not only maximum!)\n\t\\item The predictive distribution for a new data point $\\bm{x}'$ is therefore \n\t$$p(t'|\\bm{x}',D) = \\int p(t'| \\bm{x}', \\bm{w}) \\cdot p(\\bm{w}|D) d\\bm{w}$$\n\t\\item Thus, we also consider our uncertainty when predicting\n\t\\item However, we need to compute the evidence for that which is mostly quite hard (prefer less complex models)\n\\end{itemize}\n\\subsection{Model selection for supervised learning}\n\\begin{itemize}\n\t\\item Model selection comes with two main questions:\n\t\\begin{enumerate}\n\t\t\\item How can we estimate the performance of a model on unknown data?\n\t\t\\item How can we choose the optimal hyperparameters? $\\Rightarrow$ \\textbf{model selection}\n\t\\end{enumerate}\n\t\\item Common approach for large datasets: split in train, val and test dataset\n\t\\begin{description}\n\t\t\\item[Training dataset] About 80\\% of the data should be used for training. On this, we try to minimize the error/loss $L\\left(y(\\bm{x}_i),t_i\\right)$ for $(\\bm{x},t)\\in D_{train}$ and find optimal parameters $\\bm{w}^*$.\n\t\t\\item[Validation dataset] About 10\\% of the data is used for estimating the test error $L\\left(y(\\bm{x}_{\\text{val}}, \\bm{w}^*),t_{\\text{val}}\\right)$ for various $\\bm{w}^*$ from different hyperparameters. Hence, the hyperparameters are tuned on the validation dataset.\n\t\t\\item[Testing dataset] The last 10\\% of the available data provides the final test of the chosen best weights and hyperparameters. This data is used to estimate the performance on unseen data, and should therefore not be used for any parameter choosing!\n\t\\end{description}\n\t\\item However, for a small dataset, the validation and test set is very small and, hence, very noisy $\\Rightarrow$ use cross validation\n\\end{itemize}\n\\subsubsection{Cross Validation}\n\\begin{itemize}\n\t\\item Split data into $K$ folds % $D=\\left\\{\\left(x_1, t_1\\right), \\left(x_2, t_2\\right), ..., \\left(x_N, t_N\\right)\\right\\}$\n\t\\item If $K=N$, it is also called leave-one-out cross validation as the validation is one single data point\n\t\\item Train the model $y$ on $K-1$ folds, and test on the remaining fold $k$ $\\Rightarrow$ model $\\hat{y}^{\\mbox{--}k}(x)$\n\t\\item The estimation of the prediction error is the mean validation error over all folds. With the index function $\\kappa:\\left\\{1,...,N\\right\\}\\mapsto\\left\\{1,...,K\\right\\}$ (mapping data point to corresponding fold where it is used for validation), we get:\n\t$$CV(\\hat{y}) = \\frac{1}{N}\\sum\\limits_{i=1}^{N}L\\left(\\hat{y}^{\\mbox{--}\\kappa\\left(i\\right)}(\\bm{x}_i), t_i\\right)$$\n\t\\item Task of model selection: Run cross validation for each possible parameter setting and choose the one with lowest cross validation error\n\t\\item Task of test error estimation: after finding the best hyperparameters like $\\alpha^*$, retrain model on all $K$ folds, and test this model on a held-out test set\n\t\\item However, if test set is small, we again get a noisy estimation $\\Rightarrow$ Nested cross validation\n\t\\item Drawback of cross validation: it is computationally expensive and should therefore only be used for fast trainings/small datasets\n\\end{itemize}\n\\subsubsection{Nested Cross Validation}\n\\begin{itemize}\n\t\\item Cross validation for both model selection and model performance by reusing dataset for testing\n\t\\item General algorithm:\n\t\\begin{enumerate}\n\t\t\\item Split dataset into $M$ cross validation folds\n\t\t\\item For each of these folds $m=1,...,M$:\n\t\t\\begin{enumerate}\n\t\t\t\\item Let fold $m$ be the test dataset\n\t\t\t\\item Apply cross validation on the remaining data by splitting it into $K$ folds and find best hyperparameters $\\alpha^*,\\beta^*,...$\n\t\t\t\\item Retrain the model with the best hyperparameters on all data besides the fold $m$\n\t\t\t\\item Test the model on unseen data fold $m$ \n\t\t\\end{enumerate}\n\t\t\\item The final generalization error/loss on unseen data is the mean over all $M$ folds\n\t\\end{enumerate}\n\t\\item For choosing the best hyperparameters $\\alpha^*,\\beta^*,...$, we use single cross validation on the whole dataset again without a test dataset, but record the found generalization error as estimation for unknown data, also for the new model\n\\end{itemize}\n\\begin{figure}[ht]\n\t\\centering\n\t\\includegraphics[width=0.5\\textwidth]{figures/cross_validation_nested.png}\n\t\\caption{Illustration of nested cross validation. The outer loop splits dataset into test and trainval parts. Within the trainval parts, we apply cross validation to find optimal hyperparameters. Those are tested on the left-out fold from the outer loop, and the mean test error of all folds is the final generalization error. Note that every outer fold can lead to different optimal hyperparameters.}\n\t\\label{img:linear_regression_nested_cross_validation}\n\\end{figure}\n\\subsection{Bias variance decomposition}\n\\begin{itemize}\n\t\\item Frequentist view on model complexity\n\t\\item Common loss: the squared loss function, defined as $L\\left(t, y\\left(\\bm{x}\\right)\\right) = \\left(t - y\\left(\\bm{x}\\right)\\right)^2$\n\t% \\item The expected loss is $\\mathbb{E}\\left[L\\left(t, y\\left(\\bm{x}\\right)\\right)\\right]=\\int\\int \\left(t - y\\left(\\bm{x}\\right)\\right)^2 p\\left(\\bm{x}, t\\right) \\text{d}\\bm{x} \\text{ d}t$\n\t\\item An optimal model of $y\\left(\\bm{x}\\right)$ would minimize this loss which is given by $$h(\\bm{x}) = \\mathbb{E}\\left[t|\\bm{x}\\right] = \\int t\\cdot p\\left(t|\\bm{x}\\right) \\text{d}t$$\n\twhere the conditional distribution $p\\left(t|\\bm{x}\\right)$ is the actual, noisy data distribution (not known!)\n\t\\item Thus, the expected squared loss can be written as\n\t$$\\mathbb{E}\\left[L\\right] = \\int\\underbrace{\\left\\{y(\\bm{x}) - \\mathbb{E}\\left[t|\\bm{x}\\right]\\right\\}^2}_{\\text{model loss}} p(\\bm{x}) \\text{d}\\bm{x} + \\int\\underbrace{\\left\\{\\mathbb{E}\\left[t|\\bm{x}\\right] - t\\right\\}^2}_{\\text{intrinsic noise on data}} p(\\bm{x},t) \\text{d}\\bm{x}\\text{ d}t$$\n\twhere the first term, the model loss, depends on how different the model $y(\\bm{x})$ is from the actual data distribution, and the second term arises from the intrinsic noise and represents the minimum achievable expected loss\n\t\\item In Bayesian approach, we would model $y(\\bm{x}, \\bm{w})$ where the uncertainty of $\\bm{w}$ is expressed in the posterior distribution\n\t\\item However, from a frequentist viewpoint, we use multiple datasets $\\mathcal{D}$ on which we train our model and get a single estimation $\\bm{\\hat{w}}$ for each of them. The final model is the average over this ensemble of datasets.\n\t\\item To apply this approach, we take the model loss for a single input $\\bm{x}$, and add the expected model over all datasets:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\left\\{y\\left(\\bm{x};\\mathcal{D}\\right) - h\\left(\\bm{x}\\right)\\right\\}^2 & = \\left\\{y\\left(\\bm{x};\\mathcal{D}\\right) - \\mathbb{E}_{\\mathcal{D}}\\left[y\\left(\\bm{x};\\mathcal{D}\\right)\\right] + \\mathbb{E}_{\\mathcal{D}}\\left[y\\left(\\bm{x};\\mathcal{D}\\right)\\right] - h\\left(\\bm{x}\\right)\\right\\}^2\\\\\n\t\t\t& = \\left\\{y\\left(\\bm{x};\\mathcal{D}\\right) - \\mathbb{E}_{\\mathcal{D}}\\left[y\\left(\\bm{x};\\mathcal{D}\\right)\\right]\\right\\}^2 + \\left\\{\\mathbb{E}_{\\mathcal{D}}\\left[y\\left(\\bm{x};\\mathcal{D}\\right)\\right] - h\\left(\\bm{x}\\right)\\right\\}^2 \\\\\n\t\t\t& \\text{\\hspace{5mm} } + 2\\left\\{y\\left(\\bm{x};\\mathcal{D}\\right) - \\mathbb{E}_{\\mathcal{D}}\\left[y\\left(\\bm{x};\\mathcal{D}\\right)\\right]\\right\\}\\left\\{ \\mathbb{E}_{\\mathcal{D}}\\left[y\\left(\\bm{x};\\mathcal{D}\\right)\\right] - h\\left(\\bm{x}\\right)\\right\\}\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item The final model of the frequentist approach is the expected value of this loss over all datasets:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\mathbb{E}_{\\mathcal{D}}\\left[\\left\\{y\\left(\\bm{x};\\mathcal{D}\\right) - h\\left(\\bm{x}\\right)\\right\\}^2\\right] & = \\underbrace{\\mathbb{E}_{\\mathcal{D}}\\left[\\left\\{y\\left(\\bm{x};\\mathcal{D}\\right) - \\mathbb{E}_{\\mathcal{D}}\\left[y\\left(\\bm{x};\\mathcal{D}\\right)\\right]\\right\\}^2\\right]}_{\\text{\\textcolor{red}{variance}}} + \\underbrace{\\left\\{\\mathbb{E}_{\\mathcal{D}}\\left[y\\left(\\bm{x};\\mathcal{D}\\right)\\right] - h(\\bm{x})\\right\\}^2}_{\\textcolor{blue}{(\\text{bias})^2}}\n\t\t\\end{split}\n\t\\end{equation*}\n\twhere the first term is the \\textbf{variance} of a model trained on a single dataset compared to the average, and the second term is the loss of the average/expected model over all datasets, or rather the \\textbf{bias} of the model. The third term of the original equation is eliminated as only $y(\\bm{x};\\mathcal{D})$ is affected by the expectation operator $\\mathbb{E}_{\\mathcal{D}}$, and is the same as $\\mathbb{E}_{\\mathcal{D}}\\left[y\\left(\\bm{x};\\mathcal{D}\\right)\\right]$. \n\t\\item Coming back to the original expected squared loss, we can decompose it into three terms:\n\t$$\\text{expected loss} = (\\text{bias})^2 + \\text{variance} + \\text{noise}$$\n\twhere\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t(\\text{bias})^2 &= \\int \\left\\{\\mathbb{E}_{\\mathcal{D}}\\left[y\\left(\\bm{x};\\mathcal{D}\\right)\\right] - h(\\bm{x})\\right\\}^2p(\\bm{x}) \\text{d}\\bm{x}\\\\\n\t\t\t\\text{variance} &= \\int \\mathbb{E}_{\\mathcal{D}}\\left[\\left\\{y\\left(\\bm{x};\\mathcal{D}\\right) - \\mathbb{E}_{\\mathcal{D}}\\left[y\\left(\\bm{x};\\mathcal{D}\\right)\\right]\\right\\}^2\\right]p(\\bm{x}) \\text{d}\\bm{x}\\\\\n\t\t\t\\text{noise} &= \\int \\left\\{h(\\bm{x})-t\\right\\}^2p\\left(\\bm{x},t\\right)\\text{d}\\bm{x}\\text{ d}t\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Now, the task is to find the best balance between bias and variance. An example for the data distribution $\\mathbb{E}[t|x]=\\sin(2\\pi x)$ (note that noise is canceled by expectation), 24 Gaussian basis functions and regularized loss function is shown in Figure~\\ref{img:linear_regression_bias_variance_decomp_example}.\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/bias_variance_reg_comp.png}\n\t\t\\caption{Illustration of dependence of bias and variance on model complexity controlled by $\\lambda$. Less complex models (high $\\lambda$) tend to have a high bias (be far off the correct distribution) but it is more robust regarding the actual dataset (therefore, a low variance). Decreasing $\\lambda$ results in a lower bias, but a high variance as models tend to overfit and are therefore sensitive to the dataset.}\n\t\t\\label{img:linear_regression_bias_variance_decomp_example}\n\t\\end{figure}\n\t\\item Plotting the terms of the decomposed squared loss function over $\\lambda$ gives further insights of the model behavior (see Figure~\\ref{img:linear_regression_bias_variance_decomp_error_plot}). For generating such a plot, the integrals are approximated by sums over all data points $x$ as we have a limited number of samples.\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/bias_variance_loss_plot.png}\n\t\t\\caption{Plot of decomposed loss function for example of Figure~\\ref{img:linear_regression_bias_variance_decomp_example}. The goal is to minimize the test error. It is common that this is close to the minimum value of $(\\text{bias})^2+\\text{variance}$. High variance as on the left indicates overfitting, high bias error on the left shows that the model is underfitting.}\n\t\t\\label{img:linear_regression_bias_variance_decomp_error_plot}\n\t\\end{figure}\n\t\\item In conclusion, high values of $\\lambda$ reduce model complexity, and therefore increase bias loss and leads to underfitting. However, it provides a small variance.\\\\In contrast, small values of $\\lambda$ causes a low bias as the model is quite complex. Still, the variance is high indicating that the model overfits on the small datasets.\n\t\\item The bias-variance decomposition is less practical as it is better to train on one large dataset instead of splitting it into several small ones. Furthermore, this reduces the risk of overfitting for a high model complexity on the data anyways.\n\\end{itemize}\n\\subsection{Bayesian Linear Regression}\n\\begin{itemize}\n\t\\item Determining the suitable model complexity using the training data alone without overfitting\n\t\\item Result is a distribution of $\\bm{w}$ instead of single value as in maximum likelihood or posterior\n\\end{itemize}\n\\subsubsection{Parameter distribution}\n\\begin{itemize}\n\t\\item Prior over weights: $p(\\bm{w}) = \\mathcal{N}\\left(\\bm{m}_0, \\bm{S}_0\\right)$\n\t\\item Likelihood: $p\\left(t'|\\bm{x}', \\bm{w}, \\beta\\right)=\\mathcal{N}\\left(t'|\\bm{\\phi}(\\bm{x})^T\\bm{w}, \\beta^{-1}\\right)$\n\t\\item Posterior distribution: $p(\\bm{w}|\\bm{t}, \\bm{X}) = \\frac{p\\left(\\bm{t}|\\bm{X}, \\bm{w}, \\beta\\right) p\\left(\\bm{w}\\right)}{p\\left(\\bm{t}|\\bm{X}, \\beta\\right)} = \\mathcal{N}\\left(\\bm{m}_N, \\bm{S}_N\\right)$, where\\\\\n\t$$\\bm{S}_N^{-1}=\\bm{S}_0^{-1} + \\beta \\bm{\\Phi}^T\\bm{\\Phi}$$\n\t$$\\bm{m}_N = \\bm{S}_N\\left(\\bm{S}_0^{-1}\\bm{m}_0+\\beta\\bm{\\Phi}^T\\bm{t}\\right)$$\n\t\\item Maximum a posteriori corresponds by $\\bm{w}_{\\text{MAP}} = \\bm{m}_N$\n\t\\item If no prior was given ($\\bm{S}_0=\\alpha^{-1} \\bm{I}$ with $\\alpha\\to0$) the mean $\\bm{m}_N$ reduces to $\\bm{w}_{\\text{ML}}$\n\t\\item Mostly simpler Gaussian prior used: $p(\\bm{w}) = \\mathcal{N}\\left(\\bm{0}, \\alpha^{-1}\\bm{I}\\right)$\n\t\\begin{itemize}\n\t\t\\item Resulting parameters of posterior: \n\t\t$$\\bm{S}_N^{-1}=\\alpha^{-1}\\bm{I} + \\beta \\bm{\\Phi}^T\\bm{\\Phi}$$\n\t\t$$\\bm{m}_N = \\beta \\bm{S}_N\\bm{\\Phi}^T\\bm{t}$$\n\t\t$$p(\\bm{w}|\\bm{t}, \\bm{X})=\\frac{1}{\\sqrt{\\left(2\\pi\\right)^M |\\bm{S}_N|}}\\exp\\left[-\\frac{1}{2}\\left(\\bm{w}-\\bm{m}_N\\right)^T \\bm{S}_N^{-1} \\left(\\bm{w}-\\bm{m}_N\\right)\\right]$$\n\t\t\\item Corresponding log posterior: \n\t\t$$\\ln p\\left(\\bm{w}|\\bm{t}, \\bm{X}\\right) = -\\frac{\\beta}{2}\\sum\\limits_{n=1}^{N}\\left\\{t_n - \\bm{w}^T \\bm{\\phi}\\left(\\bm{x}_n\\right)\\right\\}^2 - \\frac{\\alpha}{2}\\bm{w}^T \\bm{w} + C$$\n\t\t\\item Thus, maximizing this posterior is equal to having a regularization term with $\\lambda=\\frac{\\alpha}{\\beta}$\n\t\t\\item Infinitely narrow prior by $\\alpha\\to\\infty$ ($\\alpha\\to0$ seen before ends up in maximum likelihood):\n\t\t$$\\lim\\limits_{\\alpha\\to\\infty} \\bm{S}_N = \\lim\\limits_{\\alpha\\to\\infty} \\left(\\alpha\\bm{I}+\\beta \\bm{\\Phi}^{T}\\bm{\\Phi}\\right)^{-1} = \\lim\\limits_{\\alpha\\to\\infty} \\alpha^{-1}\\bm{I} = \\bm{0}$$\n\t\t$$\\lim\\limits_{\\alpha\\to\\infty} \\bm{m}_N = \\lim\\limits_{\\alpha\\to\\infty} \\beta\\left(\\alpha\\bm{I}+\\beta \\bm{\\Phi}^{T}\\bm{\\Phi}\\right)^{-1}\\bm{\\Phi}^T \\bm{t} = \\lim\\limits_{\\alpha\\to\\infty} \\frac{\\beta}{\\alpha}\\bm{\\Phi}^T\\bm{t} = \\bm{0} = \\bm{m}_{0}$$\n\t\t\\item Infinite data $N\\to\\infty$:\n\t\t$$\\lim\\limits_{N\\to\\infty} \\bm{S}_N = \\lim\\limits_{N\\to\\infty} \\left(\\alpha\\bm{I}+\\beta \\bm{\\Phi}^{T}\\bm{\\Phi}\\right)^{-1} = \\lim\\limits_{N\\to\\infty} \\left(\\bm{\\Phi}^{T}\\bm{\\Phi}\\right)^{-1} = \\bm{0}$$\n\t\t$$\\lim\\limits_{N\\to\\infty} \\bm{m}_N = \\lim\\limits_{N\\to\\infty} \\beta\\left(\\alpha\\bm{I}+\\beta \\bm{\\Phi}^{T}\\bm{\\Phi}\\right)^{-1}\\bm{\\Phi}^T \\bm{t} = \\lim\\limits_{N\\to\\infty} \\left(\\bm{\\Phi}^T\\bm{\\Phi}\\right)^{-1}\\bm{\\Phi}^T\\bm{t} = \\bm{w}_{\\text{ML}}$$\n\t\t$\\Rightarrow$ At infinite data, all approaches agree: $\\bm{m}_N = \\bm{w}_{\\text{ML}} = \\bm{w}_{\\text{MAP}}$\n\t\\end{itemize}\n\\end{itemize}\n\n\\subsubsection{Sequential Bayesian Learning}\n\\begin{itemize}\n\t\\item Data is sequences of input $x$ and target $t$\n\t\\item Posterior after $N-1$ data points constitutes the prior for the $N$th data point!\n\t\\item Posterior 1: $p(\\bm{w}|x_1,t_1,\\alpha,\\beta)\\propto p(t_1|x_1,\\bm{w},\\beta)p(\\bm{w}|\\alpha)$\n\t\\item Posterior 2: $p\\left(\\bm{w}|(x_1,t_1),(x_2,t_2),\\alpha,\\beta\\right)\\propto p(t_2|x_2,\\bm{w},\\beta)p(\\bm{w}|x_1,t_1,\\alpha,\\beta)$\n\t\\item Posterior narrows down step by step until it gets very certain of the correct estimation \n\\end{itemize}\n\\begin{figure}[ht]\n\\centering\n\\includegraphics[width=0.5\\textwidth]{figures/sequential_bayesian_linear_regression.png}\n\\caption{Example for Sequential Bayesian Learning on target $t=-0.3+0.5x+\\epsilon$. First column: likelihood (not normalized for $\\bm{w}$, but for $t_n$!), second column: posterior, third column: sampled weights}\n\\end{figure}\n\\subsubsection{Predictive Distribution}\n\\begin{itemize}\n\t\\item Predictive distribution is defined by ($\\bm{\\mathtt{t}}$ targets in training set):\n\t$$p\\left(t|x, \\bm{\\mathtt{t}}, \\bm{X}, \\alpha, \\beta \\right) = \\int p\\left(t|x, \\bm{w},\\beta\\right)p\\left(\\bm{w}|\\bm{\\mathtt{t}}, \\bm{X}, \\alpha, \\beta\\right)d\\bm{w}$$\n\twhere $p\\left(t|x, \\bm{w},\\beta\\right) = \\mathcal{N}\\left(t|y\\left(\\bm{x},\\bm{w}\\right), \\beta^{-1}\\right)$ is the conditional distribution of target variable, and \\\\$p\\left(\\bm{w}|\\bm{\\mathtt{t}}, \\bm{X}, \\alpha, \\beta\\right) = \\mathcal{N}\\left(\\bm{w}|\\bm{m}_N, \\bm{S}_N\\right)$ the posterior weight distribution\n\t\\item Predictive distribution is convolution of two Gaussians $\\Rightarrow$ $p\\left(t|x, \\bm{\\mathtt{t}}, \\bm{X}, \\alpha, \\beta \\right)=\\mathcal{N}\\left(t|\\bm{m}_N^{T}\\bm{\\phi}(\\bm{x}),\\sigma_N^2(\\bm{x}) \\right)$\n\twhere variance $\\sigma_N^2(\\bm{x})=\\frac{1}{\\beta} + \\bm{\\phi}(\\bm{x})^T \\bm{S}_N \\bm{\\phi}(\\bm{x})$ (first term data noise, second weight uncertainty, which goes to 0 for infinite data $N\\to\\infty$)\n\t\\item Important points\n\t\\begin{enumerate}\n\t\t\\item Uncertainty is smaller near training points\n\t\t\\item Variance/uncertainty decreases with larger $N$\n\t\\end{enumerate}\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/bayesian_linear_regression_predictive_dist.png}\n\t\t\\caption{Example for predictive distributions. Green: ground truth data, blue: data points, red line: mean prediction, red area: 1-sigma area}\n\t\\end{figure}\n\t\\item The predictive distribution can be expressed by a \\textbf{kernel formulation}:\n\t\\begin{itemize}\n\t\t\\item The predictive mean is \n\t\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\ty\\left(\\bm{x}', \\bm{m}_N\\right) & = \\bm{\\phi}^T(\\bm{x}') \\bm{m}_N = \\beta \\bm{\\phi}^T(\\bm{x}')\\bm{S}_N \\bm{\\Phi}^T \\bm{t}\\\\\n\t\t\t& = \\beta \\bm{\\phi}^T(\\bm{x}')\\bm{S}_N \\sum\\limits_{n=1}^{N}\\bm{\\Phi}_{:,n}^T t_n = \\beta \\sum\\limits_{n=1}^{N} \\bm{\\phi}^T(\\bm{x}')\\bm{S}_N \\bm{\\phi}(\\bm{x}_n) t_n\\\\\n\t\t\t& = \\sum\\limits_{n=1}^{N} k\\left(\\bm{x}',\\bm{x}_n\\right) t_n \\text{\\hspace{5mm} \n\t\t\t\twhere\\hspace{5mm}} k\\left(\\bm{x}',\\bm{x}_n\\right)=\\beta \\bm{\\phi}^T(\\bm{x}')\\bm{S}_N \\bm{\\phi}(\\bm{x}_n)\n\t\t\\end{split}\n\t\t\\end{equation*}\n\t\t$\\Rightarrow$ Prediction is a linear combination of training set target values\n\t\t\\item Kernel values depend on whole dataset by $\\bm{S}_N$\n\t\t\\item Closer data points to $\\bm{x}'$ are given a higher weight than points further removed from $\\bm{x}'$\n\t\t\\item Thus, local evidence is weighted more strongly than distant evidence\n\t\t\\item Kernel can also express covariance:\n\t\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\\text{cov}\\left[t_1, t_2 | \\bm{x}_1, \\bm{x}_2\\right] & = \\text{cov}_{\\bm{w}}\\left[y(\\bm{x}_1, \\bm{w}), y(\\bm{x}_2, \\bm{w})\\right] = \\text{cov}_{\\bm{w}}\\left[\\bm{\\phi}^T(\\bm{x}_1)\\bm{w}, \\bm{w}^T\\bm{\\phi}(\\bm{x}_2)\\right]\\\\\n\t\t& = \\mathbb{E}_{\\bm{w}}\\left[\\bm{\\phi}^T(\\bm{x}_1)\\bm{w} \\bm{w}^T\\bm{\\phi}(\\bm{x}_2)\\right] - \\mathbb{E}_{\\bm{w}}\\left[\\bm{\\phi}^T(\\bm{x}_1)\\bm{w}\\right]\\mathbb{E}_{\\bm{w}}\\left[\\bm{w}^T\\bm{\\phi}(\\bm{x}_2)\\right]\\\\\n\t\t& = \\bm{\\phi}^T(\\bm{x}_1)\\text{cov}\\left[\\bm{w},\\bm{w}\\right] \\bm{\\phi}^T(\\bm{x}_2) = \\bm{\\phi}^T(\\bm{x}_1)\\bm{S}_N \\bm{\\phi}^T(\\bm{x}_2)\\\\\n\t\t& = \\frac{1}{\\beta}k\\left(\\bm{x}_1,\\bm{x}_2\\right) \n\t\t\\end{split}\n\t\t\\end{equation*}\n\t\t\\item Based on that, we can see that predictive mean at nearby points will be highly correlated (high values of the kernel), and smaller for distant points\n\t\t\t\\begin{figure}[ht]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.5\\textwidth]{figures/bayesian_kernel_formulation.png}\n\t\t\t\\caption{Right plot: matrix for $(x',x)$ of kernel $k\\left(x',x\\right)$ for Gaussian basis function. Left plot: slices of this matrix for different values of $x$}\n\t\t\\end{figure}\n\t\t\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Bayesian Model Comparison}\n\\begin{itemize}\n\t\\item By marginalizing (integrating) over the model parameters instead of making point estimates of their values, models can be directly compared on the training data instead of separate validation data\n\t\\item Compare $L$ models $\\left\\{\\mathcal{M}_i\\right\\}_{i=1}^{L}$\n\t\\item Probabilities are used to represent uncertainty in the choice of model. \n\t\\item We express our preference for different models by a prior distribution $p\\left(\\mathcal{M}_i\\right)$, so that the posterior is:\n\t$$p\\left(\\mathcal{M}_i|\\mathcal{D}\\right)\\propto p\\left(\\mathcal{M}_i\\right)p\\left(\\mathcal{D}|\\mathcal{M}_i\\right)$$\n\t\\item Important term is the \\textit{model evidence} $p\\left(\\mathcal{D}|\\mathcal{M}_i\\right)$ which updates our preference based on the seen data $\\mathcal{D}$. Marginalizes the parameters $\\bm{w}$ of a model:\n\t$$p\\left(\\mathcal{D}|\\mathcal{M}_i\\right) = \\int p\\left(\\mathcal{D}|\\bm{w}, \\mathcal{M}_i\\right)p\\left(\\bm{w}|\\mathcal{M}_i\\right)d\\bm{w}$$\n\t\\begin{itemize}\n\t\t\\item Can be viewed as the probability that $\\mathcal{D}$ is generated by a random sample of $\\bm{w}$ from the prior. \n\t\t\\item Is also the normalization constant for $p\\left(\\bm{w}|\\mathcal{D}, \\mathcal{M}_i\\right)$\n\t\\end{itemize}\n\t\\item Two models can be compared by dividing their posteriors:\n\t$$\\frac{p\\left(\\mathcal{M}_1|\\mathcal{D}\\right)}{p\\left(\\mathcal{M}_2|\\mathcal{D}\\right)} = \\frac{p\\left(\\mathcal{M}_1\\right)p\\left(\\mathcal{D}|\\mathcal{M}_1\\right)}{p\\left(\\mathcal{M}_2\\right)p\\left(\\mathcal{D}|\\mathcal{M}_2\\right)} \\text{\\hspace{5mm}where\\hspace{5mm}} \\frac{p\\left(\\mathcal{D}|\\mathcal{M}_1\\right)}{p\\left(\\mathcal{D}|\\mathcal{M}_2\\right)}\\text{ is called \\textit{Bayes factor}}$$\n\t\\item The predictive distribution is a weighted mean (based on the model probabilities) of our models:\n\t$$p\\left(t'|\\bm{x}',\\mathcal{D}\\right) = \\sum\\limits_{i=1}^{L}p\\left(t'|\\bm{x}', \\mathcal{M}_i, \\mathcal{D}\\right) p\\left(\\mathcal{M}_i | \\mathcal{D}\\right)$$\n\t\\item However, a simple approximation is using the single most probable model alone to make prediction $\\Rightarrow$ also known as \\textit{model selection}\n\\end{itemize}\n\\subsubsection{Approximated Model Evidence}\n\\begin{itemize}\n\t\\item For a single parameter $w$, assume that posterior distribution $p\\left(w|\\mathcal{D}, \\mathcal{M}_i\\right)$ is sharply peaked around the most probably value $w_{\\text{MAP}}$ with width $\\Delta w_{\\text{posterior}}$\n\t\\item Further, we assume that also the prior is a flat distribution with width $\\Delta w_{\\text{prior}}$ so that $p\\left(w|\\mathcal{M}_i\\right) = 1/\\Delta w_{\\text{prior}}$\n\t\\item Integral of model evidence can be approximated by its maximum value times the width of the peak:\n\t $$p\\left(\\mathcal{D}|\\mathcal{M}_i\\right) = \\int p\\left(\\mathcal{D}|\\bm{w}, \\mathcal{M}_i\\right)p\\left(\\bm{w}|\\mathcal{M}_i\\right)d\\bm{w}\\simeq p\\left(\\mathcal{D}|w_{\\text{MAP}}, \\mathcal{M}_i\\right)\\frac{\\Delta w_{\\text{posterior}}}{\\Delta w_{\\text{prior}}}$$\n\t \\item Taking the log leads to:\n\t $$\\ln p\\left(\\mathcal{D}|\\mathcal{M}_i\\right) \\simeq \\underbrace{\\ln p\\left(\\mathcal{D}|w_{\\text{MAP}}, \\mathcal{M}_i\\right)}_{\\text{model fit}} + \\underbrace{\\ln \\frac{\\Delta w_{\\text{posterior}}}{\\Delta w_{\\text{prior}}}}_{\\text{complexity penalty}}$$\n\t \\item The first term is the likelihood of the data, and therefore describes how good the model fits to the given data (optimal: maximized)\n\t \\item The second term penalizes model complexity as if $\\Delta w_{\\text{posterior}} < \\Delta w_{\\text{prior}}$ (distribution was finely tuned to the data), the term is negative and reduces the model evidence (optimal: minimized)\n\t \\item Hence, model evidence favors models where we have a trade-off between model fit and complexity\n\t \\begin{figure}[ht]\n\t \t\\centering\n\t \t\\includegraphics[width=0.5\\textwidth]{figures/bayesian_model_comparison_log.png}\n\t \t\\caption{Plotting the curve of $\\ln p\\left(\\mathcal{D}|\\mathcal{M}_i\\right)$ for different polynomials $M=0,1,...$ for the task of fitting a sine. As the sine is an odd function, polynomials of odd order fit the best (give the most improvement for the model fit). However, increasing the model complexity increases the penalty.}\n\t \\end{figure}\n\t \\item For a model with $K$ parameters, we get a similar approximation: \n\t $$\\ln p\\left(\\mathcal{D}|\\mathcal{M}_i\\right) \\simeq \\ln p\\left(\\mathcal{D}|\\bm{w}_{\\text{MAP}}, \\mathcal{M}_i\\right) + K \\ln \\frac{\\Delta w_{\\text{posterior}}}{\\Delta w_{\\text{prior}}}$$\n\t \\item Drawbacks of Bayesian approach:\n\t \\begin{itemize}\n\t \t\\item Still need to make assumptions about possible models\n\t \t\\item If no model is suitable for the data, the algorithm gives bad estimations\n\t \t\\item Model evidence is sensitive regarding the prior\n\t \t\\item Thus, a small test set is commonly used for Bayesian comparison\n\t \\end{itemize}\n\t \\begin{figure}[ht]\n\t \t\\centering\n\t \t\\includegraphics[width=0.4\\textwidth]{figures/bayesian_model_comparison.png}\n\t \t\\caption{Illustration of three different models and there corresponding model evidences. Horizontal axis $x$: one dimensional representation of all possible datasets; Vertical axis $y$: probability that these models generate this specific dataset based on their prior distribution of parameters $\\bm{w}$. $\\mathcal{M}_1$ is the simplest model and is therefore only able to create a small set of different data $\\mathcal{D}$. As the probability is normalized over all datasets $\\mathcal{D}$, the probability is higher than for more complex models like $\\mathcal{M}_2$ and $\\mathcal{M}_3$. Given a certain dataset $\\mathcal{D}_0$, we choose the model with the highest probability $\\Rightarrow$ model which just is enough complex to generate this dataset}\n\t \\end{figure}\n \t\n\\end{itemize}\n\\subsubsection{Model Evidence for Linear Basis Models}\n\\begin{itemize}\n\t\\item In fully Bayesian treatment, we must also consider all hyperparameters:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tp\\left(\\bm{t}|\\bm{X}, \\mathcal{M}_i\\right) & =\\int\\int\\int p\\left(\\bm{t}|\\bm{X},\\bm{w},\\beta,\\mathcal{M}_i\\right)p\\left(\\bm{w}|\\alpha\\right)p\\left(\\alpha, \\beta | \\mathcal{M}_i\\right)d\\bm{w}\\text{ }d\\alpha\\text{ }d\\beta\\\\\n\t\t\t& = \\int\\int \\underbrace{p\\left(\\bm{t}|\\bm{X}, \\beta, \\alpha, \\mathcal{M}_i\\right)}_{\\text{peaked posterior/prior}} \\underbrace{p\\left(\\alpha, \\beta | \\mathcal{M}_i\\right)}_{\\text{broad hyperprior}} d\\alpha \\text{ }d\\beta\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Note that the hyperprior can again contain new hyperparameters, for which one might have to define a new prior (and so on)\n\t\\item Approximation: take best hyperparameters $\\alpha^*$ and $\\beta^*$\n\t$$p\\left(\\bm{t}|\\bm{X}, \\mathcal{M}_i\\right) = \\arg\\max_{\\alpha, \\beta} p\\left(\\bm{t}|\\bm{X}, \\beta, \\alpha, \\mathcal{M}_i\\right)$$\n\t\\item Using this approximation, we come to following predictive distribution:\n\t$$p\\left(t'|\\bm{x}',\\bm{t}, \\bm{X}, \\mathcal{M}_i^*\\right) \\approx p\\left(t'|\\bm{x}',\\bm{t}, \\bm{X}, \\beta^*, \\alpha^*, \\mathcal{M}_i\\right)$$\n\\end{itemize}\n\\subsection{Limitations of fixed basis functions}\n\\textbf{Advantages}\n\\begin{itemize}\n\t\\item[+] Closed form solution for least-squares problem \n\t\\item[+] Tractable Bayesian treatment\n\t\\item[+] Nonlinear models mapping input variables to target variables through basis functions\n\\end{itemize}\n\\textbf{Limitations}\n\\begin{itemize}\n\t\\item[-] Assumption: Basis functions $\\phi_j(\\bm{x})$ are fixed, not learned\n\t\\item[-] \\textit{Curse of dimensionality}: to cover growing dimensions $D$ of input vectors, the number of basis functions needs to grow rapidly / exponentially\n\\end{itemize}"
  },
  {
    "path": "Machine_Learning_1/ml_neural_networks.tex",
    "content": "\\section{Neural Networks}\n\\begin{itemize}\n\t\\item Previously: fixed basis function $\\bm{\\phi}(\\bm{x}) = \\left(\\phi_0\\left(\\bm{x}\\right), \\phi_1\\left(\\bm{x}\\right), ..., \\phi_M\\left(\\bm{x}\\right)\\right)^T$\n\t\\item Neural networks: Create flexible non-linear features and learn them. \n\t\\begin{itemize}\n\t\t\\item Basis function with extra parameters: $\\phi_m\\left(\\bm{x},\\bm{w}_m^{(1)}\\right) = h\\left(\\left(\\bm{w}_m^{(1)}\\right)^T \\bm{x}\\right) = h\\left(\\sum\\limits_{d=0}^{D}w_{md}^{(1)}\\right)$ \n\t\t\\item Note that $\\bm{x}_n = \\left(1, x_{n0},...,x_{nD}\\right)^T\\Rightarrow \\bm{x}_n \\in \\mathbb{R}^D$\n\t\t\\item $h$ is the non-linear activation function\n\t\\end{itemize}\n\t\\item We can define regression for a one-layer neural network:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\ty\\left(\\bm{x}, \\bm{W}^{(1)}, \\bm{w}^{(2)}\\right) & = \\sum\\limits_{m=0}^{M} w_m^{(2)} h\\left(\\sum\\limits_{d=0}^{D}w_{md}^{(1)}\\right) \\\\\n\t\t\t& = \\left(\\bm{w}^{(2)}\\right)^T h\\left(\\bm{W}^{(1)} \\bm{x}\\right) \\text{\\hspace{5mm}where\\hspace{5mm}} \\bm{W}^{(1)} = \\left(\\begin{array}{cccc}\n\t\t\t\\mid & \\mid & \\dots & \\mid\\\\\n\t\t\t\\bm{w}_0^{(1)} & \\bm{w}_1^{(1)} & \\dots & \\bm{w}_D^{(1)}\\\\\n\t\t\t\\mid & \\mid & \\dots & \\mid\n\t\t\t\\end{array}\\right)\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item The same way, we can adjust a network for classification:\n\t$$y\\left(\\bm{x}, \\bm{W}^{(1)}, \\bm{w}^{(2)}\\right) = f\\left(\\sum\\limits_{m=0}^{M} w_m^{(2)} h\\left(\\sum\\limits_{d=0}^{D}w_{md}^{(1)}\\right)\\right) = f\\left(\\left(\\bm{w}^{(2)}\\right)^T h\\left(\\bm{W}^{(1)} \\bm{x}\\right)\\right)$$\n\twhere $f$ is sigmoid for binary and softmax for multi-class classification (then $\\bm{w}^{(2)}$ is $K\\times M$ matrix)\n\\end{itemize}\n\\subsection{Feed-forward Network Functions}\n\\begin{itemize}\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/neural_networks_overview.png}\n\t\t\\caption{Illustration of a multilayer perceptron (2 layers).}\n\t\t\\label{img:neural_networks_overview}\n\t\\end{figure}\n\t\\item Input are $D+1$ units whereas $x_0=1$ is the bias\n\t\\item First layer with $M\\times D$ weight matrix $\\bm{W}^{(1)}$ $\\Rightarrow$ $M$ activations $a_m = \\sum\\limits_{d=0}^{D} w_{md}^{(1)} x_{d}$ and bias $h^{(1)}(a_0)=1$\n\t\\item We apply an activation function on these activations to get the hidden units: $z_m = h^{(1)}(a_m)$ where $z_0=1$\n\t\\item Second layer with $K\\times M$ weight matrix $\\bm{W}^{(2)}$ $\\Rightarrow$ $K$ output units $y_k = h^{(2)}\\left(\\sum\\limits_{m=0}^{M} w_{km}^{(2)}z_m\\right)$\n\t\\item In conclusion, a output unit $y_k$ is calculated as follows:\n\t$$y_k\\left(\\bm{x}, \\bm{W}^{(1)}, \\bm{W}^{(2)}\\right) = h^{(2)}\\left(\\sum\\limits_{m=0}^{M} w_{km}^{(2)}\\cdot h^{(1)}\\left(\\sum\\limits_{d=0}^{D} w_{md}^{(1)}x_{d}\\right)\\right)$$\n\t\\item Alternative notation: $y_k = h^{(2)} \\circ \\bm{a}^{(2)} \\circ h^{(1)} \\circ \\bm{a}^{(1)} (\\bm{x})$\n\t\\item Additional forms: \n\t\\begin{itemize}\n\t\t\\item \\textit{Skip connections}: Connection between for instance first and fourth layer\n\t\t\\item \\textit{Sparse connections}: For instance convolutions, can have weight sharing\n\t\\end{itemize}\n\t\\item In general: $z_m = h\\left(\\sum\\limits_{j}w_{mj}z_{j}\\right)$ where $j$ are all incoming connections\n\t\\item Note that no closed directed cycles are allowed\n\\end{itemize}\n\\subsubsection{Universal approximator}\n\\begin{itemize}\n\t\\item Let $f$ by any continuous function on a compact area of $\\mathbb{R}^{D}$ and $h$ any fixed analytic function which is not polynomial (e.g. logistic function, tanh function, ...). Given any small number $\\epsilon > 0$ of an acceptable error, we can find a number $M$ and weights $w_{m}^{(2)}$ and $w_{md}^{(1)}\\in \\mathbb{R}$ such that:\n\t$$\\left|f(\\bm{x}) - y\\left(\\bm{x}, \\bm{W}^{(1)}, \\bm{w}^{(2)}\\right)\\right| < \\epsilon$$\n\twith $y$ as two-layer NN\n\t\\item For smaller $\\epsilon$ we need more hidden units $\\Rightarrow$ larger $M$\n\t\\item We may also take deeper networks that are usually capable to approximate more complex functions with less units\n\t\\item To approximate deep network with shallow one by error $\\epsilon$, the number of units $M$ needed scales exponentially for decreasing $\\epsilon$\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/neural_networks_universal_approximator.png}\n\t\t\\caption{Example approximations by a 2-layer network with 3 hidden units of the function (a) $f(x)=x^2$, (b) $f(x)=\\sin(x)$, (c) $f(x) = |x|$ and (d) $f(x)=\\mathbb{I}\\left(x>0\\right)$. The outputs of the three hidden units are shown as dashed lines.}\n\t\t\\label{img:neural_networks_universal_approximator}\n\t\\end{figure}\n\\end{itemize}\n\\subsection{Network Training}\n\\begin{itemize}\n\t\\item Use probabilistic interpretation of the network outputs to choose number of outputs, output activation function and loss (e.g. $p(t|\\bm{x},\\bm{w})$ for regression $\\Rightarrow$ maximizing likelihood used as error function)\n\\end{itemize}\n\\subsubsection{Network training for regression}\n\\begin{itemize}\n\t\\item Data input $\\bm{x}_n \\in \\mathbb{R}^{D}$ with continuous target $t_n \\in \\mathbb{R}$\n\t\\item Single real-valued target $\\to$ Single output unit with identity activation function $y\\left(\\bm{x}, \\bm{w}\\right) = a^{\\text{out}}$\n\t\\item Derive loss function by maximum likelihood:\n\t$$E(\\bm{w}) = -\\ln p\\left(\\bm{t}|\\bm{X}, \\bm{w}\\right) = \\frac{\\beta}{2}\\sum\\limits_{n=1}^{N}\\left\\{y(\\bm{x}_n, \\bm{w}) - t_n\\right\\}^2 - \\frac{N}{2}\\ln\\beta + \\frac{N}{2}\\ln 2\\pi $$\n\t$$\\text{equivalent to minimizing } E(\\bm{w})=\\frac{1}{2}\\sum\\limits_{n=1}^{N}\\left\\{y(\\bm{x}_n, \\bm{w}) - t_n\\right\\}^2$$\n\\end{itemize}\n\\subsubsection{Network training for binary classification}\n\\begin{itemize}\n\t\\item Targets are now binary values: $t_n\\in\\left\\{0,1\\right\\}$\n\t\\item As $p\\left(t=1|\\bm{x}\\right) = 1 - p\\left(t=0|\\bm{x}\\right)$, we model only one output unit: $y\\left(\\bm{x},\\bm{w}\\right) = p\\left(t=1|\\bm{x}\\right)$\n\t\\item The output activation function is therefore a sigmoid: $y\\left(\\bm{x}, \\bm{w}\\right) = \\sigma\\left(a^{\\text{out}}\\right)$\n\t\\item The maximum likelihood is here equivalent to minimizing BCE: $$E\\left(\\bm{w}\\right) = -\\sum\\limits_{n=1}^{N} t_n \\ln y\\left(\\bm{x}_n, \\bm{w}\\right) + (1 - t_n) \\ln \\left(1 - y\\left(\\bm{x}_n, \\bm{w}\\right)\\right)$$\n\\end{itemize}\n\\subsubsection{Network training for classification with $K$ classes}\n\\begin{itemize}\n\t\\item Targets are now one-hot vectors $\\bm{t}_n = \\left(0,...,1,...,0\\right)^T$\n\t\\item Now, we have to model all $K$ class distributions by $y_k\\left(\\bm{x}, \\bm{w}\\right) = p\\left(C_k|\\bm{x}\\right)$\n\t\\item Activation function is softmax: $y_k\\left(\\bm{x}, \\bm{w}\\right) = \\frac{\\exp\\left(a_k^{\\text{out}}\\right)}{\\sum_{j=1}^{K}\\exp\\left(a_j^{\\text{out}}\\right)}$\n\t\\item The maximum likelihood is here equivalent to:\n\t$$E\\left(\\bm{w}\\right) = - \\sum\\limits_{n=1}^{N} \\sum\\limits_{k=1}^{K} t_{nk} \\ln y_k\\left(\\bm{x}_n, \\bm{w}\\right)$$\n\\end{itemize}\n\\subsubsection{Parameter optimization}\n\\begin{itemize}\n\t\\item Optimal parameters minimize error function: $\\bm{w}^{*} = \\arg\\min\\limits_{\\bm{w}} E(\\bm{w})$\n\t\\item Problem: $E(\\bm{w})$ is not convex so that many local minima (can) exist \n\t\\item Different optimization strategies can be developed\n\t\\item \\textbf{Gradient Descent} uses full dataset for each update: $\\bm{w}^{(\\tau + 1)} = \\bm{w}^{(\\tau)} - \\eta \\triangledown E\\left(\\bm{w}^{(\\tau)}\\right)$\n\t\\item Always goes in the direction of steepest gradient\n\t\\item Will easily get stuck in local minimum when $\\triangledown E\\left(\\bm{w}\\right) = 0$\n\t\\item \\textbf{Stochastic Gradient Descent} uses single data point or minibatches for the update step: $\\bm{w}^{(\\tau + 1)} = \\bm{w}^{(\\tau)} - \\eta \\triangledown\\sum\\limits_{i=1}^{M} E_i\\left(\\bm{w}^{(\\tau)}\\right)$ \n\t\\item Converges to area around local minimum\n\t\\item More likely to escape local minimum as $\\triangledown E\\left(\\bm{w}^{(\\tau)}\\right) = 0$ does not imply $\\triangledown E_n\\left(\\bm{w}^{(\\tau)}\\right) = 0$ for all $n$\n\t\\item Is more computational efficient at the beginning as all $E_n\\left(\\bm{w}\\right)$ will point in a similar direction\n\t\\item Choose learning rate carefully to get good results\n\t\\begin{itemize}\n\t\t\\item If learning rate is too small: slow convergence\n\t\t\\item If learning rate is too high: oscillations around local minimum\n\t\t\\item Use learning rate scheduling with smaller learning rate over time\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Error Backpropagation}\n\\begin{itemize}\n\t\\item The error function is the sum of single point errors ($E(\\bm{w})=\\sum_{n}E_n(\\bm{w})$), so that we can calculate the gradients for each data point independently: $\\frac{\\partial E_n(\\bm{w})}{\\partial \\bm{w}}$ \n\t\\item Therefore, we first apply \\textit{forward propagation}: calculate all $a_j^{(l)} = \\sum_i w_{ji}^{(l)}z_i^{(l-1)}$ and $z_j^{(l)}=h^{(l)}\\left(a_j^{(l)}\\right)$\n\t\\item Then, apply \\textit{back propagation} by calculating all $\\frac{\\partial E_n}{\\partial w_{ji}^{(l)}}$\n\t\\item Backpropagation is based on the multi-dimensional chain rule: \n\t$$\\frac{\\partial f\\left(g_1\\left(x\\right), ..., g_D\\left(x\\right) \\right)}{\\partial x} = \\sum\\limits_{d=1}^{D}\\frac{\\partial f\\left(g_1\\left(x\\right), ..., g_D\\left(x\\right)\\right)}{\\partial g_d(x)}\\frac{\\partial g_d(x)}{\\partial x}$$\n\t\\item Thus, we can express the gradient regarding a single weight element by (only $a_{j}^{(l)}$ depends on $w_{ji}^{(l)}$):\n\t$$\\frac{\\partial E_n}{\\partial w_{ji}^{(l)}}=\\frac{\\partial E_n}{\\partial a_{j}^{(l)}}\\frac{\\partial a_{j}^{(l)}}{\\partial w_{ji}^{(l)}}$$\n\t\\item The second part of the derivate is just $\\frac{\\partial a_{j}^{(l)}}{\\partial w_{ji}^{(l)}}=z_{i}^{(l-1)}$, the first one we define as $\\delta_j^{(l)}\\equiv \\frac{\\partial E_n}{\\partial a_j^{(l)}}$\n\t\\item So, our derivate is $\\frac{\\partial E_n}{\\partial w_{ji}^{(l)}}=\\delta_j^{(l)}z_{i}^{(l-1)}$\n\t\\item $a_j^{(l)}$ effects the error only by its following units $a_k^{(l+1)}\\Rightarrow \\delta_j^{(l)}=\\sum_{k}\\frac{\\partial E_n}{\\partial a_k^{(l+1)}}\\frac{\\partial a_k^{(l+1)}}{a_j^{(l)}} = \\sum_{k}\\delta_{k}^{(l+1)}\\frac{\\partial a_k^{(l+1)}}{a_j^{(l)}}$\n\t\\item As $a_j^{(l)}$ effects $a_k^{(l+1)}$ only by the weight $w_{kj}^{(l+1)}$, the derivate is $\\frac{\\partial a_k^{(l+1)}}{a_j^{(l)}}=w_{kj}^{(l+1)}h^{(l)'} \\left(a_{j}^{(l)}\\right)$\n\t\\item Note that we need to be careful with skip connections\n\t\\item Overall, backpropagation can be summarized in three steps:\n\t\\begin{enumerate}\n\t\t\\item Compute $\\delta_k$ for all output units\n\t\t\\item Compute $\\delta_j$ for all hidden units through backpropagation:\n\t\t$$\\delta_j^{(l)} = h^{(l)'}\\left(a_j^{(l)}\\right)\\sum\\limits_{k}\\delta_k^{(l+1)}w_{kj}^{(l+1)}$$\n\t\t\\item Compute derivatives $\\frac{\\partial E_n}{\\partial w_{ji}^{(l)}}=\\delta_j^{(l)}z_{i}^{(l-1)}$\n\t\t\\item Apply iterative weight update: \n\t\t$w_{ji}^{(l)(\\tau+1)} = w_{ji}^{(l)(\\tau)}-\\eta \\delta_j^{(l)}z_{i}^{(l-1)}$\n\t\\end{enumerate}\n\\end{itemize}\n\\subsection{Issues with Neural Networks}\n\\begin{itemize}\n\t\\item Initialization of weights: randomly start near zero such that activations fall into linear part of activation functions (e.g. for tanh and sigmoid) and gradients don't vanish\n\t\\item Networks perform best when input has mean 0 and variance 1\n\t\\item When you have a large number of parameters, we need regularization!\n\t\\item Multiple local minima: Non-convex error function. Restart experiment with different seeds and choose model with lowest regularized error\n\t\\item Use weight sharing to reflect symmetries in data if possible\n\\end{itemize}"
  },
  {
    "path": "Machine_Learning_1/ml_summary.tex",
    "content": "\\documentclass[a4paper]{article} \n\\addtolength{\\hoffset}{-2.25cm}\n\\addtolength{\\textwidth}{4.5cm}\n\\addtolength{\\voffset}{-3.25cm}\n\\addtolength{\\textheight}{5cm}\n\\setlength{\\parskip}{0pt}\n\\setlength{\\parindent}{0in}\n\n\\usepackage{blindtext} % Package to generate dummy text\n\\usepackage{charter} % Use the Charter font\n\\usepackage[utf8]{inputenc} % Use UTF-8 encoding\n\\usepackage{microtype} % Slightly tweak font spacing for aesthetics\n\\usepackage[english]{babel} % Language hyphenation and typographical rules\n\\usepackage{amsthm, amsmath, amssymb} % Mathematical typesetting\n\\usepackage{float} % Improved interface for floating objects\n\\usepackage[final, colorlinks = true, \nlinkcolor = black, \ncitecolor = black]{hyperref} % For hyperlinks in the PDF\n\\usepackage{graphicx, multicol} % Enhanced support for graphics\n\\usepackage{xcolor} % Driver-independent color extensions\n\\usepackage{marvosym, wasysym} % More symbols\n\\usepackage{rotating} % Rotation tools\n\\usepackage{subcaption}\n\\usepackage{censor} % Facilities for controlling restricted text\n\\newcommand{\\note}[1]{\\marginpar{\\scriptsize \\textcolor{red}{#1}}} % Enables comments in red on margin\n\\usepackage{bm}\n\\usepackage{blkarray}\n\\usepackage{enumitem}\n\\usepackage{pgfplots}\n\\definecolor{colkeyword}{rgb}{0,0.4,0}\n\\definecolor{colname}{rgb}{0.4,0.4,0}\n\\definecolor{coltype}{rgb}{0.4,0,0.4}\n\\definecolor{coloperators}{rgb}{0,0,1.0}\n\\definecolor{colscopes}{rgb}{0.4,0,0}\n\n\\setcounter{tocdepth}{2}\n% Title Page\n\\title{Summary Machine Learning 1}\n\\author{Phillip Lippe}\n\n\n\\begin{document}\n\\maketitle\n\\tableofcontents\n\\newpage\n\n\\input{ml_basic_probability.tex}\n\\input{ml_linear_regression.tex}\n\\input{ml_linear_classification.tex}\n\\input{ml_neural_networks.tex}\n\\input{ml_unsupervised_learning.tex}\n\\input{ml_kernel_methods.tex}\n\\input{ml_combining_models.tex}\n\\appendix\n\\newpage\n\\input{ml_appendix.tex}\n\\end{document}"
  },
  {
    "path": "Machine_Learning_1/ml_unsupervised_learning.tex",
    "content": "\\section{Unsupervised learning}\n\\begin{itemize}\n\t\\item We can express our data distribution by marginalizing latent variables (unobserved targets/values that make it easier to understand the data). This allows us to model the data with more tractable joint distributions with simpler components to understand:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\bm{z}\\text{ continuous: } & p\\left(\\bm{x}\\right) = \\int p\\left(\\bm{x}, \\bm{z}\\right) d\\bm{z} = \\int p\\left(\\bm{x}| \\bm{z}\\right) p\\left(\\bm{z}\\right)d\\bm{z}\\\\\n\t\t\t\\bm{z}\\text{ discrete: } & p\\left(\\bm{x}\\right) = \\sum\\limits_{\\bm{z}} p\\left(\\bm{x}, \\bm{z}\\right) = \\sum\\limits_{\\bm{z}} p\\left(\\bm{x}| \\bm{z}\\right) p\\left(\\bm{z}\\right)\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Discrete latent variables are typically used for clustering, whereas continuous are applied for dimensionality reduction \n\\end{itemize}\n\\subsection{\\textit{K}-means Clustering}\n\\begin{itemize}\n\t\\item Every single data point $\\bm{x}$ is assigned to a cluster $\\to$ a discrete latent variable $\\bm{z}$\n\t\\item Number of clusters/different values for $\\bm{z}$ must be determined beforehand\n\t\\item Cluster as comprising a group of data points whose inter-point distances are small compared with the distances to points outside the cluster\n\t\\item Hence, we define $\\bm{\\mu}_k$ as a prototype (here also the mean) of the cluster $k$, and minimize the sum of squares of the distances of each data point to its closest vector $\\bm{\\mu}_k$:\n\t$$J=\\sum\\limits_{n=1}^{N}\\sum\\limits_{k=1}^{K} z_{nk} ||\\bm{x}_n - \\bm{\\mu}_k||^2$$\n\twhere $\\bm{z}_n$ is a one-hot vector with $z_{nk}=1$ if $k$ is closest cluster of $\\bm{x}_n$\n\t\\item Optimization algorithm (expectation-maximization (EM) algorithm):\n\t\\begin{enumerate}\n\t\t\\item Means $\\bm{\\mu}_k \\in \\mathbb{R}^D$ are initialized randomly\n\t\t\\item Repeat until convergence ($\\mu_k$ and $z_{nk}$ do not change for any $n$ and $k$):\n\t\t\\begin{enumerate}\n\t\t\t\\item \\textbf{Expectation step}: Find the assignment of the closest cluster for every data point:\n\t\t\t$$\\frac{\\partial J}{\\partial z_{nk}}=0 \\Rightarrow z_{nk} = \\begin{cases}\n\t\t\t1 & \\text{ if } k=\\arg\\min\\limits_j ||\\bm{x}_n - \\bm{\\mu}_j||^2\\\\\n\t\t\t0 & \\text{ otherwise }\n\t\t\t\\end{cases}$$\n\t\t\t\\item \\textbf{Maximization step}: Find the means of each cluster:\n\t\t\t$$\\frac{\\partial J}{\\partial \\bm{\\mu}_{k}}=0 \\Rightarrow \\bm{\\mu}_k = \\frac{\\sum_n z_{nk}\\bm{x}_n}{\\sum_n z_{nk}}$$\n\t\t\\end{enumerate}\n\t\\end{enumerate}\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/k_means_example.png}\n\t\t\\caption{Illustration of the $K$-means algorithm. First, an expectation step is performed where the data points are assigned to a cluster (see \\textit{(b)}, \\textit{(d)}, \\textit{(f)}, and \\textit{(h)}), and then the maximization step optimizes the means of the clusters (see \\textit{(c)}, \\textit{(e)}, \\textit{(g)}, and \\textit{(i)}).}\n\t\t\\label{img:k_means_example}\n\t\\end{figure}\n\t\\item The algorithm converges as each phase reduces the value of the objective function $J$, but they might converge to a local rather than global minimum (perform multiple random restarts and choose best minimum found)\n\t\\item \\textbf{Application}: image compression. Every pixel is a data point, and we search for $K$ clusters representing different colors in the image. The image is compressed by only using the cluster means (colors) instead of specifying a color at every pixel. A problem of this method is that the position correlations are ignored.\n\t\\item \\textbf{Failures} of $K$-means:\n\t\\begin{itemize}\n\t\t\\item $K$-means is only able to cluster spherical data due to the squared distance we try to minimize. Other shapes require different distance measures or data transformation by some basis functions.\n\t\t\\item $K$-means strongly prefers clusters of the same size/spread, and therefore tries to find cluster with the same spread in the dataset.\n\t\t\\item $K$-means is very sensitive to outliers. As it tries to minimize the squared distance, outliers may have a significant effect on the cluster means.\n\t\\end{itemize}\n\t\\item \\textbf{Improvements}:\n\t\\begin{itemize}\n\t\t\\item For a large dataset, we can use SGD to reduce computational effort. The update for a single data point would look like:\n\t\t$$\\bm{\\mu}_k^{(\\tau+1)} = \\bm{\\mu}_k^{(\\tau)} - \\eta \\left(\\frac{\\partial J}{\\partial \\bm{\\mu}_k^{(\\tau)}}\\right)^T = \\bm{\\mu}_k^{(\\tau)} + 2 \\eta \\left(\\bm{x}_n - \\bm{\\mu}_k^{(\\tau)}\\right)$$\n\t\t\\item Use other distance measure between points that is for example not so sensitive to outliers:\n\t\t$$\\tilde{J} = \\sum\\limits_{n=1}^{N}\\sum\\limits_{k=1}^{K} z_{nk} \\mathcal{V}\\left(\\bm{x}_n, \\bm{\\mu}_k\\right)$$\n\t\twhere $\\mathcal{V}$ measures the similarity of $\\bm{x}_n$ and $\\bm{\\mu}_k$.\n\t\\end{itemize}\n\t\\item \\textbf{Pros and cons} of $K$-means\n\t\\begin{itemize}\n\t\t\\item[+] Simple to implement\n\t\t\\item[+] Fast\n\t\t\\item Local minima\n\t\t\\item Only models spherical data \n\t\t\\item Sensitive to feature scales and outliers\n\t\t\\item Number of clusters $K$ must be specified in advance with prior knowledge\n\t\t\\item Cluster assignments are hard and not probabilistic\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Mixture of Gaussians and EM algorithm}\n\\begin{itemize}\n\t\\item Approximate the joint distribution $p\\left(\\bm{x},\\bm{z}\\right)=p\\left(\\bm{x}|\\bm{z}\\right)\\cdot p\\left(\\bm{z}\\right)$ by a mixture of Gaussians ($\\bm{z}$ chooses the mixture component, and points in the cluster are Gaussian distributed)\n\t\\item We define the prior as $p\\left(z_k=1\\right)=\\pi_k$, where $\\sum_k \\pi_k= 1$ and $\\pi_k \\in [0,1]$\n\t\\item The single clusters are Gaussian: $p\\left(\\bm{x}|z_k=1\\right) = \\mathcal{N}\\left(\\bm{x}|\\bm{\\mu}_k, \\bm{\\Sigma}_k\\right)$\n\t\\item Overall, the generative distribution is $p\\left(\\bm{x}\\right) = \\sum_{k=1}^{K} \\pi_k \\mathcal{N}\\left(\\bm{x}|\\bm{\\mu}_k, \\bm{\\Sigma}_k\\right)$\n\t\\item The posterior/conditional probability of $\\bm{z}$ given $\\bm{x}$ is also defined as the \\textit{responsibility} (that component $k$ in the mixture model takes for 'explaining' the observation/data point $\\bm{x}$):\n\t$$p\\left(z_k=1|\\bm{x}\\right) = \\frac{p\\left(\\bm{x}|z_k=1\\right)\\cdot p\\left(z_k=1\\right)}{\\sum_j p\\left(\\bm{x}|z_j=1\\right)\\cdot p\\left(z_j=1\\right)}=\\frac{\\pi_k \\mathcal{N}\\left(\\bm{x}|\\bm{\\mu}_k, \\bm{\\Sigma}_k\\right)}{\\sum_j \\pi_j \\mathcal{N}\\left(\\bm{x}|\\bm{\\mu}_j, \\bm{\\Sigma}_j\\right)} = \\gamma\\left(z_{k}\\right)$$\n\t\\item To optimize our parameters, we again maximize the log-likelihood:\n\t$$\\ln p\\left(\\bm{X}|\\bm{\\pi}, \\bm{\\mu}, \\bm{\\Sigma}\\right) = \\sum\\limits_{n=1}^{N} \\ln \\sum\\limits_{k=1}^{K} \\pi_k \\mathcal{N}\\left(\\bm{x}|\\bm{\\mu}_k, \\bm{\\Sigma}_k\\right)$$\n\t\\item However, maximizing the log-likelihood has no closed-form solution as stationary points depend on $\\gamma\\left(z_{nk}\\right)$ which again depends on $\\bm{\\pi}, \\bm{\\mu}$ and $\\bm{\\Sigma}$ $\\Rightarrow$ use expectation maximization algorithm by alternating the update of (\\textit{expected}) posterior $\\gamma\\left(z_{nk}\\right)$ and \\textit{maximizing} for parameters $\\bm{\\pi}, \\bm{\\mu}$ and $\\bm{\\Sigma}$\n\t\\item Maximizing with respect to $\\bm{\\mu}_k$:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t& \\frac{\\partial}{\\partial \\bm{\\mu}_k} \\sum\\limits_{n=1}^{N} \\ln p\\left(\\bm{x}_n|\\left\\{\\pi_k\\right\\}_{k=1}^{K}, \\left\\{\\bm{\\mu}_k\\right\\}_{k=1}^{K}, \\left\\{\\bm{\\Sigma}_k\\right\\}_{k=1}^{K}\\right)\\\\\n\t\t\t= & \\sum\\limits_{n=1}^{N} \\frac{1}{ p\\left(\\bm{x}_n|\\left\\{\\pi_k\\right\\}_{k=1}^{K}, \\left\\{\\bm{\\mu}_k\\right\\}_{k=1}^{K}, \\left\\{\\bm{\\Sigma}_k\\right\\}_{k=1}^{K}\\right)} \\frac{\\partial}{\\partial \\bm{\\mu}_k} p\\left(\\bm{x}_n|\\left\\{\\pi_k\\right\\}_{k=1}^{K}, \\left\\{\\bm{\\mu}_k\\right\\}_{k=1}^{K}, \\left\\{\\bm{\\Sigma}_k\\right\\}_{k=1}^{K}\\right)\\\\\n\t\t\t= & \\sum\\limits_{n=1}^{N} \\frac{\\pi_k \\mathcal{N}\\left(\\bm{x}_n|\\bm{\\mu}_k, \\bm{\\Sigma}_k\\right)}{\\sum\\limits_{j=1}^{K}\\pi_j \\mathcal{N}\\left(\\bm{x}_n|\\bm{\\mu}_j, \\bm{\\Sigma}_j\\right)} \\left(\\bm{x}_n - \\bm{\\mu}_k\\right)^T\\bm{\\Sigma}_k^{-1}\\\\\n\t\t\t= & \\sum\\limits_{n=1}^{N} y\\left(z_{nk}\\right) \\left(\\bm{x}_n - \\bm{\\mu}_k\\right)^T\\bm{\\Sigma}_k^{-1}\\\\\n\t\t\t\\Rightarrow & \\bm{\\mu}_k = \\frac{\\sum_{n=1}^{N}\\gamma\\left(z_{nk}\\right)\\bm{x}_n}{\\sum_{n=1}^{N}\\gamma\\left(z_{nk}\\right)}\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item For maximizing $\\pi_k$ we need to use the Lagrange multiplier (as the sum must be 1):\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t& \\frac{\\partial}{\\partial \\pi_k} \\sum\\limits_{n=1}^{N} \\ln p\\left(\\bm{x}_n|\\left\\{\\pi_k\\right\\}_{k=1}^{K}, \\left\\{\\bm{\\mu}_k\\right\\}_{k=1}^{K}, \\left\\{\\bm{\\Sigma}_k\\right\\}_{k=1}^{K}\\right) + \\lambda \\left(\\sum\\limits_{j=1}^{K}\\pi_j - 1\\right)\\\\\n\t\t\t= & \\sum\\limits_{n=1}^{N} \\frac{\\pi_k \\mathcal{N}\\left(\\bm{x}_n|\\bm{\\mu}_k, \\bm{\\Sigma}_k\\right)}{\\sum\\limits_{j=1}^{K}\\pi_j \\mathcal{N}\\left(\\bm{x}_n|\\bm{\\mu}_j, \\bm{\\Sigma}_j\\right)} + \\lambda \\pi_k\\\\\n\t\t\t= & \\sum\\limits_{n=1}^{N} \\gamma\\left(z_{nk}\\right) + \\lambda \\pi_k\\\\\n\t\t\t\\Rightarrow & \\pi_k = -\\frac{1}{\\lambda}\\sum\\limits_{n=1}^{N} \\gamma\\left(z_{nk}\\right)\\\\[10pt]\n\t\t\t& \\frac{\\partial}{\\partial \\lambda} \\sum\\limits_{n=1}^{N} \\ln p\\left(\\bm{x}_n|\\left\\{\\pi_k\\right\\}_{k=1}^{K}, \\left\\{\\bm{\\mu}_k\\right\\}_{k=1}^{K}, \\left\\{\\bm{\\Sigma}_k\\right\\}_{k=1}^{K}\\right) + \\lambda \\left(\\sum\\limits_{j=1}^{K}\\pi_j - 1\\right)\\\\\n\t\t\t= & \\sum\\limits_{j=1}^{K}\\pi_j - 1 = -\\frac{1}{\\lambda}\\sum\\limits_{n=1}^{N} \\underbrace{\\sum\\limits_{j=1}^{K}\\gamma\\left(z_{nj}\\right)}_{=1} - 1 = 0\\\\\n\t\t\t\\Rightarrow & \\lambda = -N, \\pi_k = \\frac{N_k}{N} \\text{\\hspace{5mm}where\\hspace{5mm}} N_k = \\sum\\limits_{n=1}^{N} \\gamma\\left(z_{nk}\\right) \\text{\\hspace{2mm}(effective number of points in \\textit{k})}\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Maximizing $\\bm{\\Sigma}_k$ is done by:\n\t$$\\bm{\\Sigma}_k = \\frac{1}{N_k} \\sum\\limits_{n=1}^{N} \\gamma\\left(z_{nk}\\right)\\left(\\bm{x}_n - \\bm{\\mu}_k\\right)^T\\left(\\bm{x}_n - \\bm{\\mu}_k\\right)$$\n\t\\item Summarized EM algorithm steps for Gaussian mixture models: \n\t\\begin{itemize}\n\t\t\\item \\textbf{Expectation step}: update the posterior:\n\t\t$$\\gamma\\left(z_{k}\\right) = \\frac{\\pi_k \\mathcal{N}\\left(\\bm{x}|\\bm{\\mu}_k, \\bm{\\Sigma}_k\\right)}{\\sum_j \\pi_j \\mathcal{N}\\left(\\bm{x}|\\bm{\\mu}_j, \\bm{\\Sigma}_j\\right)}$$\n\t\t\\item \\textbf{Maximization step}: update the parameters:\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\bm{\\mu}_k & = \\frac{\\sum_{n=1}^{N}\\gamma\\left(z_{nk}\\right)\\bm{x}_n}{\\sum_{n=1}^{N}\\gamma\\left(z_{nk}\\right)}\\\\\n\t\t\t\t\\pi_k & = \\frac{N_k}{N}\\\\\n\t\t\t\t\\bm{\\Sigma}_k & = \\frac{1}{N_k} \\sum\\limits_{n=1}^{N} \\gamma\\left(z_{nk}\\right)\\left(\\bm{x}_n - \\bm{\\mu}_k\\right)^T\\left(\\bm{x}_n - \\bm{\\mu}_k\\right)\\\\\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\\end{itemize}\n\t\\item Assigning points to clusters: either soft clusters (probability of belonging to $k$: $\\gamma\\left(z_{k}\\right) = p\\left(z_k=1|\\bm{x}\\right)$) or hard clusters (most likely cluster given by $k=\\arg\\min_j \\gamma\\left(z_j\\right)$)\n\t\\item \\textbf{Pros and cons} of GMM:\n\t\\begin{itemize}\n\t\t\\item[+] Allows soft-assignments in contrast to $K$-means\n\t\t\\item[+] More flexible as we can model different covariances per cluster\n\t\t\\item Slower than $K$-means as every step requires more computation (can use $K$-means result as initialization)\n\t\t\\item Same local convergence issues as $K$-means\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Principal Component Analysis}\n\\begin{itemize}\n\t\\item Find linear orthogonal projection to lower dimensional space to maximize variance ($\\mathbb{R}^{D}\\to \\mathbb{R}^{M}$ where $M<D$)\n\t\\item Try to find projection by capturing axes of maximal variation in the data, called \\textit{principal components}\n\t\\item Covariance is given by $S=\\frac{1}{N}\\sum\\limits_{n=1}^{N}\\left(\\bm{x}_n - \\overline{\\bm{x}}\\right)\\left(\\bm{x}_n - \\overline{\\bm{x}}\\right)^T$  which is symmetric and positive semi-definite\n\t\\item Project data into first latent dimension by $\\bm{u}_1 \\in \\mathbb{R}^D$. As we only need its direction, we make sure that $\\bm{u}_1^T \\bm{u}_1 = 1$\n\t\\item The projection is given by $\\bm{u}_1^T \\bm{x}_n$.\n\t\\item The variance of the projected data is $\\bm{u}_1^T S \\bm{u}_1$. We try to maximize this variance with respect to the constraint $\\bm{u}_1^T \\bm{u}_1 = 1$ (Lagrangian multiplier):\n\t$$\\arg\\max_{\\bm{u}_1}\\max_{\\lambda_1} \\bm{u}_1^T S \\bm{u}_1 + \\lambda_1 (1 - \\bm{u}_1^T \\bm{u}_1)$$\n\tDeriving by $\\bm{u}_1$ gives us the equation $S\\bm{u}_1 = \\lambda_1 \\bm{u}_1$ which is the eigenvalue equation. Thus, $\\bm{u}_1$ is an eigenvector and $\\lambda_1$ and eigenvalue of $S$. As we try to maximize the equation, we choose $\\lambda_1$ to be the \\textit{greatest} eigenvalue.\n\t\\item $\\bm{u}_1$ is called a \\textit{principal component}. The variance of the projected data is $\\bm{u}_1^T S \\bm{u}_1 = \\lambda_1$\n\t\\item We can repeat this procedure for $M$ orthogonal vectors and get a projection $U_M = \\left[\\bm{u}_1, ..., \\bm{u}_M\\right] \\in \\mathbb{R}^{D\\times M}$. Those are $M$ eigenvectors of $S$, where $\\lambda_1 \\geq \\lambda_2 \\geq ... \\geq \\lambda_M$. As $S$ is positive semi-definite, we can ensure that $\\lambda_i\\geq 0$\n\t\\item The variance of the projected data is $\\sum\\limits_{j=1}^{M} \\bm{u}_j^T S \\bm{u}_j = \\sum\\limits_{j=1}^{M} \\lambda_j$\n\t\\item \\textbf{PCA}: Compute $\\overline{\\bm{x}}$ and the eigen-decomposition of $S$. The \\textbf{projection} is $z = U_M^T \\left(\\bm{x} - \\overline{\\bm{x}}\\right)$\n\t\\item The idea is that the information which is lost is only noise, so that we still keep the expressiveness of the data. However, eigen-decomposition might be expensive.\n\t\\item Note that eigenvalues can be found by solving the equation $\\det\\left(S - \\lambda \\bm{I}\\right) = 0$. We can represent $S$ by its eigenvalue decomposition $S=U\\Lambda U^T$. For the eigenvectors, we can state that $\\bm{u}_j^T \\bm{u}_i = 0 \\text{ if } i\\neq j, \\text{ else } 1$.\n\t\\item \\textbf{Applications}\n\t\\begin{itemize}\n\t\t\\item \\textit{Dimensionality reduction}: which $M$ to choose? To preserve at least 90\\% of the variance we need to make sure that $\\frac{\\sum_{j=1}^{M}\\lambda_j}{\\sum_{j=1}^{D}\\lambda_j} \\geq 0.9$\n\t\t\\item \\textit{Feature de-correlation}: PCA ensures that features have no correlation in projected space. The covariance matrix is diagonal: $S'_M = \\Lambda_M$\n\t\t\\item \\textit{Whitening}: center and de-correlate features by $\\bm{z} = U_M^T (\\bm{x}-\\overline{\\bm{x}})$ where $M$ can be equals to $D$. If we want to rescale it (e.g. unit std. deviation), we apply a factor: $\\bm{z} = \\Lambda_M^{1/2} U_M^T (\\bm{x}-\\overline{\\bm{x}})$ \n\t\t\\item \\textit{Compression}: transform input to lower dimensional space. Reconstruction can be performed by $\\tilde{\\bm{x}} = U_M \\bm{z} + \\overline{\\bm{x}}$\n\t\\end{itemize}\n\t\\item \\textbf{Perspective of minimal reconstruction error}\n\t\\begin{itemize}\n\t\t\\item An alternative view on PCA is minimizing the reconstruction error of the transformed data to the original space\n\t\t$$\\min \\frac{1}{N} \\sum\\limits_{n=1}^{N} ||\\bm{x}_n - \\bm{z}_n||^2$$\n\t\t\\item We can express our data by $\\bm{x}_n = \\sum\\limits_{j=1}^{D} (\\bm{x}_n^T \\bm{u}_j) \\bm{u}_j$. The transformed data is thus $\\bm{z}_n = \\sum\\limits_{j=1}^{M} (\\bm{x}_n^T \\bm{u}_j) \\bm{u}_j + \\sum\\limits_{j=M+1}^{D} b_j \\bm{u}_j$\n\t\t\\item (By doing some math) we can show that the objective function is actually $\\sum_{j=M+1}^{D} \\bm{u}_j^T S \\bm{u}_j$. Thus, both approaches lead to the same result\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Probabilistic PCA}\n\\begin{itemize}\n\t\\item Generative probabilistic version of PCA where we learn by maximizing the likelihood (both latent and observed are Gaussian)\n\t\\item Generative model works as $\\bm{x} = W\\bm{z} + \\bm{\\mu} + \\bm{\\epsilon}$ where $\\bm{\\epsilon}$ represents the noise\n\t\\item We define $p(\\bm{z}) = \\mathcal{N}(\\bm{z}|0, \\bm{I})$, $p(\\epsilon) = \\mathcal{N}(\\epsilon|0, \\sigma^2 \\bm{I})$, and therefore $p(\\bm{x}|\\bm{z}) = \\mathcal{N}(\\bm{x}, W\\bm{z} + \\bm{\\mu}, \\sigma^2 \\bm{I})$. By marginalizing out $\\bm{z}$, we get $p(\\bm{x}) = \\mathcal{N}(\\bm{x}|\\bm{\\mu}, C)$ with $C = WW^T + \\sigma^2 \\bm{I}$\n\t\\item New data points are generated by first sampling from low dimensional $z$ space, and then sampling $x$ based on $z$\n\t\\item We can optimize this sample distribution based on the maximum likelihood of a given dataset ($\\mu$ is as usual the mean, $\\sigma^2 = \\frac{1}{D - M}\\sum_{j=M+1}^{D} \\lambda_j$)\n\t\\item \n\\end{itemize}\n\\subsubsection{Non-linear variants of PCA}\n\\begin{itemize}\n\t\\item Limitations of PCA: only linear transformation possible. So, we can also just capture variance along a linear axes through the $x$ space\n\t\\item We can get non-linear by using different forms\n\t\\item \\textbf{Kernel PCA}\n\t\\begin{itemize}\n\t\t\\item We can define the covariance matrix by $C=\\frac{1}{N}\\sum\\limits_{n=1}^{N} \\bm{\\phi}(\\bm{x})\\bm{\\phi}(\\bm{x})^T$\n\t\t\\item Then, we can state that $z_i(\\bm{x}) = \\bm{\\phi}(\\bm{x})^T \\bm{u}_i = \\sum_{n=1}^{N}a_{in}\\bm{\\phi}(\\bm{x})^T \\bm{\\phi}(\\bm{x}_n) =  \\sum_{n=1}^{N}a_{in}k(\\bm{x},\\bm{x}_n)$\n\t\t\\item By using a non-linear kernel, we are able to get non-linear projections\n\t\\end{itemize}\n\t\\item \\textbf{Autoencoders (NN)}\n\t\\begin{itemize}\n\t\t\\item Non-linear dimensionality reduction with neural networks by having a low hidden dimension size, and trying to reproduce input\n\t\t\\item In variational auto-encoders, we introduce sampling from the latent space so that we can generate new data points\n\t\\end{itemize}\n\\end{itemize}"
  },
  {
    "path": "Machine_Learning_2/ml2_appendix.tex",
    "content": "\\section{Appendix Math}\nHere we revisit some important mathematical tricks and equations to know.  \n\\subsection{Useful properties of a Gaussian}\nGiven $\\bm{x}\\sim \\mathcal{N}(\\bm{\\mu}, \\bm{\\Sigma})$, suppose $\\bm{x}=(\\bm{x}_a, \\bm{x}_b)$, define $\\bm{\\mu}=(\\bm{\\mu}_a, \\bm{\\mu}_b)$ and $\\bm{\\Sigma} = \\begin{bmatrix}\n\\Sigma_{aa} & \\Sigma_{ab}\\\\ \\Sigma_{ba} & \\Sigma_{bb}\n\\end{bmatrix}$ ($\\Sigma_{ab}=\\Sigma_{ba}^T$, $\\Sigma_{aa}=\\Sigma_{aa}^T$)\\\\\n\n\\textbf{Marginal distribution}: $p(\\bm{x}_a) = \\mathcal{N}(\\bm{x}_a\\mid \\bm{\\mu}_a, \\Sigma_{aa})$\\\\\n\n\\textbf{Conditional distribution}: $p(\\bm{x}_a\\vert \\bm{x}_b) = \\mathcal{N}(\\bm{x}_a\\mid \\bm{\\mu}_{a|b}, \\Sigma_{a|b})$\\\\ where $\\Sigma_{a|b} = \\Sigma_{aa} - \\Sigma_{ab}\\Sigma_{bb}^{-1}\\Sigma_{ba}, \\bm{\\mu}_{a\\vert b} = \\bm{\\mu}_a + \\Sigma_{ab}\\Sigma_{bb}^{-1}\\left(\\bm{x}_{b}-\\bm{\\mu}_{b}\\right)$\\\\[10pt]\n\n\\textbf{Multiplication of two Gaussians}: $\\mathcal{N}(\\bm{x}\\mid \\bm{a}, \\bm{A})\\mathcal{N}(\\bm{x}\\mid\\bm{b}, \\bm{B})=\\mathcal{N}(\\bm{x}|\\bm{c}, \\bm{C})\\overbrace{\\mathcal{N}(\\bm{a}|\\bm{b}, \\bm{A}+\\bm{B})}^{\\text{Normalization constant}}$\\\\ where $\\bm{C}=\\left(\\bm{A}^{-1}+\\bm{B}^{-1}\\right)^{-1}$, $\\bm{c}=\\bm{C}\\left(\\bm{A}^{-1}\\bm{a}+\\bm{B}^{-1}\\bm{b}\\right)$\\\\\n\n\\textbf{Conditional and marginals in graphical model}\\\\\n\\begin{wrapfigure}[3]{l}{0.1\\textwidth}\n\t\\centering\n\t\\tikz{ %\n\t\t\\node[latent] (x) {$\\bm{x}$} ; %\n\t\t\\node[latent, below=of x] (y) {$\\bm{y}$} ; %\n\t\t\n\t\t\\edge{x}{y}\n\t}\n\\end{wrapfigure}\n\\begin{equation*}\n\t\\begin{split}\n\t\tp(x) & = \\mathcal{N}(x|\\mu, \\Lambda^{-1})\\\\\n\t\tp(y|x) & = \\mathcal{N}(Ax+b, L^{-1})\\\\[10pt]\n\t\t\\Rightarrow p(y) & = \\mathcal{N}(y|A\\mu + b, L^{-1} + A\\Lambda^{-1}A^T)\\\\\n\t\t\\Rightarrow p(x|y) & = \\mathcal{N}(x|\\Sigma(A^T L (y - b) + \\Lambda\\mu), \\Sigma), \\hspace{2mm} \\Sigma=(\\Lambda + A^T L A)^{-1}\n\t\\end{split}\n\\end{equation*}\n\n\n\\subsection{Distributions from the exponential family}\nIt is useful to know the exponential form of a few most popular distributions. Remember that in general, a distribution of the exponential family can be written in the form:\n$$p(\\bm{x}|\\bm{\\eta}) = h(\\bm{x})g(\\bm{\\eta})\\exp\\left(\\bm{\\eta}^T \\cdot \\bm{u}(\\bm{x})\\right)$$\nSome tricks to keep in mind:\n\\begin{itemize}\n\t\\item $a^{b}=\\exp(b\\cdot \\log a)$ - helpful to find sufficient statistics and natural parameters\n\t\\item If we have the constraint $\\sum_k \\pi_k= 1$, replace $\\pi_K$ with $\\pi_K=1-\\sum_{k\\neq K} \\pi_k$ $\\Rightarrow$ one less parameter\n\\end{itemize}\n\\subsubsection{Gaussian}\n\\textbf{Univariate}:\n\\begin{fleqn}[\\parindent]\n\t\\begin{equation*}\n\t\\begin{split}\n\t& p(x|\\mu, \\sigma^2) = \\mathcal{N}(x|\\mu, \\sigma^2) = \\frac{1}{\\sqrt{2\\pi}\\sigma}\\exp\\left(-\\frac{(x-\\mu)^2}{2\\sigma^2}\\right)\\\\[8pt]\n\t& \\bm{\\eta} = \\begin{bmatrix}\n\t\\frac{\\mu}{\\sigma^2} & -\\frac{1}{2\\sigma^2}\\\\\n\t\\end{bmatrix}^T\\\\\n\t& \\bm{u}(\\bm{x}) = \\begin{bmatrix}\n\tx & x^2\\\\\n\t\\end{bmatrix}^T\\\\\n\t& h(\\bm{x}) = \\frac{1}{\\sqrt{2\\pi}}\\\\\n\t& g(\\bm{\\eta}) = (-2\\eta_2)^{-\\frac{1}{2}}\\exp\\left(\\frac{\\eta_1^2}{4\\cdot \\eta_2}\\right)\\\\\n\t\\end{split}\n\t\\end{equation*}\n\\end{fleqn}\n\\textbf{Multivariate}:\n\\begin{fleqn}[\\parindent]\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t& p(\\bm{x}|\\bm{\\mu}, \\bm{\\Sigma}) = \\mathcal{N}(\\bm{x}|\\bm{\\mu}, \\bm{\\Sigma}) = (2\\pi)^{-D/2}|\\Sigma|^{-1}\\cdot \\exp\\left(-\\frac{1}{2}(\\bm{x}-\\bm{\\mu})^T\\Sigma^{-1}(\\bm{x}-\\bm{\\mu})\\right)\\\\[8pt]\n\t\t\t& \\bm{\\eta} = \\begin{bmatrix}\n\t\t\t\\bm{\\Sigma}^{-1}\\bm{\\mu} & -\\frac{1}{2}\\bm{\\Sigma}^{-1}\\\\\n\t\t\t\\end{bmatrix}^T\\\\\n\t\t\t& \\bm{u}(\\bm{x}) = \\begin{bmatrix}\n\t\t\t\t\\bm{x} & \\bm{x}\\bm{x}^T\\\\\n\t\t\t\\end{bmatrix}^T\\\\\n\t\t\t& h(\\bm{x}) = (2\\pi)^{-D/2}\\\\\n\t\t\t& g(\\bm{\\eta}) = |-2\\bm{\\eta}_2|^{-\\frac{1}{2}} \\cdot \\exp\\left(\\frac{1}{4}\\bm{\\eta}_{1}^T\\bm{\\eta}_{2}^{-1}\\bm{\\eta}_{1}\\right)\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\\end{fleqn}\n\\subsubsection{Beta}\n\\begin{fleqn}[\\parindent]\n\t\\begin{equation*}\n\t\\begin{split}\n\t& p(x|\\alpha, \\beta) = \\frac{x^{\\alpha-1}(1 - x)^{\\beta-1}}{B(\\alpha, \\beta)} \\hspace{4mm}\\text{where}\\hspace{4mm}B(\\alpha, \\beta)=\\frac{\\Gamma(\\alpha)\\Gamma(\\beta)}{\\Gamma(\\alpha + \\beta)}\\\\[8pt]\n\t& \\bm{\\eta} = \\begin{bmatrix}\n\t\\alpha & \\beta\\\\\n\t\\end{bmatrix}^T\\\\\n\t& \\bm{u}(\\bm{x}) = \\begin{bmatrix}\n\t\\log x & \\frac{1}{x}\\\\\n\t\\end{bmatrix}^T\\\\\n\t& h(\\bm{x}) = 1\\\\\n\t& g(\\bm{\\eta}) = \\frac{1}{B(\\eta_1, \\eta_2)}\\\\\n\t\\end{split}\n\t\\end{equation*}\n\\end{fleqn}\n\\subsubsection{Multinomial}\n\\begin{fleqn}[\\parindent]\n\t\\begin{equation*}\n\t\\begin{split}\n\t& p(\\bm{x}|\\bm{\\pi}) = \\frac{M!}{\\prod_{i=1}^{K}x_i!}\\prod_{i=1}^{K}\\pi_i^{x_i}\\\\[8pt]\n\t& \\bm{\\eta} = \\begin{bmatrix}\n\t\\ln\\frac{\\pi_1}{1-\\sum_{i=1}^{K-1}\\pi_i} & \\ln\\frac{\\pi_2}{1-\\sum_{i=1}^{K-1}\\pi_i} & ... & \\ln\\frac{\\pi_{K-1}}{1-\\sum_{i=1}^{K-1}\\pi_i}\n\t\\end{bmatrix}^T\\\\\n\t& \\bm{u}(\\bm{x}) = \\begin{bmatrix}\n\tx_1 & x_2 & ... & x_{K-1}\n\t\\end{bmatrix}^T\\\\\n\t& h(\\bm{x}) = \\frac{M!}{\\prod_{i=1}^{K}x_i!}\\\\\n\t& g(\\bm{\\eta}) = \\exp\\left(-M\\ln \\left(1 + \\sum_{i=1}^{K-1}\\exp(\\eta_i)\\right)\\right)\\\\\n\t\\end{split}\n\t\\end{equation*}\n\\end{fleqn}\n\\subsubsection{Dirichlet}\n\\begin{fleqn}[\\parindent]\n\t\\begin{equation*}\n\t\\begin{split}\n\t& p(\\bm{x}|\\alpha_1,..,\\alpha_K) = \\frac{1}{B(\\alpha_1,..., \\alpha_K)}\\prod_{i=1}^{K}x_i^{\\alpha_i-1} \\hspace{4mm}\\text{where}\\hspace{4mm}B(\\alpha_1,...,\\alpha_K)=\\frac{\\prod_{i=1}^{K}\\Gamma(\\alpha_i)}{\\Gamma(\\sum_{i=1}^{K} \\alpha_i)}\\\\[8pt]\n\t& \\bm{\\eta} = \\begin{bmatrix}\n\t\\alpha_1 & ... & \\alpha_K\\\\\n\t\\end{bmatrix}^T\\\\\n\t& \\bm{u}(\\bm{x}) = \\begin{bmatrix}\n\t\\log x_1 & ... & \\log x_K\\\\\n\t\\end{bmatrix}^T\\\\\n\t& h(\\bm{x}) = \\frac{1}{\\prod_{i=1}^{K}x_i}\\\\\n\t& g(\\bm{\\eta}) = \\frac{1}{B(\\eta_1, ..., \\eta_K)}\\\\\n\t\\end{split}\n\t\\end{equation*}\n\\end{fleqn}\n\\subsubsection{Poisson}\n\\begin{fleqn}[\\parindent]\n\t\\begin{equation*}\n\t\\begin{split}\n\t& p(x|\\lambda) = \\frac{\\lambda^{x}\\exp(-x)}{x!}\\\\[8pt]\n\t& \\bm{\\eta} = \\begin{bmatrix}\n\t\\ln \\lambda \\\\\n\t\\end{bmatrix}^T\\\\\n\t& \\bm{u}(\\bm{x}) = \\begin{bmatrix}\n\tx\\\\\n\t\\end{bmatrix}^T\\\\\n\t& h(\\bm{x}) = \\frac{1}{x!}\\\\\n\t& g(\\bm{\\eta}) = \\exp(-\\exp(\\eta))\\\\\n\t\\end{split}\n\t\\end{equation*}\n\\end{fleqn}\n\\subsubsection{Gamma}\n\\begin{fleqn}[\\parindent]\n\t\\begin{equation*}\n\t\\begin{split}\n\t& p(x|a,b) = \\frac{1}{\\Gamma(x)}b^{a}x^{a-1}\\exp(-bx)\\\\[8pt]\n\t& \\bm{\\eta} = \\begin{bmatrix}\n\t(a-1) & -b \\\\\n\t\\end{bmatrix}^T\\\\\n\t& \\bm{u}(\\bm{x}) = \\begin{bmatrix}\n\t\\ln x & x\\\\\n\t\\end{bmatrix}^T\\\\\n\t& h(\\bm{x}) = 1\\\\\n\t& g(\\bm{\\eta}) = \\frac{(-\\eta_2)^{\\eta_1+1}}{\\Gamma(\\eta_1+1)}\\\\\n\t\\end{split}\n\t\\end{equation*}\n\\end{fleqn}\n\\subsubsection{Conjugate priors}\n\\begin{itemize}\n\t\\item Dirichlet $\\to$ Multinomial \n\t\\item Dirichlet $\\to$ Categorical \n\t\\item Beta $\\to$ Bernoulli\n\t\\item Gamma $\\to$ Poisson\n\t\\item Gaussian $\\to$ Gaussian\n\t\\item Gamma (precision) $\\to$ Gaussian (known mean)\n\\end{itemize}"
  },
  {
    "path": "Machine_Learning_2/ml2_causality.tex",
    "content": "\\section{Causality}\n\\begin{itemize}\n\t\\item Causality is about testing whether one event (\\textit{effect}) is the result of the occurrence of another event (\\textit{cause}), i.e. a change in the cause will lead to a change in the effect. It differs from correlation by \\textit{explaining}/\\textit{finding} the relationship behind variables\n\t\\item While we look at the data distribution for correlation, we are focusing on the generation mechanism of the data in causality. While in statistics we then like to predict the next observation (or its likelihood), causality is interested in what happens if we perform interventions (setting a variable to a certain value)\n\t\\item The most important operator in causality is the \\textbf{do-operator}: $p(A=a|\\Cdo(B=b))$. It differs from the standard conditional probability by not assuming that we have observed $B=b$, but that we externally set the value of $B$. This means that we cannot infer anything from its parents as in standard conditionals (if we observe $B=b$, then this usually gives us information about its parents).\n\t\n\tNote that there are cases where $p(A=a|\\Cdo(B=b))=p(A=a|B=b)$. One obvious example for this is when $B$ has no parents in its corresponding graphical model.\n\t\\item We start with a discussion about the terminology in causality, and then take a closer look at Causal Bayesian networks and causal reasoning\n\\end{itemize}\n\\subsection{Causality terminology}\n\\begin{itemize}\n\t\\item $A$ is said to \\underline{cause} $B$ if changing $A$ leads to a change in $B$\n\t\\item Similar to graphical models, we can define Causal graphs that represent causal relationships. An edge from $A$ to $B$ in the graph means that $A$ \\textit{causes} $B$ even if all other variables are kept fixed. Figure~\\ref{fig:causality_overview_causal_graphs} gives an overview.\n\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.6\\textwidth]{figures/causality_overview_causal_graphs.png}\n\t\t\\caption{Examples of causal graphs (retrieved from lecture notes).}\n\t\t\\label{fig:causality_overview_causal_graphs}\n\t\\end{figure}\n\t\n\tNote that these graphs can contain loops, which represents a feedback loop (a change in $A$ leads to a change in $B$, and a change in $B$ leads to a change in $A$). However, we will not take a closer look at those.\n\t\n\t\\item We can interpreted node relationships in the graph as causal relations:\n\t\\begin{itemize}\n\t\t\\item $A$ is a \\textit{parent} of $B$ $\\implies$ $A$ is a direct cause of $B$\n\t\t\\item $A$ is a \\textit{child} of $B$ $\\implies$ $A$ is a direct effect of $B$\n\t\t\\item $A$ is a \\textit{ancestor} of $B$ (e.g. $A\\to C \\to B$) $\\implies$ $A$ is a cause of $B$. Note that if we fix $C$, there is no effect of $A$ on $B$. Hence, there is no direct edge.\n\t\t\\item $A$ is a \\textit{descendant} of $B$ (e.g. $B\\to C\\to A$) $\\implies$ $A$ is an effect of $B$.\n\t\\end{itemize}\n\t\\item We use the notation $\\mathcal{G}_{\\overline{X}}$ to denote a sub-graph of $\\mathcal{G}$ in which the incomming edges of $X$ are removed. This is useful for discussing when $X$ is set externally (hence, no influence of parents of $X$ in that case).\n\t\n\tSimilarly, $\\mathcal{G}_{\\underline{X}}$ is $\\mathcal{G}$ without the outgoing edges of $X$.\n\t\\item  A \\textbf{perfect intervention} $\\Cdo(X=\\xi)$ means that we force $X$ to be the value $\\xi$. Thereby, the graph $\\mathcal{G}$ changes to $\\mathcal{G}_{\\overline{X}}$\n\t\\begin{itemize}\n\t\t\\item To perform intervention, we require \\textit{modularity}, meaning that we can manipulate $X$ without influencing any other variables in the graph $\\bm{V}\\setminus X$\n\t\t\\item This can be a challenge in real systems, but in our theoretical models, we can assume that we are able to do so\n\t\\end{itemize}\n\t\\item A variable $H$ is a \\textbf{confounder} of $X$ and $Y$ (i.e. $H$ confounds $X$ and $Y$) if there is a directed path from $H$ to $X$ which does not include $Y$, and same from $H$ to $Y$. Note that it is still allowed to have other paths between $X$ and $Y$. Examples of confounders are shown in Figure~\\ref{fig:causality_confounders_examples}.\n\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/causality_confounders_examples.png}\n\t\t\\caption{Examples of when a node $H$ is a confounder of $X$ and $Y$ (retrieved from lecture notes).}\n\t\t\\label{fig:causality_confounders_examples}\n\t\\end{figure}\n\n\t\\item \\textbf{Reichenbach's principle} sets correlation and causality into relation: if $X$ and $Y$ are correlated/depending on each other, then there is either a causal relation of the type $X\\to Y$, $Y\\to X$ or there exists a confounder $H$ of $X$ and $Y$.\n\t\\begin{itemize}\n\t\t\\item Note that this principle can fail if we have a selection bias, meaning that the dataset of $X$ and $Y$ was obtained by only including samples that are conditional on some (possibly latent) event.\n\t\\end{itemize}\n\\end{itemize}\n\n\\subsection{Causal Bayesian Networks}\n\\begin{itemize}\n\t\\item An subspace of causal networks with many assumptions/limitations, but therefore easier to work with, are Causal Bayesian Networks. We make the following assumptions:\n\t\\begin{itemize}\n\t\t\\item No confounding % A graph $\\mathcal{G}$ does not contain any confounder\n\t\t\\item A graph $\\mathcal{G}$ does not contain any loops\n\t\t\\item We do not have any selection bias in the data, nor measurements error or time dependencies\n\t\\end{itemize}\n\t\\item We call a Bayesian Network causal if:\n\t\\begin{itemize}\n\t\t\\item Directed edges correspond with directed causal relations\n\t\t\\item After a perfect intervention $\\Cdo(X_{I}=x_I)$, the probability density becomes:\n\t\t\\begin{equation*}\n\t\t\\tcbox[nobeforeafter]{\\(\n\t\t\t\\begin{split}\n\t\t\t\tp\\left(\\bm{X}_{\\bm{V}\\setminus I}|\\Cdo(X_I=x_I)\\right) = \\prod_{i\\in \\bm{V}\\setminus I} p\\left(x_i|\\bm{x}_{\\text{pa}_i}\\right)\n\t\t\t\\end{split}\n\t\t\t\\)}\n\t\t\\end{equation*}\n\t\t\n\t\\end{itemize}\n\\end{itemize}\n\n\\subsection{Causal Reasoning}\n\\begin{itemize}\n\t\\item The goal of causal reasoning is to estimate $p(y|\\Cdo(X=x))$. If we can express it in terms of the observational distribution $p(x,y,...)$ we say it is \\textbf{identifiable} from the observational distribution. \n\t\n\tNote that it does not necessarily require all variables to be observable.\n\t\\item Assume we have the following Bayesian Causal Network:\n\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\tikz{ %\n\t\t\t\\node[latent] (X) {$X$} ; %\n\t\t\t\\node[latent, right=of X] (H) {$H$} ; %\n\t\t\t\\node[latent, below=of H] (Y) {$Y$} ; %\n\t\t\t\n\t\t\t\\edge{H}{X};\n\t\t\t\\edge{H}{Y};\n\t\t\t\\edge{X}{Y};\n\t\t}\n\t\\end{figure}\n\n\tThe standard conditional distribution is:\n\t$$p(y|x)=\\int p(y|h,x)p(h|x)dh$$\n\n\tNow, assume we perform a perfect intervention on $X$, i.e. $\\Cdo(X=x)$. What happens is that we neglect the effect of $H$ on $X$, \\textit{but} we still need to consider the effect of $H$ on $Y$. Hence, the conditional becomes:\n\t$$p(y|\\Cdo(X=x))=\\int p(h)p(y|h,x)dh$$\n\tThe important thing is that we have to prevent that changing $X$ influences $H$ by ``back-reasoning'' (i.e. $H$ causes $X$, but observing $X$ gives us information of $H$), which again influence $Y$. This is because we cannot change $H$ by just forcing $X$ to a value, as we \\textit{overwrite} the effect of $H$ on $X$. Hence, we have to explicitly remove its dependency on $X$ in the integral.\n\t\n\t%The result indicates that we had to \\textit{adjust} the conditional probability for the effect of $H$. But suppose, we would not have the connection between $H$ and $Y$. Then, we would not have to adjust for $H$, and get $p(y|\\Cdo(X=x))=p(y|x)$.\n\t\\item We can derive a more general algorithm for deciding, for which variables we need to \\textit{adjust} our conditional probability for. This can be done in a very similar manner to d-separation, as we need to find all variables, that are implicitly changed by setting $X$ to a certain value (i.e. variables that influence the decision of which value $X$ can have), but then also influence $Y$. We do not want this influence because by forcing $X$ to be a certain value, we cannot change variables that cause $X$. Hence, we are trying to find a set of variables $S$ which break these kind of influences, and remove their dependency with $X$.\n\t\n\t\\item In general, we can determine whether $S$ is a sufficient set of variables we are adjusting by the following check:\n\t\n\t\\begin{tcolorbox}[colback=white!80!gray,colframe=gray!75!black,title=Back-door criterion]\n\t\tA set of variables $S$ satisfies the back-door criterion relative to a variable pair ($X$, $Y$), if:\n\t\t\\begin{enumerate}\n\t\t\t\\item $X, Y\\not\\in S$\n\t\t\t\\item No node of $S$ is a descendant (i.e. child of a child etc.) of $X$\n\t\t\t\\item $S$ blocks all paths from $Y$ to $X$ where we have an incoming edge to $X$ (other directions irrelevant for path itself). A path is blocked by $S$ if:\n\t\t\t\\begin{enumerate}\n\t\t\t\t\\item It contains a collider $...\\rightarrow u \\leftarrow ...$ such that $u$ is not an ancestor of a node in $S$\n\t\t\t\t\\item It contains a non-collider $...\\rightarrow u$, $...\\rightarrow u \\rightarrow ...$, $...\\leftarrow u \\rightarrow ...$ such that $u$ is in $S$\n\t\t\t\\end{enumerate}\n\t\t\\end{enumerate}\n\t\tThen $S$ is admissible for adjustment to find the causal effect of $X$ on $Y$:\n\t\t$$p(y|\\Cdo(X=x))=\\int p(y|X=x,S=s)p(S=s)ds$$\n\t\tIf $S=\\emptyset$: $p(y|\\Cdo(X=x))=p(y|X=x)$\n\t\\end{tcolorbox}\t\n\n\tTo find the actual set $S$, we can simply perform the algorithm backwards. First, find all paths from $Y$ to $X$ with an incoming edge to $X$. Then, we start with $S=\\emptyset$, and try to block all paths by adding variables to $S$. In case that all paths were blocked from the beginning on, or no paths exist, we can stop with $S=\\emptyset$.\n\t\n\tNote that simplest solution of $S$ is the set of all the nodes with an edge to $X$. It might not be the smallest set, but should be always a valid one as we do not allow loops in BNs.\n\t\n\t\\item \\underline{Examples}: \n\t\\begin{itemize}\n\t\t\\item Consider the examples in Figure~\\ref{fig:causality_backdoor_example}. Note that we usually try to find the smallest set of admissible variables as this simplifies the integral we have to take.\n\t\t\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.5\\textwidth]{figures/causality_backdoor_example.png}\n\t\t\t\\caption{Example of admissible sets of variables for adjustment (retrieved from lecture notes).}\n\t\t\t\\label{fig:causality_backdoor_example}\n\t\t\\end{figure}\n\t\t\\newpage\n\t\t\\item Consider the following, slightly more complicated Causal Bayesian Network:\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\tikz{ %\n\t\t\t\t\\node[latent] (X1) {$X_1$} ; %\n\t\t\t\t\\node[latent, below=of X1] (X3) {$X_3$} ; %\n\t\t\t\t\\node[latent, right=of X3] (X4) {$X_4$} ; %\n\t\t\t\t\\node[latent, right=of X4] (X5) {$X_5$} ; %\n\t\t\t\t\\node[latent, above=of X5] (X2) {$X_2$} ; %\n\t\t\t\t\\node[latent, below=of X3] (X7) {$X_7$} ; %\n\t\t\t\t\\node[latent, right=of X7] (X6) {$X_6$} ; %\n\t\t\t\t\\node[latent, right=of X6] (X8) {$X_8$} ; %\n\t\t\t\t\n\t\t\t\t\\edge{X1}{X3};\n\t\t\t\t\\edge{X1}{X4};\n\t\t\t\t\\edge{X3}{X7};\n\t\t\t\t\\edge{X4}{X7};\n\t\t\t\t\\edge{X4}{X8};\n\t\t\t\t\\edge{X7}{X6};\n\t\t\t\t\\edge{X6}{X8};\n\t\t\t\t\\edge{X5}{X8};\n\t\t\t\t\\edge{X2}{X5};\n\t\t\t\t\\edge{X2}{X4};\n\t\t\t}\n\t\t\\end{figure}\t\n\t\n\t\tWe are trying to find the set $S$ admissible for adjustment for the causal effect of $X_7$ on $X_8$ (the two nodes on the bottom, left and right). We have to consider all paths with incoming edges to $X_7$, so from $X_3$ and $X_4$. To block the path $X_8\\to X_4\\to X_7$, we add $X_4$ to $S$: $S=\\{X_4\\}$. However, by doing this, we unblocked another path: $X_8\\to X_5\\to X_2\\to X_4\\to X_1\\to X_3\\to X_7$. $X_4$ is not longer a collider anymore, as it is included in $S$. So, we can either add $X_3$, or $X_5$, or even $X_1$ or $X_2$ to block this path. For example, we can take $S=\\{X_3,X_4\\}$, which is then admissible for adjustment as all paths are blocked.\n\t\\end{itemize}\n\t\\item Although we found a way to estimate what happens when we perform an intervention, the best way to find causal relations is to use randomized controlled trials. In a drug test, this would mean that we completely random assign a person to take the drug or not, ensuring that no underlying selection bias is in the process. By that, we should break all back-door paths (as there is nothing besides a coin flip that causes the event of  \"taking the drug\")\n\n\\end{itemize}"
  },
  {
    "path": "Machine_Learning_2/ml2_exponential_family.tex",
    "content": "\\section{Introduction to popular distributions and their properties}\n\\begin{itemize}\n\t\\item This section (lecture 1 and 2) reviews different kinds of distributions, including the exponential family, Student-t distribution and common distributions for binary and discrete variables\n\t\\item Furthermore, we shortly introduce Independent Component Analysis and Information theory\n\t\\item In general, the first two lectures gave some fundamental knowledge we will use a couple of times for the rest of the course\n\t\\item More mathematical tricks or examples of the exponential family can be found in the appendix\n\\end{itemize}\n\\subsection{Exponential family distributions}\n\\textbf{(Bishop 2.4)}\n\\begin{itemize}\n\t\\item A distribution is considered a member of the exponential family if it can be written as follows:\n\t\\begin{equation*}\n\t\\tcbox[nobeforeafter]{\\(\n\t\t\\begin{split}\n\t\t\tp(\\bm{x}|\\bm{\\eta}) & = h(\\bm{x})g(\\bm{\\eta})\\exp\\left(\\bm{\\eta}^T \\cdot \\bm{u}(\\bm{x})\\right)\\\\[5pt]\n\t\t\t\\bm{\\eta} & \\hspace{3mm}\\text{natural parameters}\\\\\n\t\t\t\\bm{u}(\\bm{x}) & \\hspace{3mm}\\text{sufficient statistics}\\\\\n\t\t\\end{split}\n\t\\)}\n\t\\end{equation*}\n\t\\item $\\bm{u}(\\bm{x})$ is called sufficient statistics because for the maximum likelihood estimate of $\\bm{\\eta}$, it is sufficient to record $\\sum_{n=1}^{N}\\bm{u}(\\bm{x}_n)$ instead of the whole dataset $\\left\\{\\bm{x}_n\\right\\}_{n=1}^{N}$ (see below for ML estimate)\n\t\\item An important property of the exponential families is that the moments of distributions (i.e. mean and variance) can be determined by deriving $-\\ln g(\\bm{\\eta})$ by $\\bm{\\eta}$:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\text{Normalization constant}\\hspace{2mm} z(\\bm{\\eta}) & = \\frac{1}{g(\\bm{\\eta})} = \\int h(\\bm{x})\\exp\\left(\\bm{\\eta}^T \\cdot \\bm{u}(\\bm{x})\\right) d\\bm{x}\\\\\n\t\t\t\\frac{\\partial}{\\partial \\bm{\\eta}} -\\ln g(\\bm{\\eta}) & = \\frac{1}{z(\\bm{\\eta})} \\int h(\\bm{x})\\bm{u}(\\bm{x})\\exp\\left(\\bm{\\eta}^T \\cdot \\bm{u}(\\bm{x})\\right) d\\bm{x} = \\E[\\bm{u}(\\bm{x})\\vert \\bm{\\eta}]\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\begin{itemize}\n\t\t\\item Note that these moments are of the sufficient statistics $\\bm{u}(\\bm{x})$, and not $\\bm{x}$\n\t\t\\item Additionally, the second moment around the mean can be determined by: $\\nabla_{\\bm{\\eta}}^2 -\\ln g(\\bm{\\eta})$\n\t\\end{itemize}\n\t\\item From the first moment, we can show that the \\underline{MLE solution} of the natural parameters are:\n\t$$\\tcbox[nobeforeafter]{\\(-\\nabla_{\\bm{\\eta}}\\ln g(\\bm{\\eta}) = \\E[\\bm{u}(\\bm{x})\\vert \\bm{\\eta}] \\implies -\\nabla_{\\bm{\\eta}}\\ln g(\\bm{\\eta}_{\\text{ML}}) = \\frac{1}{N}\\sum_{n=1}^{N} \\bm{u}(\\bm{x})\\)}$$\n\\end{itemize}\n\\subsubsection{Conjugate priors}\n\\begin{itemize}\n\t\\item A conjugate prior $p(\\bm{\\eta})$ is conjugate to the likelihood so that the posterior $p(\\bm{\\eta}|\\bm{X})$ has the same form as the prior\n\t\\item Each member of the exponential family has a conjugate prior\n\t\\item To find the conjugate prior for a exponential distribution as likelihood, we only have to look at $\\bm{\\eta}$ of the likelihood and $\\bm{u}(\\bm{x})$ of the prior take on the same form. Then, we simply get:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\bm{u}(\\bm{x})_{\\text{posterior}} & = \\bm{\\eta}_{\\text{likelihood}} = \\bm{u}(\\bm{x})_{\\text{prior}}\\\\\n\t\t\t\\bm{\\eta}_{\\text{posterior}} & = \\bm{u}(\\bm{x})_{\\text{likelihood}} + \\bm{\\eta}_{\\text{prior}}\n\t\t\\end{split}\n\t\\end{equation*}\n\\end{itemize}\n\\subsubsection{Bayesian Inference for Gaussian}\n\\begin{itemize}\n\t\\item We can demonstrate the conjugate prior idea for Gaussians (one dimensional), where we have to distinguish three cases\n\\end{itemize}\n\\begin{enumerate}\n\t\\item \\underline{Variance known, mean estimated}\n\t\\begin{itemize}\n\t\t\\item Conjugate prior is a Gaussian $p(\\mu)=\\mathcal{N}(\\mu\\vert\\mu_0, \\sigma_0^2)$ such that our posterior has the distribution:\n\t\t\\begin{equation*}\n\t\t\\tcbox[nobeforeafter]{\\(\n\t\t\t\\begin{split}\n\t\t\t\t& \\textbf{Variance known, mean estimated}\\\\\n\t\t\t\t& p(\\mu|\\mathcal{D})=\\mathcal{N}(\\mu|\\mu_N, \\sigma_N^2), \\hspace{5mm}\\mu_N= \\frac{\\sigma^2 \\mu_0 + N\\sigma_0^2 \\mu_{\\text{ML}}}{N\\sigma_0^2 + \\sigma^2}, \\hspace{5mm}\\frac{1}{\\sigma_N^2}=\\frac{1}{\\sigma_0^2} + \\frac{N}{\\sigma^2}\n\t\t\t\\end{split}\n\t\t\\)}\n\t\t\\end{equation*}\n\t\\end{itemize}\n\t\\item \\underline{Mean unknown, variance estimated}\n\t\\begin{itemize}\n\t\t\\item Conjugate prior for the precision $\\lambda=\\frac{1}{\\sigma^2}$ is a Gamma distribution $\\text{Gamma}(\\lambda|a_0, b_0)$ such that the posterior is:\n\t\t\\begin{equation*}\n\t\t\\tcbox[nobeforeafter]{\\(\n\t\t\t\\begin{split}\n\t\t\t& \\textbf{Mean known, variance estimated}\\\\\n\t\t\t& p(\\lambda|\\mathcal{D})=\\text{Gamma}(\\lambda|a_N,b_N), \\hspace{5mm}a_N=a_0+\\frac{N}{2},\\hspace{5mm}b_N = b_0+\\frac{1}{2}\\sum_n(x_n-\\mu)^2\n\t\t\t\\end{split}\n\t\t\t\\)}\n\t\t\\end{equation*}\n\t\\end{itemize}\n\t\\item \\underline{Variance and mean estimated}\n\t\\begin{itemize}\n\t\t\\item If both are unknown, we have a ``normal-Gamma'' distribution as prior and posterior: $p(\\mu,\\lambda|a,b,\\mu_0, \\beta)=\\mathcal{N}(\\mu|\\mu_0, (\\beta \\lambda^{-1}))\\text{Gamma}(\\lambda|a,b)$\n\t\t\\item Finding the posterior is harder in this case because of the combined distribution. For details, see Bishop, but in the lecture it was not further discussed\n\t\\end{itemize}\n\\end{enumerate}\n\\subsection{Student's-t distribution}\n\\begin{itemize}\n\t\\item The Student-t distribution is \"heavy-tailed\", meaning that the probability for data points decreases slower with the distance from the mean/center than for a Gaussian (polynomial $\\text{St}(x)\\propto |x|^{-\\alpha}$ instead of exponential $\\mathcal{N}\\propto e^{-\\frac{x^2}{\\sigma^2}}$)\n\t\\item This makes the distribution more \\underline{robust against outliers} as the MLE solution is less influenced by those and focuses more on the biggest data point mass (see Figure~\\ref{fig:exponential_families_student_t})\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\begin{subfigure}{0.25\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=\\textwidth]{figures/exponential_families_student_t.png}\n\t\t\t\\caption{MLE estimate}\n\t\t\t\\label{fig:exponential_families_student_t}\n\t\t\\end{subfigure}\n\t\t\\hspace{10mm}\n\t\t\\begin{subfigure}{0.3\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=\\textwidth]{figures/exponential_families_student_t_nu.png}\n\t\t\t\\caption{Effect of parameter $\\nu$}\n\t\t\t\\label{fig:exponential_families_student_t_nu}\n\t\t\\end{subfigure}\n\t\t\\caption{(a) Comparison of MLE solution of Student-t distribution (red) and Gaussian (green). (b) The parameter $\\nu$ for fixed $\\mu=0$ and $\\lambda=1$. For $\\nu\\to\\infty$, }\n\t\\end{figure}\n\t\\item It emerges from a infinite mixture of Gaussians with a fixed mean and the precision (i.e. inverse variance) distributed as a Gamma distribution:\n\t\\begin{enumerate}\n\t\t\\item Draw precision $\\tau \\sim \\text{Gamma}(a,b)$\n\t\t\\item Draw $x\\sim \\mathcal{N}(\\mu, \\tau^{-1})$\n\t\\end{enumerate}\n\tThen the resulting $x$ will be distributed according to the Student-t distribution\n\t$$p(x) \\sim \\text{St}(x\\mid \\mu, \\lambda=a/b, \\nu=2a)$$\n\t\\item By marginalizing out $\\tau$, we can derivate the PDF of the student distribution:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\text{Scalar}\\hspace{2mm}&\\text{St}(x\\mid\\mu, \\lambda=a/b, \\nu=2a) = \\frac{b^a}{\\Gamma(a)\\sqrt{2\\pi}}\\left(b + \\frac{(x-\\mu)^2}{2}\\right)^{-a-\\frac{1}{2}}\\Gamma\\left(a+\\frac{1}{2}\\right)\\\\[8pt]\n\t\t\t\\text{d-dimensional}\\hspace{2mm} &  \\text{St}(\\bm{x}\\mid\\bm{\\mu}, \\bm{\\Sigma}, \\nu) = \\frac{\\Gamma\\left(\\frac{d}{2} + \\frac{\\nu}{2}\\right)}{\\Gamma\\left(\\frac{d}{2}\\right)}\\frac{1}{\\left(\\pi\\nu\\right)^{d/2}\\left|\\bm{\\Sigma}\\right|^{1/2}}\\left(1+\\nu^{-1}\\left(\\bm{x}-\\bm{\\mu}\\right)^T \\bm{\\Sigma}^{-1}\\left(\\bm{x}-\\bm{\\mu}\\right)\\right)^{-\\frac{d}{2}-\\frac{\\nu}{2}}\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item The parameter $\\lambda$ is often called precision, but does not exactly represent the inverse of the variance.\n\t\\item $\\nu$ is called the degrees of freedom (see Figure~\\ref{fig:exponential_families_student_t_nu}). For $\\nu\\to\\infty$, the student-t distribution becomes a Gaussian $\\mathcal{N}(x\\vert\\mu, \\lambda^{-1})$\n\\end{itemize}\n\\subsection{Distributions for Binary and Discrete Variables}\n\\begin{itemize}\n\t\\item In this section, we review common distributions for binary and discrete distributions. We can actually find one-to-one correlations in the binary and categorical space:\n\t\\begin{table}[ht!]\n\t\t\\centering\n\t\t\\begin{tabular}{c|c}\n\t\t\tBinary & Discrete\\\\\\hline\n\t\t\tBernoulli & Categorical\\\\\n\t\t\tBinomial & Multinomial\\\\\n\t\t\tBeta & Dirichlet\n\t\t\\end{tabular}\n\t\t\\vspace{-5mm}\n\t\\end{table}\n\\end{itemize}\n\\subsubsection{Binary}\n\\begin{description}\n\t\\item[Bernoulli distribution] can be interpreted as a coin flip, and models a single binary outcome:\n\t$$\\text{Bern}(x|\\mu)=\\mu^{x}(1-\\mu)^{1-x}, \\hspace{3mm}x\\in\\{0,1\\}$$\n\t\\begin{itemize}\n\t\t\\item Expectation $\\E[x|\\mu]=\\mu$\n\t\t\\item Variance $\\mathbb{V}\\text{ar}[x]=\\E[x^2]-\\E[x]^2=\\mu(1-\\mu)$\n\t\t\\item Maximum likelihood estimate $\\mu_{\\text{ML}}=\\frac{1}{N}\\sum_{n=1}^{N} x_n$ (sensitive to overfitting for small dataset)\n\t\t\\item Exponential family $p(x|\\eta)=\\sigma(-\\eta)\\exp(\\eta\\cdot x), \\eta=\\ln \\frac{\\mu}{1-\\mu}$\n\t\\end{itemize}\n\n\t\\item[Binomial distribution] models $N$ i.i.d. Bernoulli experiments, where we define $m$ as $m=\\sum_{i=1}^{N}x_i$, i.e. the number of times the outcome is $1$:\n\t$$\\text{Bin}(m|N,\\mu)=\\frac{N!}{(N-m)!m!}\\mu^{m}(1 - \\mu)^{N-m}$$\t\n\t\\begin{itemize}\n\t\t\\item Expectation $\\E[m]=\\sum_{i=1}^{N}\\E[x_i]=N\\cdot \\mu$\n\t\t\\item Variance $\\mathbb{V}\\text{ar}[x]=N\\cdot \\mu(1-\\mu)$\n\t\t\\item Maximum likelihood estimate $\\mu_{\\text{ML}}=\\frac{m}{N}$\n\t\t\\item Exponential family $p(m|\\eta)=\\frac{N!}{(N-m)!m!}\\cdot \\exp(N \\log 1-\\mu) \\cdot \\exp(m\\log\\frac{\\mu}{1-\\mu})$, $\\eta=\\log \\frac{\\mu}{1-\\mu}$\n\t\t\\item Conjugate prior: Beta distribution. The posterior is: $\\text{Beta}(\\mu|a+m,b+N-m)$\n\t\\end{itemize}\n\n\t\\item[Beta distribution] is the conjugate prior for the binomial distribution\n\t$$\\text{Beta}(\\mu|a,b)=\\frac{\\Gamma(a+b)}{\\Gamma(a)\\Gamma(b)}\\mu^{a-1}(1-\\mu)^{b-1}$$\n\t\\begin{itemize}\n\t\t\\item Expectation $\\E[\\mu]=\\frac{a}{a+b}$\n\t\t\\item Variance $\\mathbb{V}\\text{ar}[x]=\\frac{ab}{(a+b)^2(a+b+1)}$\n\t\t\\item Exponential family: see Appendix\n\t\\end{itemize}\n\\end{description}\n\\subsubsection{Discrete}\n\\begin{description}\n\t\\item[Categorical distribution] considers a single sample, and assign each category a different probability. The input $\\bm{x}$ is a one-hot vector.\n\t$$\\text{Cat}(\\bm{x}|\\bm{\\mu})=\\prod_{k=1}^{K}\\mu_k^{x_k}=\\mu_{x_k}, \\hspace{3mm}\\sum_k \\mu_k = 1$$\n\t \\begin{itemize}\n\t \t\\item Expectation $\\E[\\bm{x}]=\\bm{\\mu}$\n\t \t\\item Covariance $\\text{Cov}[\\bm{x}]=\\text{diag}(\\bm{\\mu}(1-\\bm{\\mu}))$\n\t \t\\item Maximum likelihood estimate  $\\bm{\\mu}_{\\text{ML}}=\\frac{1}{N}\\sum_{i=1}^{N} \\bm{x}$\n\t \t\\item Exponential family $p(\\bm{x}|\\bm{\\eta})=\\frac{1}{1+\\sum_{k=1}^{K-1}\\exp(\\eta_k)}\\cdot \\exp(\\bm{\\eta}^T\\bm{x})$, $\\eta_k=\\ln\\frac{\\mu_k}{1-\\sum_{j=1}^{K-1}\\mu_j}$\n\t \\end{itemize}\n\t\\item[Multinomial distribution] takes $N$ i.i.d. categorical observations into account, where $m_k=\\sum_{n=1}^{N} x_{nk}$.\n\t$$\\text{Mult}(m_1,...,m_K|N,\\bm{\\mu})=\\frac{N!}{\\prod_{k=1}^{K}m_k!}\\prod_{k=1}^{K} \\mu_{k}^{m_k}$$\n\t\\begin{itemize}\n\t\t\\item Expectation: $\\E[\\bm{x}]=N\\cdot \\mu$\n\t\t\\item Covariance: $\\text{Cov}[\\bm{x},\\bm{x}]=N(\\text{diag}(\\bm{\\mu})-\\bm{\\mu}\\bm{\\mu}^T)$\n\t\t\\item Maximum likelihood estimation: $\\bm{\\mu}_{\\text{ML}}=\\frac{\\bm{m}}{N}$\n\t\t\\item Exponential family: see Appendix\n\t\\end{itemize}\n\t\\item[Dirichlet distribution] is the conjugate prior for multinomial\n\t$$\\text{Dir}(\\bm{\\mu}|\\bm{\\alpha})=\\frac{\\Gamma(\\sum_k \\alpha_k)}{\\prod_k\n\t\\Gamma(\\alpha_k)} \\prod_{k=1}^{K}\\mu_k^{\\alpha_k-1}$$\n\t\\begin{itemize}\n\t\t\\item Expectation $\\E[\\bm{x}]=\\frac{1}{\\sum_k \\alpha_k}\\bm{\\alpha}$\n\t\t\\item Covariance $\\text{Cov}[\\bm{x}]=-\\frac{1}{\\sum_k \\alpha_k + 1}\\bm{\\alpha}\\bm{\\alpha}^T$\n\t\t\\item Exponential family: see Appendix\n\t\\end{itemize}\n\\end{description}\n\\subsection{Independent Component Analysis}\n\\begin{itemize}\n\t\\item Independent Component Analysis (ICA) tries to reconstruct source signals from linearly mixed measurements. For example, for two sources $S(t)=\\begin{bmatrix}\n\tS_1(t)\\\\S_2(t)\n\t\\end{bmatrix}$, we assume to have the measurements:\n\t$$X(t)=\\begin{bmatrix}\n\tX_1(t)\\\\X_2(t)\n\t\\end{bmatrix} = \\begin{bmatrix}\n\t\\alpha_1 S_1(t) + \\beta_1 S_2(t)\\\\\\alpha_2 S_2(t) + \\beta_2 S_2(t)\n\t\\end{bmatrix}$$\n\tThe goal is now to find the parameter matrix\n\t$$\\bm{A}=\\begin{bmatrix}\n\t\\alpha_1 & \\beta_1\\\\ \\alpha_2 & \\beta_2\n\t\\end{bmatrix}$$\n\tto reconstruct our signals $S(t)$ from the measurements $\\bm{X}(t)=\\bm{A}\\bm{S}(t)$\n\t\\item Note that we can only reconstruct $S(t)$ up to permutation and scaling/multiplicative factors as these give the same result\n\t\\item As we assume the sources to be independent, we can write the joint probability distribution as:\n\t$$p(S_1,...,S_I)=\\prod_{i=1}^{I}p(S_i)$$\n\tOne crucial element of ICA is that these prior distributions need to be designed by the user. This requires pre-knowledge of how the source signals can look like (e.g. Gaussian, bounded Uniform, etc.). The performance of the algorithm depend on this design choice, and can lead to ICA failing if the prior has a very different distribution than points in the sources.\n\t\\item We will again use a maximum likelihood  approach where we try to increase the probability of the observed data, which can be derived as:\n\t$$\\ln p(\\bm{x}|\\bm{A})=\\ln |\\det \\bm{A}| + \\frac{1}{N}\\sum_{n=1}^{N}\\sum_{i=1}^{I}\\ln p_i\\left(\\sum_{j=1}^{I} \\left(A^{-1}\\right)_{ij} x^{(n)}_j\\right)$$\n\tFor simplicity, we replace $\\bm{A}^{-1}=\\bm{W}$, and aim to learn $\\bm{W}$ which is slightly easier.\n\t\\item We take now the derivative with respect to $\\bm{W}$, and end up with the following expression:\n\t$$\\bm{W}^{t+1}=\\bm{W}^{t} + \\alpha \\cdot \\frac{1}{N}\\sum_{n=1}^{N}\\left(\\nabla_{\\bm{S}} \\log p(\\bm{S})\\Big\\vert_{S=S_n}\\bm{S}_n^T+\\bm{I}\\right)\\bm{W}$$\n\twhere we estimate $\\bm{S}=\\bm{W}\\bm{X}$. In addition, we see here that what we actually need from our prior is the derivative of its log. Hence, the prior is mostly designed to have a simple form of $\\Phi_i=\\frac{\\partial \\ln p_i(a_i)}{\\partial a_i}$.\n\t\\item We can slightly simplify the gradient calculation by splitting it into multiple parts. Summarizing the full algorithm, we get:\n\t\\begin{tcolorbox}[colback=white!85!gray,colframe=gray!75!black,title=Independent Component Analysis]\n\t\t\\begin{algorithm}[H]\n\t\t\t\\SetAlgoLined\n\t\t\tChoose prior and calculate log derivative $\\Phi_i=\\frac{\\partial \\ln p_i(a_i)}{\\partial a_i}$\\;\n\t\t\tSet learning rate $\\eta$\\;\n\t\t\tInitialize $\\bm{W}=\\bm{A}^{-1}$\\;\n\t\t\t\\While{$\\nabla \\bm{W}^{(t)} > \\epsilon$}{\n\t\t\t\tLet $\\hat{\\bm{S}}=\\bm{W}\\bm{X}$ be the current estimate of $\\bm{S}$\\;\n\t\t\t\tLet $\\bm{Z}_i=\\Phi_i(\\hat{\\bm{S}}_i)$\\;\n\t\t\t\tLet $\\bm{X}' = \\bm{W}^T\\hat{\\bm{S}}$\\;\n\t\t\t\tCalculate the gradients $\\nabla \\bm{W}^{(t)}= \\bm{W}^{(t)} + \\frac{1}{N}\\left[ \\bm{Z}{\\bm{X}'}^T\\right]$\\;\n\t\t\t\tApply gradient with learning rate $\\bm{W}^{(t+1)}=\\bm{W}^{(t)}+\\eta \\nabla \\bm{W}^{(t)}$\\;\n\t\t\t}\n\t\t\tReconstruct signals $\\bm{S}_n=\\bm{W}\\bm{X}_n$\\;\n\t\t\\end{algorithm}\n\t\\end{tcolorbox} \n\t\\item One issue with ICA is that the signals are not allowed to be Gaussian. If this would be the case, we can not reconstruct the signal up to rotation as Gaussians are rotation invariant. Hence, the signals will be messed up although we find an optimum\n\\end{itemize}\n\\subsection{Information theory}\n\\begin{itemize}\n\t\\item The information of an event $A$ can be measured by:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\th(A) & = -\\log_2 p(A)\\hspace{4mm}\\text{(in bits)}\\\\\n\t\t\t& = - \\ln p(A)\\hspace{4mm}\\text{(in nats)}\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item An important measurement of a distribution in information theory is the Shannon entropy, which can be interpreted as the expected information of an event according to the distribution $p$:\n\t\\begin{equation*}\n\t\t\\tcbox[nobeforeafter]{\\(\n\t\t\tH(X) = -\\sum_{x\\in D_x} p(x)\\log_2 p(x)\n\t\t\\)}\n\t\\end{equation*}\n\tIn case we have $N$ independent events, the entropy is the sum of the single entropy of each of the $N$ events.\n\t\\item The entropy can also be defined for continuous space. It is then referred to as the differential entropy:\n\t$$H(\\bm{x})=-\\int p(\\bm{x})\\log_2 p(\\bm{x})d\\bm{x}$$\n\t\\item We can also define conditional entropy, which is as follows:\n\t\\begin{equation*}\n\t\\tcbox[nobeforeafter]{\\(\n\t\tH(\\bm{y}|\\bm{x}) = -\\int p(\\bm{x})\\left[\\int p(\\bm{y}|\\bm{x})\\ln p(\\bm{y}|\\bm{x})d\\bm{y}\\right]d\\bm{x}\n\t\t\\)}\n\t\\end{equation*}\n\twith the property $H(\\bm{x},\\bm{y})=H(\\bm{x})+H(\\bm{y}|\\bm{x})=H(\\bm{y})+H(\\bm{x}|\\bm{y})$\n\t\\item Another well-known measurement is the Kullback-Leiber divergence (also referred to as relative entropy):\n\t\\begin{equation*}\n\t\t\\tcbox[nobeforeafter]{\\(\n\t\t\t\\text{KL}(p(\\bm{x})||q(\\bm{x})) = -\\int p(\\bm{x})\\ln \\frac{q(\\bm{x})}{p(\\bm{x})}d\\bm{x}\n\t\t\\)}\n\t\\end{equation*}\n\tSome properties of this divergence are:\n\t\\begin{itemize}\n\t\t\\item Always positive: $\\text{KL}(p||q)\\geq 0$\n\t\t\\item If $\\text{KL}(p||q) = 0$, then $p=q$ (if $p$,$q$ are sufficient regular, i.e. strictly positive and integral defined)\n\t\t\\item The triangular inequality does not hold for KL, thus it is not a distance measure: \n\t\t\n\t\t$\\text{KL}(p||q)+\\text{KL}(q||r)\\not\\geq \\text{KL}(p||r)$\n\t\\end{itemize}\n\t\\item Mutual information describes the amount of information that is shared among $x$ and $y$:\n\t\\begin{equation*}\n\t\t\\tcbox[nobeforeafter]{\\(\n\t\t\tI(\\bm{x};\\bm{y}) = \\text{KL}(p(\\bm{x},\\bm{y})||p(\\bm{x}),p(\\bm{y})) = H(\\bm{x})-H(\\bm{x}|\\bm{y}) = H(\\bm{y}) - H(\\bm{y}|\\bm{x})\n\t\t\\)}\n\t\\end{equation*}\n\tIn other words, how much information about $y$ do I get by observing $x$. In a diagram, mutual information can be visualized as follows:\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.3\\textwidth]{figures/information_theory_mutual_information.png}\n\t\t\\caption{Visualizing the relationship between mutual information and entropy.}\n\t\\end{figure}\n\\end{itemize}\n"
  },
  {
    "path": "Machine_Learning_2/ml2_graphical_models.tex",
    "content": "\\section{Probabilistic graphical models}\n\\begin{itemize}\n\t\\item It is often beneficial to visualize a probabilistic model as a diagram, which we call \\textit{(probabilistic) graphical models}. \n\t\\item They are good for:\n\t\\begin{itemize}\n\t\t\\item causal reasoning/modeling\n\t\t\\item calculating inference and conditional distributions efficiently \n\t\t\\item Designing and communicating statistical model\n\t\t\\item Encoding (conditional) independence relations\n\t\\end{itemize} \n\t\\item Note that there are often multiple ways to express the same probability distribution. For example, take a joint distribution $p(A,B,C)$, which we can either write as $p(A,B,C)=p(A)p(B|A)p(C|A,B)$ (see Figure~\\ref{fig:graphical_models_example_1}) or $p(A,B,C)=p(C)p(A|C)p(B|A,C)$ (see Figure~\\ref{fig:graphical_models_example_2}). Nevertheless, what we are interested in is the graphical representation with the least number of edges, as e.g. if $A$ and $B$ are independent (conditionally on $C$), we can drop the edge between those.\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\begin{subfigure}{0.4\\textwidth}\n\t\t\t\\centering\n\t\t\t\\tikz{ %\n\t\t\t\t\\node[latent] (A) {$A$} ; %\n\t\t\t\t\\node[latent, right=of A] (B) {$B$} ; %\n\t\t\t\t\\node[latent, below=of B] (C) {$C$} ; %\n\t\t\t\t\n\t\t\t\t\\edge{A}{B};\n\t\t\t\t\\edge{A}{C};\n\t\t\t\t\\edge{B}{C};\n\t\t\t}\n\t\t\t\\caption{$p(A,B,C)=p(A)p(B|A)p(C|A,B)$}\n\t\t\t\\label{fig:graphical_models_example_1}\n\t\t\\end{subfigure}\n\t\t\\hspace{10mm}\n\t\t\\begin{subfigure}{0.4\\textwidth}\n\t\t\t\\centering\n\t\t\t\\tikz{ %\n\t\t\t\t\\node[latent] (A) {$A$} ; %\n\t\t\t\t\\node[latent, right=of A] (B) {$B$} ; %\n\t\t\t\t\\node[latent, below=of B] (C) {$C$} ; %\n\t\t\t\t\n\t\t\t\t\\edge{A}{B};\n\t\t\t\t\\edge{C}{A};\n\t\t\t\t\\edge{C}{B};\n\t\t\t}\n\t\t\t\\caption{$p(A,B,C)=p(C)p(A|C)p(B|A,C)$}\n\t\t\t\\label{fig:graphical_models_example_2}\n\t\t\\end{subfigure}\n\t\t\\caption{Two different graphical models (here Bayesian Networks) for the same joint distribution $p(A,B,C)$.}\n\t\\end{figure}\n\t\\item We distinguish between directed acyclic graphs, which we call \\textit{Bayesian networks} (BN), and undirected graphs, which are \\textit{Markov Random Fields} (MRF)\n\\end{itemize}\n\n\\subsection{Bayesian Networks}\n\\begin{itemize}\n\t\\item There is a simple way for creating a Bayesian network for a given statistical model.\n\t\\begin{enumerate}\n\t\t\\item Determine the ordering of the variables (``\\textit{topological ordering}'')\n\t\t\\item In this ordering, call the parents of the random variable $X_i$: $\\text{pa}_i$, or $\\text{pa}(X_i)$ which is a subset of variables with lower ordering: $\\text{pa}_i \\subseteq \\left\\{1,...,i-1\\right\\}$. The joint probability distribution can be written as:\n\t\t$$p(X_1,...,X_M) = \\prod_{i=1}^{M}p(X_i|X_{\\text{pa}_i})$$\n\t\t\\item In the graphical model, draw an edge from $X_j$ to $X_i$ if $j\\in \\text{pa}_i$\n\t\\end{enumerate}\n\t\\item \\underline{Example}: (first-order) Markov Chain\n\t\\begin{itemize}\n\t\t\\item The joint probability distribution of a Markov Chain can be expressed by:\n\t\t$$p(X_1,...,X_M)=p(X_1)\\cdot \\prod_{i=2}^{M} p(X_i|X_{i-1})$$\n\t\t\\item The corresponding Bayesian Network looks as follows:\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\tikz{ %\n\t\t\t\t\\node[obs] (x1) {$X_1$} ; %\n\t\t\t\t\\node[obs, right=of x1] (x2) {$X_2$} ; %\n\t\t\t\t\\node[obs, right=of x2] (x3) {$X_3$} ; %\n\t\t\t\t\\node[const, right=of x3] (xetc1) { \\hspace{2mm}...\\hspace{2mm} } ; %\n\t\t\t\t\\node[obs, right=of xetc1] (xM) {$X_M$} ; %\n\t\t\t\t\n\t\t\t\t\\edge{x1}{x2};\n\t\t\t\t\\edge{x2}{x3};\n\t\t\t\t\\edge{x3}{xetc1};\n\t\t\t\t\\edge{xetc1}{xM};\n\t\t\t}\n\t\t\\end{figure}\n\t\n\t\twhere the filling expresses that $X_i$ is an observed variable.\n\t\\end{itemize}\n\t\\item \\underline{Example}: Regression\n\t\\begin{itemize}\n\t\t\\item Suppose we have a simple regression problem where we want to learn parameters $W$ to predict targets $T$ from input $X$. We further assume that we know our sensory noise $\\sigma^2$, and have a prior with hyperparameters $\\alpha$.\n\t\t\\item \n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\tikz{ %\n\t\t\t\t\\node[obs] (x) {$X_n$} ; %\n\t\t\t\t\\node[obs, right=of x] (t) {$T_n$} ; %\n\t\t\t\t\\plate{xt}{(x)(t)}{$n=1,...,N$};\n\t\t\t\t\n\t\t\t\t\\node[latent, above=of t] (w) {$W$} ; %\n\t\t\t\t\\node[const, left=of w] (alpha) {$\\alpha$} ; %\n\t\t\t\t\\node[const, right=of t] (sigma) {$\\sigma^2$} ; %\n\t\t\t\t\n\t\t\t\t\\edge{x}{t};\n\t\t\t\t\\edge{w}{t};\n\t\t\t\t\\edge{alpha}{w};\n\t\t\t\t\\edge{sigma}{t};\n\t\t\t}\n\t\t\\end{figure}\n\t\n\t\tWe can express this in the graphical model above, which represents the probability distribution\n\t\t$$p(W, \\left\\{T_n\\right\\}, \\left\\{X_n\\right\\}|\\alpha, \\sigma^2)p(W|\\alpha)\\prod_{n=1}^{N} \\left[p(T_n|X_n, W, \\sigma^2)p(X_n)\\right]$$\n\t\tNote that in the graphical model, $\\alpha$ and $\\sigma^2$ are assumed to be fixed and known, and the ``plate'' can be interpreted as copying the content $N$ times (i.e. we have $N$ $X_i$ and $T_i$ variables with the same edges).\n\t\t\n\t\tAlso, if desired, we could have used a constant for the data points $X_i$ as well as these are often assumed to be fixed.\n\t\t\\item If we also want to express the predictive distribution $p(T^{*}|X^{*},W, \\left\\{T_n\\right\\}, \\left\\{X_n\\right\\},\\alpha, \\sigma^2)$, we can extend our model as follows:\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\tikz{ %\n\t\t\t\t\\node[obs] (x) {$X_n$} ; %\n\t\t\t\t\\node[obs, right=of x] (t) {$T_n$} ; %\n\t\t\t\t\\plate{xt}{(x)(t)}{$n=1,...,N$};\n\t\t\t\t\n\t\t\t\t\\node[latent, above=of t] (w) {$W$} ; %\n\t\t\t\t\\node[const, left=of w] (alpha) {$\\alpha$} ; %\n\t\t\t\t\\node[const, right=of t] (sigma) {$\\sigma^2$} ; %\n\t\t\t\t\n\t\t\t\t\\node[latent, right=of w] (tstar) {$T^{*}$} ; %\n\t\t\t\t\\node[obs, right=of tstar] (xstar) {$X^{*}$} ; %\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\\edge{x}{t};\n\t\t\t\t\\edge{w}{t};\n\t\t\t\t\\edge{w}{tstar};\n\t\t\t\t\\edge{xstar}{tstar};\n\t\t\t\t\\edge{alpha}{w};\n\t\t\t\t\\edge{sigma}{t};\n\t\t\t\t\\edge{sigma}{tstar};\n\t\t\t}\n\t\t\\end{figure}\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Conditional independence and D-separation}\n\\begin{itemize}\n\t\\item A useful property of graphical models is that we can easily study the independence relations between random variables in our model. \n\t\\item We call $X$ and $Y$ being independent iff $p(X,Y)=p(X)p(Y)$. The notation for this is $X\\independent Y$\n\t\\item $X$ is \\textit{conditionally} independent of $Y$ given $Z$ if $p(X,Y|Z)=p(X|Z)p(Y|Z)$. The notation for this is $X\\independent Y|Z$. Note that if $X$ and $Y$ are generally independent, we can also write $X\\independent Y|\\emptyset$\n\t\\item For proving/testing conditional independence, we can use \\textbf{d-separation}. Supposed $A$, $B$, $C$ are sets of variables. If $A$ is d-separated from $B$ given $C$, then $p(X_A,X_B|X_C)=p(X_A|X_C)p(X_B|X_C)$, which we can also write as $A\\perp B|C\\implies X_A\\independent X_B|X_C$ \n\t\\item Note that the other way round, $X_A\\independent X_B|X_C\\not\\Rightarrow A\\perp B|C $ is not always valid (but mostly) as we will show in a later example. Hence, if $A$ and $B$ are not d-separated, it does not necessarily mean that $X_A$ and $X_B$ are not conditional independent.\n\t\n\t\\item The algorithm can be summarized as follows:\n\t\\begin{tcolorbox}[colback=white!85!gray,colframe=gray!75!black,title=D-separation]\n\t\tGiven the sets of variables $A$, $B$, $C$:\n\t\t\\begin{enumerate}\n\t\t\t\\item Consider all paths (sequence of nodes, connected by edges,  s.t. no node repeats) between any node in $A$ and any node in $B$\n\t\t\t\\item Mark a path as \\underline{blocked} by $C$ if\n\t\t\t\\begin{enumerate}\n\t\t\t\t\\item It contains a collider $...\\rightarrow u \\leftarrow ...$ such that $u$ is not an ancestor of a node in $C$\n\t\t\t\t\\item It contains a non-collider $...\\rightarrow u$, $...\\rightarrow u \\rightarrow ...$, $...\\leftarrow u \\rightarrow ...$ such that $u$ is in $C$\n\t\t\t\\end{enumerate}\n\t\t\t\\item If all paths are marked as blocked by $C$, then $A$ is d-separated from $B$ given $C$\n\t\t\\end{enumerate}\n\t\\end{tcolorbox}\t\n\t\\item \\underline{Examples}: \n\t\\begin{itemize}\n\t\t\\item Consider the following graphical model:\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\tikz{ %\n\t\t\t\t\\node[latent] (xA) {$X_A$} ; %\n\t\t\t\t\\node[latent, right=of xA] (xB) {$X_B$} ; %\n\t\t\t\t\\node[latent, below=of xB] (xC) {$X_C$} ; %\n\t\t\t\t\n\t\t\t\t\\edge{xC}{xA};\n\t\t\t\t\\edge{xC}{xB};\n\t\t\t}\n\t\t\\end{figure}\n\t\n\t\t$A$ is d-separated from $B$ given $C$ as the only way from $B$ to $A$ is through $X_C$, and it represents a non-collider: $X_A\\independent X_B|X_C$.\n\t\t\n\t\tNote that $A$ is not d-separated from $B$ given $\\emptyset$ because $X_C$ is then neither a non-collider nor a collider.\n\t\t\\item Consider the following graphical model:\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\tikz{ %\n\t\t\t\t\\node[latent] (xA) {$X_A$} ; %\n\t\t\t\t\\node[latent, right=of xA] (xC) {$X_C$} ; %\n\t\t\t\t\\node[latent, right=of xC] (xB) {$X_B$} ; %\n\t\t\t\t\n\t\t\t\t\\edge{xA}{xC};\n\t\t\t\t\\edge{xC}{xB};\n\t\t\t}\n\t\t\\end{figure}\n\t\n\t\tSimilarly to the previous model, $A$ is d-separated from $B$ given $C$ as the only way from $B$ to $A$ is through $X_C$, and it represents a non-collider: $X_A\\independent X_B|X_C$.\n\t\t\n\t\tHowever, here we can show a special case where conditional independence does not imply d-separation. Suppose that we model $p(C|A)=\\delta_{C,A}$, hence being a deterministic mapping. Now, $C\\independent B|A$ holds because if we know $A$, we know $C$ for certain. Nevertheless, the d-separation is not valid because there is a direct path from $C$ to $B$! \n\t\t\n\t\t\\item Consider the following graphical model:\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\tikz{ %\n\t\t\t\t\\node[latent] (xA) {$X_A$} ; %\n\t\t\t\t\\node[latent, right=of xA] (xC) {$X_C$} ; %\n\t\t\t\t\\node[latent, above=of xC] (xB) {$X_B$} ; %\n\t\t\t\t\\node[const, right=of xC] (xetc) {\\hspace{2mm}...\\hspace{2mm} } ; %\n\t\t\t\t\\node[latent, right=of xetc] (xD) {$X_D$} ; %\n\t\t\t\t\n\t\t\t\t\\edge{xA}{xC};\n\t\t\t\t\\edge{xB}{xC};\n\t\t\t\t\\edge{xC}{xetc};\n\t\t\t\t\\edge{xetc}{xD};\n\t\t\t}\n\t\t\\end{figure}\n\t\n\t\t$A$ is d-separated from $B$ given the empty set as $X_C$ represents a collider which is not in the empty set: $X_A\\independent X_B|\\emptyset$.\n\t\t\n\t\t$A$ is \\textit{not} d-separated from $B$ given $C$ because $X_C$ is then not a collider anymore: $A\\not\\perp B|C$.\n\t\t\n\t\t$A$ is \\textit{not} d-separated from $B$ given $D$ because $X_C$ is an ancestor of a node in $D$, and hence, not a collider: $A\\not\\perp B|D$.\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Markov blanket}\n\\begin{itemize}\n\t\\item A Markov blanket of a variable $X_i$ is defined as the set of variables which are the parents, children or children's parents of $X_i$, except $X_i$ itself:\n\t$$\\text{MB}(X_i)=\\text{pa}_i \\cup \\text{ch}_i \\cup \\left(\\text{pa}_{\\text{ch}_i}\\setminus i\\right)$$\n\t\\item The important property of the Markov blanket is that, for a random variable $X_i$ in any BN, given its Markov blanket $\\text{MB}(X_i)$, it is conditionally independent of the rest of the graph:\n\t$$p\\left(X_i|X_{\\text{MB}(X_i)}, X_{\\text{res}}\\right) = p\\left(X_i|X_{\\text{MB}(X_i)}\\right)$$\n\t\\item \\underline{Example}: For the graphical model of the regression problem, the Markov blanket of $T^{*}$ is $\\text{MB}(T^{*})=\\left\\{X^{*}, W\\right\\}$. This result is intuitive as once we have trained our model, we do not need to revisit our data or our prior over $W$. Note that $\\sigma^2$ is a constant, and hence not in the Markov blanket.\n\\end{itemize}\n\\subsection{Markov Random Fields}\n\\begin{itemize}\n\t\\item A Markov Random Field is a undirected graphical models. Hence, our model consists now of two parts: the undirected graph $G$, and so called (maximum) cliques potentials $\\left\\{\\psi_A\\right\\}$\n\t\\item A clique in a undirected graph $G$ is a fully connected subset of nodes. Hence, also single nodes are considered as a clique.\n\t\\begin{itemize}\n\t\t\\item A clique is \\textit{maximal} if there is no clique that strictly contains it, i.e. we cannot add another node to the clique which is fully connected to all others. \n\t\t\\item \\underline{Example}: Consider the following graphical model:\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\tikz{ %\n\t\t\t\t\\node[latent] (xA) {$X_A$} ; %\n\t\t\t\t\\node[latent, right=of xA] (xB) {$X_B$} ; %\n\t\t\t\t\\node[latent, below=of xA] (xC) {$X_C$} ; %\n\t\t\t\t\\node[latent, right=of xC] (xD) {$X_D$} ; %\n\t\t\t\t\\node[latent, right=of xD] (xE) {$X_E$} ; %\n\t\t\t\t\\node[latent, left=of xA] (xF) {$X_F$} ; %\n\t\t\t\t\n\t\t\t\t\\edge[-]{xA}{xC};\n\t\t\t\t\\edge[-]{xA}{xB};\n\t\t\t\t\\edge[-]{xC}{xD};\n\t\t\t\t\\edge[-]{xB}{xD};\n\t\t\t\t\\edge[-]{xB}{xE};\n\t\t\t\t\\edge[-]{xE}{xD};\n\t\t\t}\n\t\t\\end{figure}\n\t\n\t\tThen our maximum cliques are $\\{X_A,X_C\\}, \\{X_A, X_B\\}, \\{X_C,X_D\\},\\{X_B,X_D,X_E\\},\\{X_F\\}$\n\t\\end{itemize}\n\t\\item We can now write our joint probability distribution in terms of the maximum cliques $\\left\\{\\psi_A\\right\\}$:$$p(x_1,...,x_N)=\\frac{1}{Z}\\prod_A \\psi_A(x_A)$$\n\tNote that we now need a normalization constant $Z$ which we did not need for Bayesian networks. The reason for this is that clique potentials might not be normalized. The only requirement for them is to be positive for any $x_A$, and are thus often modeled by a energy function $\\psi_A(x_A) = \\exp(f(x_A))$ (hence the name \\textit{potential})\n\t\\item For the previous example, our probability distribution can be now written as:\n\t$$p(x_A,...,x_E)=\\frac{1}{Z}\\psi_{A,B}(x_A,x_B)\\psi_{A,C}(x_A,x_C)\\psi_{C,D}(x_C,x_D)\\psi_{B,D,E}(x_B,x_D,x_E)\\psi_{F}(x_F)$$\n\twhere $Z=\\sum_{x_A}\\sum_{x_B}...\\sum_{x_F}\\psi_{A,B}(x_A,x_B)\\psi_{A,C}(x_A,x_C)...\\psi_{F}(x_F)$\n\t\\item One disadvantage of undirected graphs, as we can see here, is that we need to calculate $Z$ which grows exponentially with the number of variables.\n\t\\item We can also define the properties of separation and Markov blanket for undirected graphs:\n\t\\begin{description}\n\t\t\\item[Separation] similarly to d-separation in BNs, two subsets of nodes $A$ and $B$ are \\underline{separated} given $C$ if each path between a node in $A$ and a node in $B$ passes through (at least one) node $C$:\n\t\t$$A\\perp B|C\\implies X_A\\independent X_B|X_C$$\n\t\t\\item[Markov blanket] The Markov blanket for MRFs is defined as the neighbors of $i$, i.e. the nodes adjacent to $i$. In the previous example, the Markov blanket of $X_B$ is: $\\text{MB}(X_B) = \\{X_A,X_D,X_E\\}$\n\t\\end{description}\n\\end{itemize}\n\\subsubsection{Converting Bayesian network to MRFs}\n\\begin{itemize}\n\t\\item Sometimes it is the case that we want to represent a same statistical model which we have as a Bayesian network, also as a MRF. This is the case when we want to apply algorithms which are generally defined for undirected graphs (e.g. sum-product)\n\t\\item The \\textit{Hammersley-Clifford} theorem states that any strictly positive, joint distribution $p(\\bm{X})\\ge 0$ can be represented as a MRF. Hence, we can also do it with any Bayesian network\n\t\\item Nevertheless, note that by converting a BN to a MRF, some properties/information might be lost, such as (conditional) independence relations. \n\t\\item \\underline{Examples}:\n\t\\begin{itemize}\n\t\t\\item Consider a first-order Markov chain:\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\begin{subfigure}{0.46\\textwidth}\n\t\t\t\t\\centering\n\t\t\t\t\\tikz{ %\n\t\t\t\t\t\\node[obs] (x1) {$x_1$} ; %\n\t\t\t\t\t\\node[obs, right=of x1] (x2) {$x_2$} ; %\n\t\t\t\t\t\\node[const, right=of x2] (xetc) {\\hspace{2mm}...\\hspace{2mm} } ; %\n\t\t\t\t\t\\node[obs, right=of xetc] (xM) {$x_M$} ; %\n\t\t\t\t\t\n\t\t\t\t\t\\edge{x1}{x2};\n\t\t\t\t\t\\edge{x2}{xetc};\n\t\t\t\t\t\\edge{xetc}{xM};\n\t\t\t\t}\n\t\t\t\t\\caption{BN}\n\t\t\t\\end{subfigure}\n\t\t\t\\hspace{5mm}\n\t\t\t\\begin{subfigure}{0.46\\textwidth}\n\t\t\t\t\\centering\n\t\t\t\t\\tikz{ %\n\t\t\t\t\t\\node[obs] (x1) {$x_1$} ; %\n\t\t\t\t\t\\node[obs, right=of x1] (x2) {$x_2$} ; %\n\t\t\t\t\t\\node[const, right=of x2] (xetc) {\\hspace{2mm}...\\hspace{2mm} } ; %\n\t\t\t\t\t\\node[obs, right=of xetc] (xM) {$x_M$} ; %\n\t\t\t\t\t\n\t\t\t\t\t\\edge[-]{x1}{x2};\n\t\t\t\t\t\\edge[-]{x2}{xetc};\n\t\t\t\t\t\\edge[-]{xetc}{xM};\n\t\t\t\t}\n\t\t\t\t\\caption{MRF}\n\t\t\t\\end{subfigure}\n\t\t\\end{figure}\n\t\n\t\tAs Bayesian network, we can represent it with the probability density function $p(x_1)\\prod_{i=2}^{M}p(x_i|x_{i-1})$.\n\t\t\n\t\tIn the case of the MRF, we have $\\frac{1}{Z}\\prod_{i=2}^{M}\\psi_{i-1,i}(x_{i-1},x_i)$. Note that the prior $\\psi_1(x_1)$ is integrated in $\\psi_{1,2}(x_1,x_2)$ as the clique potentials are more flexible than the conditional probabilities in Bayesian networks.\n\t\t\n\t\t\\item Consider the following Bayesian network:\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\begin{subfigure}{0.25\\textwidth}\n\t\t\t\t\\centering\n\t\t\t\t\\tikz{ %\n\t\t\t\t\t\\node[latent] (xA) {$X_A$} ; %\n\t\t\t\t\t\\node[latent, right=of xA] (xC) {$X_C$} ; %\n\t\t\t\t\t\\node[latent, below=of xC] (xB) {$X_B$} ; %\n\t\t\t\t\t\\node[latent, below=of xA] (xD) {$X_D$} ; %\n\t\t\t\t\t\n\t\t\t\t\t\\edge{xA}{xC};\n\t\t\t\t\t\\edge{xB}{xC};\n\t\t\t\t\t\\edge{xD}{xA};\n\t\t\t\t}\n\t\t\t\t\\caption{BN}\n\t\t\t\\end{subfigure}\n\t\t\t\\hspace{5mm}\n\t\t\t\\begin{subfigure}{0.25\\textwidth}\n\t\t\t\t\\centering\n\t\t\t\t\\tikz{ %\n\t\t\t\t\t\\node[latent] (xA) {$X_A$} ; %\n\t\t\t\t\t\\node[latent, right=of xA] (xC) {$X_C$} ; %\n\t\t\t\t\t\\node[latent, below=of xC] (xB) {$X_B$} ; %\n\t\t\t\t\t\\node[latent, below=of xA] (xD) {$X_D$} ; %\n\t\t\t\t\t\n\t\t\t\t\t\\edge[-]{xA}{xC};\n\t\t\t\t\t\\edge[-]{xB}{xC};\n\t\t\t\t\t\\edge[-]{xD}{xA};\n\t\t\t\t}\n\t\t\t\t\\caption{(Potential) MRF}\n\t\t\t\t\\label{fig:graphical_models_BN_to_MRF_2}\n\t\t\t\\end{subfigure}\n\t\t\t\\hspace{5mm}\n\t\t\t\\begin{subfigure}{0.25\\textwidth}\n\t\t\t\t\\centering\n\t\t\t\t\\tikz{ %\n\t\t\t\t\t\\node[latent] (xA) {$X_A$} ; %\n\t\t\t\t\t\\node[latent, right=of xA] (xC) {$X_C$} ; %\n\t\t\t\t\t\\node[latent, below=of xC] (xB) {$X_B$} ; %\n\t\t\t\t\t\\node[latent, below=of xA] (xD) {$X_D$} ; %\n\t\t\t\t\t\n\t\t\t\t\t\\edge[-]{xA}{xC};\n\t\t\t\t\t\\edge[-]{xB}{xC};\n\t\t\t\t\t\\edge[-]{xA}{xB};\n\t\t\t\t\t\\edge[-]{xD}{xA};\n\t\t\t\t}\n\t\t\t\t\\caption{MRF via mortalization}\n\t\t\t\t\\label{fig:graphical_models_BN_to_MRF_3}\n\t\t\t\\end{subfigure}\n\t\t\t\\caption{Comparing different conversions from BN to MRF}\n\t\t\\end{figure}\n\t\n\t\tIn this BN, $X_A\\independent X_B$, and (typically) $X_A\\not\\independent X_B|X_C$. If we just replace the directed edges by undirected ones (see Figure~\\ref{fig:graphical_models_BN_to_MRF_2}), we loose the independence $X_A\\not\\independent X_B$. Furthermore, we would need to design the potentials in a way that is captures $p(X_C|X_A,X_B)$ correctly.\n\t\t\n\t\tThe easiest way is transforming BNs by \\textbf{mortalization} (see Murphy, chapter 20.3). For each node, we ``marry the parents'', i.e. adding an edge between those if not already existing. By that, we ensure that we express all maximum clique potentials by the conditional probabilities of the Bayesian network. For example, see the MRF in Figure~\\ref{fig:graphical_models_BN_to_MRF_3} which we got via mortalization. The clique potentials are now simply: $\\psi_{A,D}(X_A,X_D)=p(X_D)p(X_A|X_D)$, $\\psi_{A,B,C}(X_A,X_B,X_C)=p(X_C|X_A,X_B)p(X_B)$\n\t\t\n\t\t\\item There are also MRFs which cannot be fully modeled by a Bayesian network. Consider for example the following graphical model:\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\begin{subfigure}{0.25\\textwidth}\n\t\t\t\t\\centering\n\t\t\t\t\\tikz{ %\n\t\t\t\t\t\\node[latent] (xA) {$X_A$} ; %\n\t\t\t\t\t\\node[latent, right=of xA] (xC) {$X_C$} ; %\n\t\t\t\t\t\\node[latent, below=of xC] (xB) {$X_B$} ; %\n\t\t\t\t\t\\node[latent, below=of xA] (xD) {$X_D$} ; %\n\t\t\t\t\t\n\t\t\t\t\t\\edge[-]{xA}{xC};\n\t\t\t\t\t\\edge[-]{xB}{xC};\n\t\t\t\t\t\\edge[-]{xD}{xA};\n\t\t\t\t\t\\edge[-]{xD}{xB};\n\t\t\t\t}\n\t\t\t\t\\caption{MRF}\n\t\t\t\t\\label{fig:graphical_models_MRF_to_BN_MRF}\n\t\t\t\\end{subfigure}\n\t\t\t\\hspace{5mm}\n\t\t\t\\begin{subfigure}{0.25\\textwidth}\n\t\t\t\t\\centering\n\t\t\t\t\\tikz{ %\n\t\t\t\t\t\\node[latent] (xA) {$X_A$} ; %\n\t\t\t\t\t\\node[latent, right=of xA] (xC) {$X_C$} ; %\n\t\t\t\t\t\\node[latent, below=of xC] (xB) {$X_B$} ; %\n\t\t\t\t\t\\node[latent, below=of xA] (xD) {$X_D$} ; %\n\t\t\t\t\t\n\t\t\t\t\t\\edge{xA}{xC};\n\t\t\t\t\t\\edge{xB}{xC};\n\t\t\t\t\t\\edge{xD}{xA};\n\t\t\t\t\t\\edge{xD}{xB};\n\t\t\t\t}\n\t\t\t\t\\caption{(Potential) BN}\n\t\t\t\t\\label{fig:graphical_models_MRF_to_BN}\n\t\t\t\\end{subfigure}\n\t\t\\end{figure}\n\t\n\t\tThe MRF models the following independence relations: $C\\perp D|\\{A,B\\}$, $A\\perp B|\\{C,D\\}$. If we would now want to model the same in a BN, we get into trouble as for the model in Figure~\\ref{fig:graphical_models_MRF_to_BN}, as although $C\\perp D|\\{A,B\\}$, we have $A\\not\\perp B|\\{C,D\\}$ because $X_C$ is not a collider (is in set $\\{C,D\\}$) and also not non-collider.\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Factor graphs}\n\\begin{itemize}\n\t\\item The third form of graphical models are Factor graphs. The idea is to represent the connections between variables by their factors in a bipartite graph. Hence, we have two sets of nodes: variable nodes, and factor nodes.\n\t\\item Consider the statistical model $p(X_A,X_B,X_C)=p(X_A)p(X_B)p(X_C|X_A,X_B)$. The factor graph representation of this is:\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\tikz{ %\n\t\t\t\\node[latent] (xA) {$X_A$} ; %\n\t\t\t\\node[latent, right=of xA] (xB) {$X_B$} ; %\n\t\t\t\\node[latent, right=of xB] (xC) {$X_C$} ; %\n\t\t\t\n\t\t\t\\factor[above=of xA] {f1} {$p(A)$} {} {} ;\n\t\t\t\\factor[above=of xB] {f2} {$p(B)$} {} {} ;\n\t\t\t\\factor[above=of xC] {f3} {$p(C|A,B)$} {} {} ;\n\t\t\t\n\t\t\t\\factoredge[-]{xA}{f1}{} ;\t\n\t\t\t\\factoredge[-]{xB}{f2}{} ;\t\n\t\t\t\\factoredge[-]{xA}{f3}{} ;\t\n\t\t\t\\factoredge[-]{xB}{f3}{} ;\t\n\t\t\t\\factoredge[-]{xC}{f3}{} ;\t\n\t\t}\n\t\\end{figure}\n\t\\item Similarly, for Markov Random Fields such as in Figure~\\ref{fig:graphical_models_MRF_to_BN_MRF}, the factor graph representation is:\n\t\n\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\tikz{ %\n\t\t\t\\node[latent] (xA) {$X_A$} ; %\n\t\t\t\\node[latent, right=of xA] (xC) {$X_C$} ; %\n\t\t\t\\node[latent, right=of xC] (xB) {$X_B$} ; %\n\t\t\t\\node[latent, right=of xB] (xD) {$X_D$} ; %\n\t\t\t\n\t\t\t\\factor[above=of xA] {f1} {$\\psi_{A,C}$} {} {} ;\n\t\t\t\\factor[above=of xC] {f2} {$\\psi_{C,B}$} {} {} ;\n\t\t\t\\factor[above=of xB] {f3} {$\\psi_{B,D}$} {} {} ;\n\t\t\t\\factor[above=of xD] {f4} {$\\psi_{D,A}$} {} {} ;\n\t\t\t\\factor[right=of f4] {f0} {$\\frac{1}{Z}$} {} {} ;\n\t\t\t\n\t\t\t\\factoredge[-]{xA}{f1}{} ;\t\n\t\t\t\\factoredge[-]{xC}{f1}{} ;\t\n\t\t\t\\factoredge[-]{xC}{f2}{} ;\t\n\t\t\t\\factoredge[-]{xB}{f2}{} ;\t\n\t\t\t\\factoredge[-]{xD}{f3}{} ;\t\n\t\t\t\\factoredge[-]{xB}{f3}{} ;\t\n\t\t\t\\factoredge[-]{xA}{f4}{} ;\t\n\t\t\t\\factoredge[-]{xD}{f4}{} ;\t\n\t\t}\n\t\\end{figure}\n\t\n\twhere we could have merged $\\frac{1}{Z}$ into any other factor if wanted.\n\t\\item Note that in contrast ot MRFs, factor graph do not require to take the \\textit{maximum} cliques. Hence, for the same statistical model, there exist different factor graphs.\n\\end{itemize}\n\\subsection{Learning in graphical models}\n\\begin{itemize}\n\t\\item One of the applications of graphical models is to learn the conditional probabilities/potentials they model. We will first discuss the learning process for Bayesian networks, and afterwards do the same for MRFs\n\\end{itemize}\n\\subsubsection{Learning in Bayesian networks}\n\\begin{itemize}\n\t\\item Suppose that we replace all conditionals $p(x_i|\\text{pa}_i)$ by a learnable function $\\theta_i(x_i, \\text{pa}_i)$ (equal to $f_i$ with parameters $\\theta_i$) with the constraint of $\\sum_{x_i} \\theta_i(x_i, \\text{pa}_i) = 1$.\n\t\\item The likelihood of a dataset can then be written as:\n\t$$p(\\left\\{\\tilde{x}_{in}\\right\\}|\\bm{\\theta}) = \\prod_{i=1}^{d} \\prod_{n=1}^{N}\\theta_i(\\tilde{x}_{in}, \\tilde{x}_{\\text{pa}_i,n}) = \\prod_i \\prod_{n=1}^{N} \\prod_{x_i} \\prod_{x_{\\text{pa}_i}} \\theta_i({x}_{in}, x_{\\text{pa}_i,n})^{\\delta(x_i=\\tilde{x}_{in})\\delta(x_{\\text{pa}_i}=\\tilde{x}_{\\text{pa}_i,n})}$$\n\t\\item Our objective to optimize is the log likelihood with the Lagrange multipliers:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\mathcal{L}(\\bm{\\theta}, \\bm{X}, \\bm{\\lambda}) & = \\sum_{i=1}^{d} \\sum_{n=1}^{N} \\sum_{x_i} \\sum_{x_{\\text{pa}_i}} \\delta(x_i=\\tilde{x}_{in})\\cdot \\delta(x_{\\text{pa}_i}=\\tilde{x}_{\\text{pa}_i,n})\\cdot \\ln \\theta_i({x}_{in}, x_{\\text{pa}_i,n}) - \\sum_{i=1}^{d} \\sum_{x_{\\text{pa}_i}} \\lambda_{i,x_{\\text{pa}_i}}\\left(\\sum_{x_i} \\theta_i(x_i,x_{\\text{pa}_i}) - 1\\right)\\\\\n\t\t\t& = \\sum_{i=1}^{d} \\sum_{x_i} \\sum_{x_{\\text{pa}_i}} N(x_i, x_{\\text{pa}_i})\\cdot \\ln \\theta_i({x}_{i}, x_{\\text{pa}_i}) - \\sum_{i=1}^{d} \\sum_{x_{\\text{pa}_i}} \\lambda_{i,x_{\\text{pa}_i}}\\left(\\sum_{x_i} \\theta_i(x_i,x_{\\text{pa}_i}) - 1\\right)\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\twhere $N(x_i, x_{\\text{pa}_i})$ represent a counter of how often the combination of values for $x_i$ and $x_{\\text{pa}_i}$ co-occur\n\t\\item By taking the derivative, we get the following solution:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\frac{\\partial \\mathcal{L}(\\bm{\\theta}, \\bm{X}, \\bm{\\lambda})}{\\partial \\theta_i(x_i, x_{\\text{pa}_i})} & = \\frac{N(x_i, x_{\\text{pa}_i})}{\\theta_i(x_i, x_{\\text{pa}_i})} - \\lambda_{i,x_{\\text{pa}_i}} \\overset{!}{=} 0\\\\\n\t\t\t\\Leftrightarrow \\theta_i(x_i, x_{\\text{pa}_i}) & = \\frac{N(x_i, x_{\\text{pa}_i})}{\\lambda_{i,x_{\\text{pa}_i}}}\\\\[8pt]\n\t\t\t\\frac{\\partial \\mathcal{L}(\\bm{\\theta}, \\bm{X}, \\bm{\\lambda})}{\\partial \\lambda_{i,x_{\\text{pa}_i}}} & = \\sum_{x_i} \\theta_i(x_i, x_{\\text{pa}_i}) - 1 \\overset{!}{=} 0\\\\\n\t\t\t\\Leftrightarrow \\sum_{x_i} \\frac{N(x_i, x_{\\text{pa}_i})}{\\lambda_{i,x_{\\text{pa}_i}}} & = 1\\\\\n\t\t\t\\Leftrightarrow \\lambda_{i,x_{\\text{pa}_i}} & = \\frac{1}{N(x_{\\text{pa}_i})}\\\\[8pt]\n\t\t\t\\implies \\theta_i(x_i, x_{\\text{pa}_i}) & = \\frac{N(x_i, x_{\\text{pa}_i})}{N(x_{\\text{pa}_i})}\n\t\t\\end{split}\n\t\\end{equation*}\n\tHence, the optimal conditionals $\\theta_i(x_i=a, x_{\\text{pa}_i}=b)$ are simply the average number of times we have seen $x_i=a$ when $x_{\\text{pa}_i}=b$.\n\t\\item For each conditional $\\theta_i$, the optimum solely depends on $x_i$ and its parents, and not over all variables as we had before (log-likelihood has the sum over all variables). This is why learning in Bayesian networks is fast and efficient\n\\end{itemize}\n\\subsubsection{Learning in Markov Random Fields}\n\\begin{itemize}\n\t\\item In the case of MRFs, we are interested in learning the potential $\\psi_A$ for all cliques. With a similar rewriting with the indicator function, we get the log-likelihood of the data as:\n\t$$\\mathcal{L}(\\bm{\\psi}, \\bm{X}) = \\sum_{n=1}^{N} \\sum_A \\sum_{x_A} \\delta(x_A=\\tilde{x}_A)\\ln \\psi_A(x_A) - N\\ln Z = \\sum_A \\sum_{x_A} N(x_A)\\ln \\psi_A(x_A) - N\\ln Z$$\n\t\\item The derivative of the log-likelihood is:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\frac{\\partial \\mathcal{L}(\\bm{\\psi}, \\bm{X})}{\\partial \\psi_A(x_A)} = \\frac{N(x_A)}{\\psi_A(x_A)} - \\frac{N}{\\psi_A(x_A)}\\E_{\\psi}[\\delta(x_A=\\cdot)] \\overset{!}{=} 0\n\t\t\\end{split}\n\t\\end{equation*}\n\twhere $\\E_{\\psi}[\\delta(x_A=\\cdot)]$ is the expected fraction of observations of $x_A$ over all potentials $\\bm{\\psi}$. In other words, how much probability mass is assigned to states where $x_A=\\tilde{x}_A$ (and other $x$ anything), compared to all other states. This terms comes from our normalization constant $Z$ which is of course influenced by our potentials.\n\t\\item The optimal is therefore found when the fraction of observed $x_A=\\tilde{x}_A$ is equal to the expected number of observations. To find a solution, we can use sampling to approximate the expectation:\n\t$$\\E_{\\psi}[\\delta(x_A=\\cdot)] \\approx \\frac{N_{\\psi}(x_A)}{N_{\\psi}}$$\n\twhere $N_{\\psi}$ is the sample size, and $N_{\\psi}(x_A)$ the number of times we observed $x_A=\\tilde{x}_A$ during sampling.\n\t\\item Hence, the (approximated) optimum is:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\frac{\\partial \\mathcal{L}(\\bm{\\psi}, \\bm{X})}{\\partial \\psi_A(x_A)} & = \\frac{N(x_A)}{\\psi_A(x_A)} - \\frac{N}{\\psi_A(x_A)}  \\frac{N_{\\psi}(x_A)}{N_{\\psi}}\\overset{!}{=} 0\\\\\n\t\t\t\\Leftrightarrow \\psi_A(x_A) & = \\frac{N(x_A)/N}{N_{\\psi}(x_A)/N_{\\psi}}\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\tTo interpret the result, if we sample less times $x_A$ than in our dataset, we increase $\\psi_A$. Otherwise, we reduce it. For stability, we can view this optimum as an update step, and repeat this procedure a couple of times until we converge \n\\end{itemize}\n\\subsection{Inference in graphical models}\n\\begin{itemize}\n\t\\item Another important aspect of graphical models is inference to be efficient. Here, our goal is to either marginalize out variables, or set some to observed, and calculate the posterior distribution of others\n\t\\item Let's first consider again a first-order Markov chain. Its joint probability distribution can be expressed by:\n\t$$p(x_1,...,x_d)=\\frac{1}{Z}\\psi_{1,2}(x_1,x_2)\\cdot \\psi_{2,3}(x_2,x_3)\\cdot ... \\cdot \\psi_{d-1,d}(x_{d-1},x_d)$$\n\t\\item Now suppose we want to calculate the marginal distribution $p(x_j)$. We can do this by marginalizing over all other variables:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tp(x_j) & = \\sum_{x_1}...\\sum_{x_{j-1}}\\sum_{x_{j+1}}...\\sum_{x_d} p(x_1,...,x_{j-1},x_{j+1},...,x_d)\\\\\n\t\t\t& = \\sum_{x_1}...\\sum_{x_{j-1}}\\sum_{x_{j+1}}...\\sum_{x_d} \\frac{1}{Z}\\psi_{1,2}(x_1,x_2)\\cdot \\psi_{2,3}(x_2,x_3)\\cdot ... \\cdot \\psi_{d-1,d}(x_{d-1},x_d)\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\tNow, we can move the sums as the potentials only contain two variables, and are hence independent of all others:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\tp(x_j) & = \\frac{1}{Z} \\sum_{x_1}\\sum_{x_2}...\\sum_{x_{j-1}}\\sum_{x_{j+1}}...\\sum_{x_d}  \\psi_{1,2}(x_1,x_2)\\cdot \\psi_{2,3}(x_2,x_3)\\cdot ... \\cdot \\psi_{d-1,d}(x_{d-1},x_d)\\\\\n\t\t& =  \\frac{1}{Z}\\sum_{x_1}\\sum_{x_2}...\\sum_{x_{j-1}} \\psi_{1,2}(x_1,x_2)\\cdot ...\\psi_{j-1,j}(x_{j-1},x_j) \\cdot \\\\&\\hspace{23mm}\\sum_{x_{j+1}} \\psi_{j,j+1}(x_{j},x_{j+1}) \\sum_{x_{j+2}} \\psi_{j+1,j+2}(x_{j+1},x_{j+2}) ... \\sum_{x_{d}} \\psi_{d-1,d}(x_{d-1},x_d)\\\\\n\t\t& = \\frac{1}{Z} \\underbrace{\\sum_{x_{j-1}} \\psi_{j-1,j}(x_{j-1},x_{j}) \\sum_{x_{j-2}} \\psi_{j-2,j-1}(x_{j-2},x_{j-1}) ... \\sum_{x_{1}} \\psi_{1,2}(x_{1},x_2)}_{\\mu_{\\alpha}(x_j)} \\cdot \\\\&\\hspace{8mm}\\underbrace{\\sum_{x_{j+1}} \\psi_{j,j+1}(x_{j},x_{j+1}) \\sum_{x_{j+2}} \\psi_{j+1,j+2}(x_{j+1},x_{j+2}) ... \\sum_{x_{d}} \\psi_{d-1,d}(x_{d-1},x_d)}_{\\mu_{\\beta}(x_j)}\\\\\n\t\t& = \\frac{1}{Z}\\mu_{\\alpha}(x_j)\\mu_{\\beta}(x_j)\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item This shows us that we can split the marginal into two separate parts, $\\mu_{\\alpha}$ and $\\mu_{\\beta}$, which both are a recursive functions:\n\t$$\\mu_{\\alpha}(x_j) = \\sum_{x_{j-1}} \\psi_{j-1,j}(x_{j-1},x_j) \\mu_{\\alpha}(x_{j-1})$$ \n\t$$\\mu_{\\alpha}(x_j) = \\sum_{x_{j+1}}\\psi_{j,j+1}(x_{j},x_{j+1}) \\mu_{\\beta}(x_{j+1})$$ \n\tWe can view these recursive functions also as \\textit{messages} which are passed across the chain. $\\mu_{\\alpha}(x_{j})$ and $\\mu_{\\beta}(x_{j})$ would be then the incoming messages of $x_j$, and $\\mu_{\\alpha}(x_{j+1})$ and $\\mu_{\\beta}(x_{j-1})$ the outgoing messages.\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/graphical_models_sum_product_message_passing.pdf}\n\t\t\\caption{Message passing in graphical models (Bishop 8.38)}\n\t\\end{figure}\n\t\\item Even the normalization term $Z$ can be expressed by the messages: $Z=\\sum_{x_n}\\mu_{\\alpha}(x_n)\\mu_{\\beta}(x_n)$\n\t\\item The benefit of this recursive implementation is that we don't have to take a sum over $d$ variables ($\\mathcal{O}(K^{d})$), but only have to take sums over two variables at a time ($\\mathcal{O}(K^{2}d)$). Hence, the computational time scales linear with number of nodes instead exponential.\n\t\n\tFurthermore, we can share calculations between nodes, as the same messages can be re-used. Thus, calculating the marginals for all variables reduces to $\\mathcal{O}(2\\cdot K^2{d})=\\mathcal{O}(K^2{d})$\n\\end{itemize}\n\\subsubsection{Sum-product algorithm}\n\\begin{itemize}\n\t\\item The message passing idea is not limited to a Markov chain, but can be applied to any graph. For simplicity, we focus here on \\textit{trees}, i.e. graphs where all nodes are connected, but without any loops/cycles, independent of the direction of the edges.\n\t\\item In our discussion, we will focus on factor graphs as they represent the most general form of graphical models. Furthermore, cycles in MRFs can often be resolved in a factor graph so that we have less problem getting a tree-structured graph\n\t\\item Now we will send messages from variables to factors, and from factors to variables. As a result, we get the marginalization for all variables. This algorithm is called \\textbf{sum-product} algorithm as we only take sums and products\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.2\\textwidth]{figures/graphical_models_sum_product_messages_factor.png}\n\t\t\\caption{Message passing to and from a factor node (Bishop 8.47)}\n\t\\end{figure}\n\t\\item The messages can be calculated as follows where we start at leaf nodes, and recursively go to the center of the tree:\n\t\\begin{equation*}\n\t\\tcbox[nobeforeafter]{\\(\n\t\t\\begin{split}\n\t\t\t\\textbf{Factor$\\to$Variable:} & \\hspace{2mm} \\mu_{\\alpha\\to i}(x_i)=\\sum_{\\bm{x}_{\\alpha\\setminus i}} f_{\\alpha}(\\bm{x}_{\\alpha})\\prod_{j\\in \\alpha\\setminus i}\\mu_{j\\to\\alpha}(x_j)\\\\\n\t\t\t& \\text{If $\\alpha$ leaf node:} \\hspace{2mm}\\mu_{\\alpha\\to i}(x_i)=\\sum_{\\bm{x}_{\\alpha}} f_{\\alpha}(\\bm{x}_{\\alpha})\\\\\n\t\t\t\\textbf{Variable$\\to$Factor:} & \\hspace{2mm} \\mu_{j\\to \\alpha}(x_j)=\\prod_{\\beta\\in \\text{ne}(j)\\setminus \\alpha}\\mu_{\\beta\\to j}(x_j)\\\\\n\t\t\t&\\text{If $j$ leaf node:} \\hspace{2mm}\\mu_{j\\to \\alpha}(x_j)=1\n\t\t\\end{split}\n\t\t\\)}\n\t\\end{equation*}\n\tThe marginalizations/beliefs are in the end:\n\t\\begin{equation*}\n\t\\tcbox[nobeforeafter]{\\(\n\t\t\\begin{split}\n\t\t\\textbf{Variable belief:} & \\hspace{2mm} p(x_i)=\\frac{1}{Z}\\prod_{\\alpha\\in\\text{ne}(i)}\\mu_{\\alpha\\to i}(x_i)\\hspace{5mm}\\text{where}\\hspace{2mm}Z=\\sum_{x_i}\\prod_{\\alpha\\in\\text{ne}(i)}\\mu_{\\alpha\\to i}(x_i)\\\\\n\t\t\\textbf{Factor belief:} & \\hspace{2mm} p(\\bm{x}_{\\alpha})=\\frac{1}{Z}f_{\\alpha}(\\bm{x}_\\alpha) \\prod_{i \\in\\text{ne}(\\alpha)}\\mu_{i \\to\\alpha}(x_i)\\\\\n\t\t\\end{split}\n\t\t\\)}\n\t\\end{equation*}\n\twhere a factor belief is the marginalization of all variables except those with an direct edge to the factor.\n\t\\item The complexity of this algorithm scales with $\\mathcal{O}(EK^{M})$ where $E$ are the number of edges, $M$ the maximum number of variables that are connected to a factor, and $K$ the maximum domain size\n\t\\item Note that this algorithm is only exact on trees or forest (group of trees). For general graphs, we can first bring them into the shape of a tree by e.g. \\textbf{variable elimination}\n\t\\begin{itemize}\n\t\t\\item Given a MRF or factor graph, we will marginalize out these variable nodes which cause a loop in our graphical model. Let's for example consider this model:\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\tikz{ %\n\t\t\t\t\\node[latent] (xA) {$X_A$} ; %\n\t\t\t\t\\node[latent, right=of xA] (xB) {$X_B$} ; %\n\t\t\t\t\\node[latent, below=of xA] (xC) {$X_C$} ; %\n\t\t\t\t\\node[latent, right=of xC] (xD) {$X_D$} ; %\n\t\t\t\t\\node[latent, right=of xD] (xE) {$X_E$} ; %\n\t\t\t\t\n\t\t\t\t\\edge[-]{xA}{xB};\n\t\t\t\t\\edge[-]{xB}{xD};\n\t\t\t\t\\edge[-]{xC}{xD};\n\t\t\t\t\\edge[-]{xE}{xD};\n\t\t\t\t\\edge[-]{xE}{xB};\n\t\t\t}\n\t\t\\end{figure}\n\t\t\\item We can do this by simply writing down the joint probability distribution and determine a order of variables $X_1,...,X_M$ where the last variables, e.g. $X_M$, should be those which are eliminated (likely the easiest to marginalize). We then sort the sums according to the selected order.\n\t\t\n\t\tIn our example, we want to eliminate $X_E$ as it has the least connections from those in the loop. The sum order is therefore:\n\t\t\\begin{equation*}\n\t\t\t\\hspace{-10mm}p(X_A,...,X_E) = \\frac{1}{Z}\\sum_{X_B}\\sum_{X_A} \\psi_{A,B}(X_A, X_B)\\sum_{X_D}\\psi_{B,D}(X_B, X_D) \\sum_{X_C}\\psi_{C,D}(X_C, X_D)\\sum_{X_E} \\psi_{B,E}(X_B, X_E)\\psi_{D,E}(X_D, X_E)\n\t\t\\end{equation*}\n\t\t\\item Finally, replace the marginalized terms by a new factor, and remove node from graph. In the example, we would replace $\\tau(X_B, X_D) = \\sum_{X_E} \\psi_{B,E}(X_B, X_E)\\psi_{D,E}(X_D, X_E)$. This can be merged into the potential $\\psi_{B,D}$, hence not changing the graph structure. Our final graph is a tree again, and looks as follows:\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\tikz{ %\n\t\t\t\t\\node[latent] (xA) {$X_A$} ; %\n\t\t\t\t\\node[latent, right=of xA] (xB) {$X_B$} ; %\n\t\t\t\t\\node[latent, below=of xA] (xC) {$X_C$} ; %\n\t\t\t\t\\node[latent, right=of xC] (xD) {$X_D$} ; %\n\t\t\t\t\n\t\t\t\t\\edge[-]{xA}{xB};\n\t\t\t\t\\edge[-]{xB}{xD};\n\t\t\t\t\\edge[-]{xC}{xD};\n\t\t\t}\n\t\t\\end{figure}\n\t\\end{itemize}\n\t\\item The sum-product algorithm so far only calculated marginals. To set some observed variables as observed, we can use the same algorithm, but simply add additional ``hard evidence'' factor nodes, which is nothing else than a hard prior that $x_j$ has the value $\\xi_j$:\n\t$$f_{\\xi_j}(x_j)=\\delta(x_j=\\xi_j)$$\n\tThen if we apply the sum product algorithm on the extended graph again, we get the conditionals $p(x_i|x_j=\\xi_j)$\n\\end{itemize}\n\\subsubsection{Max-sum algorithm}\n\\begin{itemize}\n\t\\item The sum-product algorithm calculates the full marginal distribution $p(x_j)$, but sometimes, we just want to know the most likely value of $x_j$, especially if we set some variables to observed states\n\t\\item As it turns out, we can use a very similar algorithm for this, but simply replace sums by maximum operators, and products by sums.\n\t\\item First, let's consider what the optimum is in the general case:\n\t$$\\bm{x}^{*}=\\arg\\max_{\\bm{x}} \\prod_{\\alpha} f_{\\alpha}(\\bm{x}_{\\alpha})$$\n\twhere we can ignore the normalization constant. Furthermore, to simplify optimization, we can apply the log:\n\t$$\\bm{x}^{*} = \\arg\\max_{\\bm{x}} \\sum_{\\alpha} \\ln f_{\\alpha}(\\bm{x}_{\\alpha})$$\n\t\\item Our messages passed across the graph are then as follows:\n\t\\begin{equation*}\n\t\\tcbox[nobeforeafter]{\\(\n\t\t\\begin{split}\n\t\t\\textbf{Factor$\\to$Variable:} & \\hspace{2mm} \\nu_{\\alpha\\to i}(x_i)=\\max_{\\bm{x}_{\\alpha\\setminus i}} \\log f_{\\alpha}(\\bm{x}_{\\alpha}) + \\sum_{j\\in \\alpha\\setminus i}\\nu_{j\\to\\alpha}(x_j)\\\\\n\t\t& \\text{If $\\alpha$ leaf node:} \\hspace{2mm}\\nu_{\\alpha\\to i}(x_i)=\\max_{\\bm{x}_{\\alpha\\setminus i}}\\log  f_{\\alpha}(\\bm{x}_{\\alpha})\\\\\n\t\t\\textbf{Variable$\\to$Factor:} & \\hspace{2mm} \\nu_{j\\to \\alpha}(x_j)=\\sum_{\\beta\\in \\text{ne}(j)\\setminus \\alpha}\\nu_{\\beta\\to j}(x_j)\\\\\n\t\t&\\text{If $j$ leaf node:} \\hspace{2mm}\\nu_{j\\to \\alpha}(x_j)=0\n\t\t\\end{split}\n\t\t\\)}\n\t\\end{equation*}\n\t\\item The maximum beliefs/marginals are:\n\t\\begin{equation*}\n\t\\tcbox[nobeforeafter]{\\(\n\t\t\\begin{split}\n\t\t\\textbf{Max-marginals:} & \\hspace{2mm} q_i(x_i)=\\sum_{\\alpha\\in\\text{ne}(i)}\\nu_{\\alpha\\to i}(x_i)\\\\\n\t\t\\end{split}\n\t\t\\)}\n\t\\end{equation*}\n\t\\item In the case that $q_i(x_i)$ has a unique maximum, we can simply take the argmax to get the optimum: $x^{*}_i = \\arg\\max_{x_i} q_i(x_i)$\n\t\n\tIf this is not the case, we need to run the Viterbi algorithm (Bishop 8.4.5, we do not go in detail for this in the exam) to get the global optimum. The general idea is that some optima might depend on each other. For example, if we have something similar to a XOR, $(x_0=0, x_1=1)$ and $(x_0=1,x_1=0)$ are the optimums, but just looking at the independent marginals, $(x_0=0,x_1=0)$ and $(x_0=1,x_1=1)$ would be also optima. We can prevent this by slightly extending the messages passed around.\n\\end{itemize}"
  },
  {
    "path": "Machine_Learning_2/ml2_graphical_models.tex.recover.bak~",
    "content": "\\section{Probabilistic graphical models}\n\\begin{itemize}\n\t\\item It is often beneficial to visualize a probabilistic model as a diagram, which we call \\textit{(probabilistic) graphical models}. \n\t\\item They are good for:\n\t\\begin{itemize}\n\t\t\\item causal reasoning/modeling\n\t\t\\item calculating inference and conditional distributions efficiently \n\t\t\\item Designing and communicating statistical model\n\t\t\\item Encoding independence relations\n\t\\end{itemize} \n\t\\item Note that there are often multiple ways to express the same probability distribution. For example, take a joint distribution $p(A,B,C)$, which we can either write as $p(A,B,C)=p(A)p(B|A)p(C|A,B)$ (see Figure~\\ref{}) or $p(A,B,C)=p(C)p(A|C)p(B|A,C)$ (see Figure~\\ref{}). Nevertheless, what we interested in the end is the graphical representation with the least number of edges, as e.g. if $A$ and $B$ are independent, we can drop the edge between those.\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\begin{subfigure}{0.4\\textwidth}\n\t\t\t\\centering\n\t\t\t\\tikz{ %\n\t\t\t\t\\node[latent] (A) {$A$} ; %\n\t\t\t\t\\node[latent, right=of A] (B) {$B$} ; %\n\t\t\t\t\\node[latent, below=of B] (C) {$C$} ; %\n\t\t\t\t\n\t\t\t\t\\edge{A}{B};\n\t\t\t\t\\edge{A}{C};\n\t\t\t\t\\edge{B}{C};\n\t\t\t}\n\t\t\t\\caption{$p(A,B,C)=p(A)p(B|A)p(C|A,B)$}\n\t\t\\end{subfigure}\n\t\t\\hspace{10mm}\n\t\t\\begin{subfigure}{0.4\\textwidth}\n\t\t\t\\centering\n\t\t\t\\tikz{ %\n\t\t\t\t\\node[latent] (A) {$A$} ; %\n\t\t\t\t\\node[latent, right=of A] (B) {$B$} ; %\n\t\t\t\t\\node[latent, below=of B] (C) {$C$} ; %\n\t\t\t\t\n\t\t\t\t\\edge{A}{B};\n\t\t\t\t\\edge{C}{A};\n\t\t\t\t\\edge{C}{B};\n\t\t\t}\n\t\t\t\\caption{$p(A,B,C)=p(C)p(A|C)p(B|A,C)$}\n\t\t\t\\label{fig:graphical_models_example_2}\n\t\t\\end{subfigure}\n\t\t\\caption{Two different graphical models for the same joint distribution $p(A,B,C)$.}\n\t\\end{figure}\n\t\\item We distinguish between directed acyclic graphs, which we call \\textit{Bayesian networks} (BN), and undirected graphs, which are \\textit{Markov Random Fields} (MRF)\n\\end{itemize}"
  },
  {
    "path": "Machine_Learning_2/ml2_sampling_methods.tex",
    "content": "\\section{Sampling methods}\n\\begin{itemize}\n\t\\item In the previous chapter, we have seen that we can perform inference by approximating the posterior distribution. However, as alternative, we can also consider Monte Carlo techniques, i.e. sampling\n\t\\item In most practical cases, we often want to evaluate expectations over the posterior. This we can approximate by an average over samples:\n\t$$\\E_{p(\\bm{x})}\\left[f(\\bm{x})\\right] \\approx \\frac{1}{N}\\sum_{n=1}^{N} f(\\bm{x}_n), \\hspace{2mm} \\bm{x}_n\\sim p(\\bm{x})$$\n\tThis can be for example used for prediction:\n\t$$p(y^{*}|x^{*}) = \\int p(y^{*}\\vert x^{*}, \\bm{\\theta})p(\\bm{\\theta}|\\bm{X}, \\bm{Y})d\\bm{\\theta} = \\E_{p(\\bm{\\theta}|\\bm{X}, \\bm{Y})}\\left[p(y^{*}\\vert x^{*}, \\bm{\\theta})\\right] \\approx \\frac{1}{K}\\sum_{k=1}^{K} p(y^{*}\\vert x^{*}, \\bm{\\theta}^{(k)}), \\hspace{2mm} \\bm{\\theta}^{(k)}\\sim p(\\bm{\\theta}|\\bm{X}, \\bm{Y})$$\n\t\\item Hence, we will look at different techniques for sampling from more complex distributions than standard Gaussians\n\\end{itemize}\n\\subsection{Regular Sampling}\n\\begin{itemize}\n\t\\item As an introduction, we look at how we can sample from simple, known distributions, and verify the correctness of the Monte Carlo approximation\n\t\\item Assume we draw $N$ samples from $p(z)$: $z_i\\sim p(z)$, $i=1,...,N$\n\t\\item We can calculate $\\E\\left[f\\right]\\approx \\widehat{E\\left[f\\right]}=\\left<f\\right> = \\frac{1}{N}\\sum_{n=1}^{N}f(z_i)$ for approximating the expectation. Note that we introduced here the different notations for the Monte Carlo approximation\n\t\\item To verify that our approximation makes sense, we first check whether we have an \\textit{unbiased} estimator:\n\t$$\\E\\left[\\left<f\\right>\\right] = \\E\\left[\\frac{1}{N}\\sum_{i=1}^{N}f(z_i)\\right]=\\frac{1}{N}\\sum_{i=1}^{N}\\E\\left[f(z_i)\\right]=\\E\\left[f(z)\\right]$$\n\t\\item Furthermore, we would like that with an infinite amount of samples, our variance goes to zero:\n\t$$\\mathbb{V}\\text{ar}\\left[\\left<f\\right>\\right]=\\frac{1}{N^2}\\sum_{i=1}^{N}\\mathbb{V}\\text{ar}\\left[f(z_i)\\right] = \\frac{1}{N}\\mathbb{V}\\text{ar}\\left[f\\right]$$\n\tHence, with $N\\to\\infty$, we linearly reduce the variance compared to a single sample.\n\t\\item As a last part, we will look into how we can sample from any \\textit{known} distribution, given a uniform sampler\n\t\\begin{description}\n\t\t\\item[Discrete random variables]  In case of a discrete random variable $z\\in\\left\\{1,2,...,K\\right\\}$, and given the distribution $p(z)$, we first have to calculate the cumulative density function: $p(z\\leq \\zeta)$. Then, given $u_i\\sim U(0,1)$, the sample $k$ of $p(z)$ is where $p(z\\leq k-1)\\leq  u_i < p(z\\leq k)$\n\t\t\\item[Continuous random variables] For continuous variables, we need to calculate the CDF by an integral:\n\t\t$$F(\\zeta)=\\int_{-\\infty}^{\\zeta} p(z)dz = p(z\\leq \\zeta)$$\n\t\t, and take its inverse $F^{-1}$. Then, given $u_i\\sim U(0,1)$, our sample is $z_i=F^{-1}(u_i)$ which is a change of variables.\n\t\\end{description}\n\\end{itemize}\n\\subsection{Rejection sampling}\n\\begin{itemize}\n\t\\item Assume we have a probability density $p(z)$ from which we want to sample. Choose another distribution $q(z)$, called \\textit{proposal} distribution, from which we can sample. Further constraints are that the unnormalized distributions $\\tilde{p}(z)\\propto p(z), \\tilde{q}(z)\\propto q(z)$ fulfill:\n\t$$\\int \\tilde{q}(z)dz < \\infty, \\tilde{p}(z)\\leq \\tilde{q}(z) \\forall z$$\n\t\\item Now, we can generate samples from $p(z)$ by a simple algorithm:\n\t\\begin{tcolorbox}[colback=white!85!gray,colframe=gray!75!black,title=Pseudocode for rejection sampling]\n\t\t\\begin{algorithm}[H]\n\t\t\t\\SetAlgoLined\n\t\t\t\\For{$n=1,...,N$}{\n\t\t\t\t\\While{No sample for $z_n$ accepted}{\n\t\t\t\t\tSample $\\hat{z}$ from $q(z)$\\;\n\t\t\t\t\tSample $u\\sim U(0,1)$\\;\n\t\t\t\t\t\\eIf{$u<\\frac{\\tilde{p}(\\hat{z})}{\\tilde{q}(\\hat{z})}$}{Accept sample $z_n=\\hat{z}$\\;}{Reject sample $\\hat{z}$ and re-sample\\;}\n\t\t\t\t}\n\t\t\t}\n\t\t\tReturn samples $\\left\\{z_n\\right\\}_{n=1}^{N}$\\;\n\t\t\\end{algorithm}\n\t\\end{tcolorbox}\n\t\\item The principle of rejection sampling is visualized in Figure~\\ref{fig:sampling_rejection_sampling}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.3\\textwidth]{figures/sampling_rejection_sampling.png}\n\t\t\\caption{Visualizing rejection sampling. In the figure, $\\tilde{q}(z)$ is denoted as $kq(z)$, and $x_0$ is the initial sample from $q$.}\n\t\t\\label{fig:sampling_rejection_sampling}\n\t\\end{figure}\n\t\\item To show that we actually generate samples from $p(z)$, we can write down the probability for a value $z_i$ to be picked. First, the chance of $z_i$ being generated in first place is $q(z_i)$. Next, the chance of $z_i$ being accepted, is $\\frac{\\tilde{p}(z_i)}{\\tilde{q}(z_i)}$. Together, we get the probability of $z_i$:\n\t$$\\hat{p}(z_i) = q(z_i)\\frac{\\tilde{p}(z_i)}{\\tilde{q}(z_i)} \\propto p(z_i)$$\n\tHence, we actually generate samples from $z_i$ although we initially sample from $q(z)$\n\t\\item One requirement of rejection sampling to work well is that the area between $\\tilde{p}(z)$ and $\\tilde{q}(z)$ is small. The efficience of this sampler can be measured by the acceptance rate, which is $\\E_{z_i\\sim q}\\left[\\frac{\\tilde{p}(z_i)}{\\tilde{q}(z_i)} \\right]$. If this value is low, it means that a lot of samples are rejected, hence the sampling process takes longer. This gets especially critical in higher dimensions as we need to make sure that \\textit{for all} $z_i$, $\\tilde{q}(z_i)$ is greater than $\\tilde{p}(z_i)$. Finding a simple distribution in high dimensions that fulfills this requirement is often not trivial\n\\end{itemize}\n\\subsection{Importance sampling}\n\\begin{itemize}\n\t\\item Another approach for estimating an expectation is not generating actual samples from $p$, but simply weight samples by their \\textit{importance}\n\t\\item Again, we need two distributions: $p$ over which we want to determine the expectation, and $q$ that we actually sample from. Note that for this algorithm to work, none of these distributions need to be normalized. The only thing that is required is that we can sample from the normalized density of $q$.\n\t\\item The pseudo-code for the importance sampling is fairly simple and straight forward: \\begin{tcolorbox}[colback=white!85!gray,colframe=gray!75!black,title=Pseudocode for importance sampling]\n\t\t\\begin{algorithm}[H]\n\t\t\t\\SetAlgoLined\n\t\t\tSample $\\left\\{z_n\\right\\}_{n=1}^{N}$ from $q(z)$\\;\n\t\t\tCalculate the weights $w_n=\\frac{p(z_n)}{q(z_n)}$\\;\n\t\t\tDetermine expectation by $\\E_p[f]\\approx \\frac{\\sum_n w_n f(z_n)}{\n\t\t\t\\sum_n w_n}$\\;\n\t\t\\end{algorithm}\n\t\\end{tcolorbox}\n\t\\item The intuition of importance sampling can be shown when plugging in the two distributions:\n\t$$\\E_p[f]=\\int p(z)f(z)dz=\\frac{\\int q(z)\\frac{p(z)}{q(z)}f(z)dz}{\\underbrace{\\int q(z)\\frac{p(z)}{q(z)}dz}_{=1}} = \\frac{\\E_q\\left[w_z\\cdot f(z)\\right]}{\\E_q\\left[w_z\\right]} \\approx \\E_q\\left[\\frac{\\sum_i w_i f(z_i)}{\\sum_i w_i}\\right]$$\n\t\\item Although importance sampling can use all samples, it has two major drawbacks:\n\t\\begin{itemize}\n\t\t\\item The estimate of $\\E[f]$ is \\underline{not} unbiased. Imagine we would sample only a single time ($N=1$ in pseudo code). As the sum drops out, we end up with $f(z_i)$ where $z_i$ is sampled from $q$, and not $p$! This bias decreases with the number of samples, but should always be kept in mind as an accurate estimate might require more samples, especially when it occurs with the second drawback.\n\t\t\\item The importance weighting estimate has a high variance if $p$ differs strongly from $q$, especially when $p(z_i)\\gg q(z_i)$ as the weight is very high for this data point. Imagine $q$ is a Gaussian with a large variance, while $p$ is a peaked Gaussian around 0. For most of the samples, $p(z)\\ll q(z)$, and hence their weight is low. But for a small set of points, namely those close to 0, $p(z)$ is much greater than $q(z)$ resulting in a high weight. If we now sample e.g. 100 times, all points except those around 0 are neglected due to their low weight. And as those important points are rarely sampled from $q$, we need many samples to reduce the variance. This problem occurs even stronger in high-dimensional spaces.\n\t\\end{itemize}\n\t\\item Furthermore, note that importance sampling can only be used for approximating an expectation, and not for generating independent samples from $p$\n\\end{itemize}\n\\subsection{Ancestral Sampling}\n\\begin{itemize}\n\t\\item Assume we have given a Bayesian network, and want to sample from the joint probability. We can write the joint probability as:\n\t$$p(\\bm{z})=\\prod_{i=1}^{d} p(z_i\\vert z_{\\text{pa}(i)})$$\n\twhere we use a topological ordering $z_1,...,z_d$ with $z_j<z_i$ if $j\\in \\text{pa}(i)$\n\t\\item Now we can simply sample from the joint distribution by sampling from each of the conditionals, in the topological ordering. For example, assume we have the following Bayesian network:\n\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\tikz{ %\n\t\t\t\\node[latent] (z1) {$z_1$} ; %\n\t\t\t\\node[latent, right=of z1] (z3) {$z_3$} ; %\n\t\t\t\\node[latent, above=of z3] (z2) {$z_2$} ; %\n\t\t\t\\node[latent, below=of z3] (z4) {$z_4$} ; %\n\t\t\t\\node[latent, right=of z2] (z5) {$z_5$} ; %\n\t\t\t\\node[latent, right=of z4] (z6) {$z_6$} ; %\n\t\t\t\n\t\t\t\\edge{z1}{z2};\n\t\t\t\\edge{z1}{z3};\n\t\t\t\\edge{z1}{z4};\n\t\t\t\\edge{z2}{z5};\n\t\t\t\\edge{z3}{z5};\n\t\t\t\\edge{z3}{z6};\n\t\t\t\\edge{z2}{z6};\n\t\t\t\\edge{z4}{z6};\n\t\t}\n\t\\end{figure}\n\n\tThen we can sample as follows:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\tilde{z}_1 & \\sim p(z_1)\\\\\n\t\t\t\\tilde{z}_2 & \\sim p(z_2|z_1=\\tilde{z}_1)\\\\\n\t\t\t\\tilde{z}_3 & \\sim p(z_3|z_1=\\tilde{z}_1)\\\\\n\t\t\t\\tilde{z}_4 & \\sim p(z_4|z_1=\\tilde{z}_1)\\\\\n\t\t\t\\tilde{z}_5 & \\sim p(z_5|z_2=\\tilde{z}_2,z_3=\\tilde{z}_3)\\\\\n\t\t\t\\tilde{z}_6 & \\sim p(z_6|z_2=\\tilde{z}_2,z_3=\\tilde{z}_3,z_4=\\tilde{z}_4)\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\n\t\\item Note that for sampling from each of the individual distributions, we can use the other techniques like rejection or importance sampling. \n\t\n\t% \\item \\TODO{Question: lecture notes say that it works badly with high dimensions, but other sources say it works well. Why would it not work in high dimensions?}\n\\end{itemize}\n\\subsection{Markov-Chain Monte Carlo}\n\\begin{itemize}\n\t\\item Given a target distribution $p(x)$ that we want to sample from, we setup a Markov chain such that $p(x_n)\\to p(x)$ as $N\\to\\infty$, i.e. such that $p(x)$ is its equilibrium distribution\n\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\tikz{ %\n\t\t\t\\node[latent] (x1) {$x_1$} ; %\n\t\t\t\\node[latent, right=of x1] (x2) {$x_2$} ; %\n\t\t\t\\node[latent, right=of x2] (x3) {$x_3$} ; %\n\t\t\t\\node[const, right=of x3] (xetc1) { \\hspace{2mm}...\\hspace{2mm} } ; %\n\t\t\t\\node[latent, right=of xetc1] (xN) {$x_N$} ; %\n\t\t\t\\node[const, right=of xN] (xetc2) { \\hspace{2mm}...\\hspace{2mm} } ; %\n\t\t\t\\node[latent, right=of xetc2] (xInfty) {$x_{\\infty}$} ; %\n\t\t\t\n\t\t\t\\edge{x1}{x2};\n\t\t\t\\edge{x2}{x3};\n\t\t\t\\edge{x3}{xetc1};\n\t\t\t\\edge{xetc1}{xN};\n\t\t\t\\edge{xN}{xetc2};\n\t\t\t\\edge{xetc2}{xInfty};\n\t\t}\n\t\\end{figure}\n\t\\item Note that as MCMC runs ancestral sampling on a chain, two consecutive samples are no longer independent. Still, we can assume that the dependency between two fairly distant samples is neglectable\n\t\\item We start sampling $x_1$ from an initial distribution $q(x_1)$ which can be chosen arbitrarily (but the closer $q$ to $p$, the faster the convergence and hence, faster sampling). For the next samples, we use a \\underline{transition kernel} $T$ (which we have to specify/find by ourselves) to get $x_2\\sim T(x_2|x_1)$\n\t\\item The transition kernel is usually independent of time (most efficient), but can be extended to multiple steps (e.g. $x_t$ is sampled by $T_1$, $x_{t+1}$ from $T_2$, and $x_{t+2}$ again from $T_1$)\n\t\\item The marginal distribution can be determined by:\n\t$$q(x_{n}) = \\int q(x_{n-1})T(x_{n}|x_{n-1})dx_{n-1}$$\n\tand the joint probability for all $x_1,...,x_N$ is:\n\t$$q(x_1,...,x_N) = q(x_1)\\prod_{i=2}^{N}T(x_i|x_{i-1})$$\n\t\\item The equilibrium is reached when $x_N\\sim p(x)$ (i.e. $p(x)$ is an invariant of the chain):\n\t$$p(x_{N+1}) = \\int T(x_{N+1}|x_N)p(x_N)dx_N$$ \n\tIn this case, $p(x)$ is called to be an invariant of the chain. Note that a chain can have multiple invariants, as if $T$ is the identity transformation, any distribution is an invariant of that chain\n\t\\item A sufficient but not necessary condition of an invariant $p(x)$ is that it satisfies the property of \\underline{detailed balance}:\n\t$$p(x_t)T(x_{t+1}|x_t) = p(x_{t+1})T(x_t|x_{t+1})$$\n\tThis property can be interpreted as the Markov chain being reversible. Although detailed balance is not required, it is mostly easier to fulfill when designing a kernel\n\t\\item Another property we are looking for is \\underline{ergodicity}. A sufficient condition for ergodicity is that any state $x_N$ has a positive probability, for any $N$.\n\t\\item If we have an ergodic Markov chain and $p^{\\star}(x)$ being an invariant, then $p^{\\star}(x)$ is a unique equilibrium, where for any start distribution $q(x_0)$, the distribution $q(x_N)$ with $N\\to\\infty$ converges to the required distribution $p^{\\star}(x)$.\n\t\\item The sketch of the sampling process is:\n\t\n\t\\begin{algorithm}[H]\n\t\tSample initial state $x_0$ from $q(x_0)$\\;\n\t\t\\For{$t=0,...,N$}{\n\t\t\tSample $x_{t+1}\\sim T(x_{t+1}|x_{t})$\\;\n\t\t}\n\t\tOutput $x_N$ as sample of $p^{\\star}(x)$\\;\n\t\\end{algorithm}\n\t$N$ has to be large enough in this case, and is often referred to as \\textit{burn-in} time. \n\t\n\tIn addition, note that the required $N$ can be reduced by reducing the distance between $q$ and $p$. Thus, for generating multiple samples, we can take the first sample $x_N$, and continue the chain for additional $M$ steps, where usually $M\\ll N$ (but $M$ must be certain size to guarantee independence of samples, depends on $T$). Then, $x_{N+M}$ is a new sample from $p$.\n\t\n\t\\item The problem of MCMC is how we can find the transition kernel $T$ for a distribution $p$. Once we have found two kernels, we can combine those to new kernels:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tT_3 & = \\alpha T_1 + (1 - \\alpha) T_2, \\hspace{2mm}(\\alpha \\in [0,1])\\\\\n\t\t\tT_3 & = T_2 \\circ T_1 \\hspace{5mm}(\\text{composition, first apply $T_1$, then $T_2$})\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item One way to overcome this problem is the Metropolis-Hastings algorithm, which uses a form of rejection sampling to allow any transition kernel $T$\n\\end{itemize}\n\\subsubsection{Metropolis-Hastings algorithm}\n\\begin{itemize}\n\t\\item Choose a proposal transition kernel $Q(x_{t+1}|x_{t})$ (e.g. a random walk). Then we can sample from $p^{\\star}(x)$ as follows:\n\t\\begin{tcolorbox}[colback=white!85!gray,colframe=gray!75!black,title=Pseudocode for Metropolis-Hastings algorithm]\n\t\t\\begin{algorithm}[H]\n\t\t\t\\SetAlgoLined\n\t\t\tSample initial state $x_0$ from $q(x_0)$\\;\n\t\t\t\\For{$t=0,...,N$}{\n\t\t\t\tSample $\\tilde{x}_{t+1}\\sim Q(\\tilde{x}_{t+1}|x_{t})$\\;\n\t\t\t\tCompute acceptance probability $\\alpha(\\tilde{x}_{t+1}|x_t) = \\min\\left(1, \\frac{p^{\\star}(\\tilde{x}_{t+1}) Q(x_t|\\tilde{x}_{t+1})}{p^{\\star}(x_t)Q(\\tilde{x}_{t+1}|x_{t})}\\right)$\\;\n\t\t\t\tSample $u_t\\sim U(0,1)$\\;\n\t\t\t\t\\eIf{$u_t \\leq \\alpha(\\tilde{x}_{t+1}|x_t) $}{accept sample $x_{t+1}=\\tilde{x}_{t+1}$\\;}{reject sample and stay at current state: $x_{t+1}=x_t$\\;}\n\t\t\t}\n\t\t\tOutput $x_N$ as sample of $p^{\\star}(x)$\\;\n\t\t\\end{algorithm}\n\t\\end{tcolorbox}\t\n\tWe can view the combination of $Q$ and acceptance probability $\\alpha$ as our transition kernel $T=\\alpha\\circ Q$.\n\t\\item Given this simple algorithm, we can design the transition kernel $Q$ with minimal knowledge of $p^{\\star}$. For example, a kernel which mostly works well is to combine larger and smaller random walk steps. This can be very helpful in high-dimensional space as it can explore in different scales, and hence, in contrast to rejection and importance sampling, still works well in high dimensions\n\t\\item Nonetheless, keep in mind that the performance now depends on the transition kernel $Q$. If it is designed poorly, many samples are rejected and we again end up with an inefficient sampling process. For example, if we have a highly multi-modal distribution, we need to have large enough steps to be able to jump between modes. But as said before, this is much less knowledge we need of $p^{\\star}$ compared to the other discussed sampling algorithms\n\t\\item We can sketch the proof here for the detailed balance. If a new sample $x_{t+1}$ is accepted, it has the probability:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tp^{\\star}(x_t)T(x_{t+1}|x_{t}) & = p^{\\star}(x_t)Q(x_{t+1}|x_{t})\\min\\left(1, \\frac{p^{\\star}(x_{t+1}) Q(x_t|x_{t+1})}{p^{\\star}(x_t)Q(x_{t+1}|x_{t})}\\right)\\\\\n\t\t\t& = \\min\\left(p^{\\star}(x_t)Q(x_{t+1}|x_{t}), p^{\\star}(x_{t+1}) Q(x_t|x_{t+1})\\right)\\\\\n\t\t\t& = p^{\\star}(x_{t+1})Q(x_{t}|x_{t+1})\\min\\left(1, \\frac{p^{\\star}(x_{t}) Q(x_{t+1}|x_{t})}{p^{\\star}(x_{t+1})Q(x_{t}|x_{t+1})}\\right)\\\\\n\t\t\t& = p^{\\star}(x_{t+1})T(x_{t}|x_{t+1})\n\t\t\\end{split}\n\t\\end{equation*}\n\tAs we can interchange $x_t$ and $x_{t+1}$, $p^{\\star}$ is an invariant of the Markov chain.\n\t\n\tIn case $\\tilde{x}_{t+1}$ was rejected, we stay at $x_t$ which satisfies the detailed balance anyways.\n\\end{itemize}\n\\subsubsection{Gibbs sampling}\n\\begin{itemize}\n\t\\item Gibbs sampling is a special case of the Metropolis-Hastings algorithm, where the acceptance probability is always 1. The idea is that we cannot easily sample from a big joint distribution, but sampling a single variable given all others is feasible/much simpler. \n\t\\item Hence, we sample a D-dimensional vector $(x_1,x_2,...,x_D)$ by sampling from the conditional distributions of a single variable $x_i$ where we keep all other variables fixed. In pseudo-code, we can define Gibbs sampling as:\n\t\\begin{tcolorbox}[colback=white!85!gray,colframe=gray!75!black,title=Pseudocode for Gibbs sampling]\n\t\t\\begin{algorithm}[H]\n\t\t\t\\SetAlgoLined\n\t\t\tChoose an initial state $\\left\\{x_i:i=1,...,M\\right\\}$\\;\n\t\t\t\\For{$t=0,...,N$}{\n\t\t\t\tSample $x_1^{(t+1)}\\sim p(x_1|x_2^{(t)}, x_3^{(t)},...,x_M^{(t)})$\\;\n\t\t\t\tSample $x_2^{(t+1)}\\sim p(x_2|x_1^{(t+1)}, x_3^{(t)},...,x_M^{(t)})$\\;\n\t\t\t\t... \\\\\n\t\t\t\tSample $x_M^{(t+1)}\\sim p(x_M|x_2^{(t+1)}, x_3^{(t+1)},...,x_{M-1}^{(t+1)})$\\;\n\t\t\t}\n\t\t\tOutput $\\left\\{x_1^{(N)},...,x_M^{(N)}\\right\\}$ as sample of $p(x_1,...,x_M)$\\;\n\t\t\\end{algorithm}\n\t\\end{tcolorbox}\t\n\t\\item Clearly, $p(\\bm{x})$ is an invariant of the Markov chain because at each step, we sample from the correct conditional $p(x_i,\\bm{x}_{\\setminus i})$. But note that detailed balance can only be guaranteed if the order of $x_i$'s is randomized every iteration.\n\t\\item For ergodicity, we just need to make sure that the conditionals are not zero for any point. Otherwise, we have to prove ergodicity explicitly.\n\t\\item The acceptance probability of a sample according to the Metropolis-Hastings algorithm is in case of Gibbs sampling:\n\t$$\\alpha(\\bm{x}^{(t+1)}|\\bm{x}^{(t)}) = \\frac{p^{\\star}(\\bm{x}^{(t+1)}) Q(\\bm{x}^{(t)}|\\bm{x}^{(t+1)})}{p^{\\star}(\\bm{x}^{(t)})Q(\\bm{x}^{(t+1)}|\\bm{x}^{(t)})} = \\frac{p^{\\star}(x_i^{(t+1)}|\\bm{x}_{\\setminus i}^{(t+1)})p^{\\star}(\\bm{x}_{\\setminus i}^{(t+1)})\\cdot p^{\\star}(x_i^{(t)}|\\bm{x}_{\\setminus i}^{(t+1)})}{p^{\\star}(x_i^{(t)}|\\bm{x}_{\\setminus i}^{(t)})p^{\\star}(\\bm{x}_{\\setminus i}^{(t)})\\cdot p^{\\star}(x_i^{(t+1)}|\\bm{x}_{\\setminus i}^{(t)})} = 1$$\n\twhere $\\bm{x}_{\\setminus i}^{(t)}=\\bm{x}_{\\setminus i}^{(t+1)}$ as the other variables do not change.\n\\end{itemize}"
  },
  {
    "path": "Machine_Learning_2/ml2_sequential_data.tex",
    "content": "\\section{Sequential Data}\n\\begin{itemize}\n\t\\item Most models we discussed so far assumed that multiple data points $\\bm{x}_n$ are independent of each other. However, in many use-cases, we have e.g. temporal data where consecutive data points dependent on each other\n\t\\item The likelihood of such data can be written as:\n\t$$p(x_1,...,x_N) = \\prod_{n=1}^{N} p(x_n|x_1,...,x_{n-1})$$\n\t\\item Here, we will focus on Markov models and its different variations\n\\end{itemize}\n\\subsection{Markov models}\n\\begin{itemize}\n\t\\item One of the simplest models for sequential data are Markov models, where we limit the conditionals to a fixed size. For example, a \\textit{first-order} Markov model has the likelihood $p(x_1)\\prod_{n=2}^{N}p(x_n|x_{n-1})$:\n\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\tikz{ %\n\t\t\t\\node[obs] (x1) {$x_1$} ; %\n\t\t\t\\node[obs, right=of x1] (x2) {$x_2$} ; %\n\t\t\t\\node[obs, right=of x2] (x3) {$x_3$} ; %\n\t\t\t\\node[const, right=of x3] (xetc1) { \\hspace{2mm}...\\hspace{2mm} } ; %\n\t\t\t\\node[obs, right=of xetc1] (xN) {$x_N$} ; %\n\t\t\t\n\t\t\t\\edge{x1}{x2};\n\t\t\t\\edge{x2}{x3};\n\t\t\t\\edge{x3}{xetc1};\n\t\t\t\\edge{xetc1}{xN};\n\t\t}\n\t\\end{figure}\n\n\tA second-order MM would add connections between $x_1$ and $x_3$, $x_2$ and $x_4$ etc.\n\t\\item We call a Markov model \\underline{homogeneous} if all conditionals $p(x_{n}|x_{n-1})$ are the same, i.e. the transition probability is steady over time:\n\t$$p(x_{n}=a|x_{n-1}=b) = p(x_{n+m}=a|x_{n+m-1}=b)$$\n\t\\item Suppose each $x_n$ has $K$ possible states. Then the parameters for a homogeneous MM increases over the order by:\n\t\\begin{table}[ht!]\n\t\t\\centering\n\t\t\\begin{tabular}{c|cc}\n\t\t\tOrder & \\multicolumn{2}{c}{Number of params}\\\\\n\t\t\t& Prior & Conditionals\\\\\n\t\t\t\\hline \n\t\t\t$0$ & $K-1$ & $0$\\\\\n\t\t\t$1$ & $K-1$ & $K(K-1)$\\\\\n\t\t\t$2$ & $K^2 - 1$ & $K^2(K-1)$\\\\\n\t\t\t... & ... & ...\\\\\n\t\t\t$M$ & $K^{M}-1$ & $K^{M}(K-1)$\\\\\n\t\t\\end{tabular}\n\t\\end{table}\n\n\tFor inhomogeneous MM, we would have $N-M$-times the conditional parameters where $N$ is the length of the chain (needs fixed size if conditionals are not shared!). \n\t\n\tAs the number of parameters increases exponentially with the order $M$, it is often impractical to use high-order MM. \n\t\\item Our next discussions will focus on the first-order MM but can be applied to any order. This can be easily shown by reducing a M'th order MM to 1st order MM. Suppose we\n\tintroduce new variables $y_n=(x_n,x_{n+1},...,x{n+M-1})$, then we can express our MM by:\n\t$$\\hspace{-10mm}p(y_n=(x_n,x_{n+1},...,x_{n+M-1})|y_{n-1}=(\\tilde{x}_{n-1},\\tilde{x}_n,...,\\tilde{x}_{n+M-2}))=\\begin{cases}\n\t0 & \\text{if for any } i, x_i\\neq \\tilde{x}_i\\\\\n\tp(x_{n+M-1}|x_n,...,x_{n+M-2}) & \\text{otherwise}\n\t\\end{cases}$$\n\t\\item Often, we want Markov models that are more expressive than a first-order, but at the same time, prevent having too many parameters. One way of enriching the class of MMs is by introducing latent variables as shown in this graphical model:\n\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\tikz{ %\n\t\t\t\\node[latent] (z1) {$z_1$} ; %\n\t\t\t\\node[latent, right=of z1] (z2) {$z_2$} ; %\n\t\t\t\\node[latent, right=of z2] (z3) {$z_3$} ; %\n\t\t\t\\node[obs, below=of z1] (x1) {$x_1$} ; %\n\t\t\t\\node[obs, below=of z2] (x2) {$x_2$} ; %\n\t\t\t\\node[obs, below=of z3] (x3) {$x_3$} ; %\n\t\t\t\\node[const, right=of z3] (zetc1) { \\hspace{2mm}...\\hspace{2mm} } ; %\n\t\t\t\n\t\t\t\\edge{z1}{z2};\n\t\t\t\\edge{z2}{z3};\n\t\t\t\\edge{z3}{zetc1};\n\t\t\t\n\t\t\t\\edge{z1}{x1};\n\t\t\t\\edge{z2}{x2};\n\t\t\t\\edge{z3}{x3};\n\t\t}\n\t\\end{figure}\n\t\\item The joint distribution for this model is given by: $$p(x_1,...,x_N,z_1,...,z_N)=p(z_1)\\cdot \\left[\\prod_{n=2}^{N} p(z_n|z_{n-1})\\right]\\cdot \\left[\\prod_{n=1}^{N} p(x_n|z_{n})\\right]$$\n\t\\item We now look at two variants of this model:\n\t\\begin{enumerate}\n\t\t\\item \\textit{Hidden Markov Models} assume that the latent space $z$ is discrete\n\t\t\\item \\textit{Linear Dynamical Systems} use a continuous latent space $z$ such as linear Gaussian\n\t\\end{enumerate}\n\\end{itemize}\n\n\\subsection{Hidden Markov Models}\n\\begin{itemize}\n\t\\item Commonly, when using discrete latent variables, we assume that we have a mixture model, and $z_n$ as discrete multinomial variables specify the component from which $x_n$ was generated\n\t\\item For the homogeneous case, we get the joint distribution:\n\t$$p(\\bm{x}_1,...,\\bm{x}_N|\\bm{\\pi},\\bm{A},\\bm{\\phi}) = \\sum_{\\bm{z}_1}...\\sum_{\\bm{z}_N} p(\\bm{z}_1|\\bm{\\pi})\\left[\\prod_{n=2}^{N} \\underbrace{p(\\bm{z}_n|\\bm{z}_{n-1},\\bm{A})}_{\\text{Transition probabilities}}\\right]\\cdot \\left[\\prod_{n=1}^{N} \\underbrace{p(\\bm{x}_n|\\bm{z}_n,\\bm{\\phi})}_{\\text{Emission probabilities}}\\right]$$\n\twhere $\\bm{A}$ can be seen as a table with $A_{jk}\\equiv p(z_{nk}=1|z_{n-1,j}=1)$ which gives the constraints $\\sum_k A_{jk}=1$, $0\\leq A_{jk}\\leq 1$. \n\t\n\tThe parameter $\\bm{\\pi}$ is again a prior over latent states for $z_1$, where $\\sum_k \\pi_k = 1$.\n\t\n\tThe mapping between latent and observed variable is described as emission probabilities, which we can write as $p(\\bm{x}_n|\\bm{z}_n,\\bm{\\phi})=\\prod_{k=1}^{K} p(\\bm{x}_n|\\bm{\\phi}_k)^{z_{nk}}$\n\t\n\t\\item For optimizing the parameters, we again use the EM algorithm because otherwise we would need to calculate $p(\\bm{X}|\\bm{\\theta})=\\sum_Z p(\\bm{X},\\bm{Z}|\\bm{\\theta})$. The sum has a complexity which increases exponentially with the number of hidden variables, and makes it inefficient for large $N$.\n\t\n\\end{itemize}\n\\subsubsection{Maximum Likelihood for HMM}\n\\begin{itemize}\n\t\\item Remember that our objective in the EM algorithm was:\n\t$$Q(\\bm{\\theta}, \\bm{\\theta}^{\\text{old}})=\\sum_{\\bm{Z}} p(\\bm{Z}|\\bm{X},\\bm{\\theta}^{\\text{old}})\\ln p(\\bm{X},\\bm{Z}|\\bm{\\theta})=\\E_{\\bm{Z}\\sim p(\\bm{Z}|\\bm{X},\\bm{\\theta}^{\\text{old}})}\\left[\\ln p(\\bm{X},\\bm{Z}|\\bm{\\theta})\\right]$$\n\twhich can be written in our case as:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tQ(\\bm{\\theta}, \\bm{\\theta}^{\\text{old}}) & = \\E_{\\bm{z}_1\\sim p(\\bm{z}_1|\\bm{X},\\bm{\\theta}^{\\text{old}})}\\left[\\sum_{k=1}^{K} z_{1k}\\ln \\pi_k\\right] + \\\\\n\t\t\t& \\hspace{5mm}\\sum_{n=2}^{N} \\E_{(\\bm{z}_n, \\bm{z}_{n-1})\\sim p(\\bm{z}_n, \\bm{z}_{n-1}|\\bm{X},\\bm{\\theta}^{\\text{old}})}\\left[\\sum_{j=1}^{K}\\sum_{k=1}^{K}z_{nj}\\cdot z_{n-1,k}\\cdot \\ln A_{jk}\\right] + \\\\\n\t\t\t& \\hspace{5mm} \\sum_{n=1}^{N} \\E_{\\bm{z}_n\\sim p(\\bm{z}_n|\\bm{X},\\bm{\\theta}^{\\text{old}})} \\left[\\sum_{k=1}^{K} z_{nk} \\ln p(\\bm{x}_n|\\bm{\\phi}_k)\\right]\\\\\n\t\t\t& = \\sum_{k=1}^{K} \\gamma(z_{1k})\\ln \\pi_k + \\sum_{n=2}^{N} \\sum_{j=1}^{K}\\sum_{k=1}^{K}\\zeta(z_{n-1,j}, z_{nk})\\cdot \\ln A_{jk} + \\sum_{n=1}^{N} \\sum_{k=1}^{K} \\gamma(z_{nk})\\ln p(\\bm{x}_n|\\bm{\\phi}_k)\n\t\t\\end{split}\n\t\\end{equation*}\n\twhere we use $\\gamma(\\bm{z}_n)=p(\\bm{z}_n|\\bm{X},\\bm{\\theta}^{\\text{old}})$ and $\\zeta(\\bm{z}_{n-1},\\bm{z}_n)=p(\\bm{z}_{n-1},\\bm{z}_{n}|\\bm{X},\\bm{\\theta}^{\\text{old}})$.\n\t\\item Now, let's take a closer look at the steps of the EM algorithm\n\t\\begin{description}\n\t\t\\item[E-step] We need to determine $p(z_1,...,z_N|\\bm{X},\\bm{\\theta}^{\\text{old}})$ which we split into $\\gamma(\\bm{z}_n)$ and $\\zeta(\\bm{z}_{n-1},\\bm{z}_n)$. Hence, we need to calculate marginals, which can be done efficiently with the sum-product algorithm.\n\t\t\n\t\tFirst, we need to convert the Bayesian network into a factor graph. We do this by replacing the prior with  $h(\\bm{z}_1)=p(\\bm{z}_1|\\bm{\\pi}^{\\text{old}})p(\\bm{x}_1|\\bm{z}_1, \\bm{\\phi}^{\\text{old}})$, and the transitions by $f_n(\\bm{z}_{n-1},\\bm{z}_n)=p(\\bm{z}_n|\\bm{z}_{n-1}, \\bm{A}^{\\text{old}})p(\\bm{x}_n|\\bm{z}_n,\\bm{\\theta}^{\\text{old}})$. The corresponding factor graph looks like in Figure~\\ref{fig:HMM_factor_graph}.\n\t\t\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\tikz{ %\n\t\t\t\t\\factor[] {h1} {$h_1$} {} {} ;\n\t\t\t\t\\node[latent, right=of h1] (z1) {$\\bm{z}_{1}$} ; %\n\t\t\t\t\\factor[right=of z1] {f2} {$f_2$} {} {} ;\n\t\t\t\t\\node[latent, right=of f2] (z2) {$\\bm{z}_{2}$} ; %\n\t\t\t\t\\factor[right=of z2] {f3} {$f_3$} {} {} ;\n\t\t\t\t\\node[latent, right=of f3] (z3) {$\\bm{z}_{3}$} ; %\n\t\t\t\t\\factor[right=of z3] {f4} {$f_4$} {} {} ;\n\t\t\t\t\\node[const, right=of f4] (zetc) {\\hspace{2mm}...\\hspace{2mm} } ; %\n\t\t\t\t\\factor[right=of zetc] {fN} {$f_N$} {} {} ;\n\t\t\t\t\\node[latent, right=of fN] (zN) {$\\bm{z}_{N}$} ; %\n\t\t\t\t\t\t\n\t\t\t\t\\factoredge[-]{}{h1}{z1} ;\t\n\t\t\t\t\\factoredge[-]{z1}{f2}{z2} ;\t\n\t\t\t\t\\factoredge[-]{z2}{f3}{z3} ;\t\n\t\t\t\t\\factoredge[-]{z3}{f4}{zetc} ;\t\n\t\t\t\t\\factoredge[-]{zetc}{fN}{zN} ;\t\n\t\t\t}\n\t\t\t\\caption{Drawing of the factor graph. }\n\t\t\t\\label{fig:HMM_factor_graph}\n\t\t\\end{figure}\n\t\n\t\tUsing the factor graph, we want to determine the normalized beliefs $p(\\bm{z}_n|\\bm{X},\\bm{\\theta}^{\\text{old}})$. The sum product updates are:\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\mu_{z_{n-1}\\to f_n}(\\bm{z}_{n-1}) & = \\mu_{f_{n-1}\\to z_{n-1}}(\\bm{z}_{n-1})\\\\\n\t\t\t\t\\alpha_n(\\bm{z}_n) = \\mu_{f_n\\to z_n}(\\bm{z}_{n}) & = \\sum_{\\bm{z}_{n-1}} f_n(\\bm{z}_{n-1},\\bm{z}_n)\\alpha_{n-1}(\\bm{z}_{n-1})\\\\\n\t\t\t\t\\mu_{z_n \\to f_n}(\\bm{z}_n) & = \\mu_{f_{n+1} \\to z_n}(\\bm{z}_n) \\\\\n\t\t\t\t\\beta_n(\\bm{z}_n) & = \\sum_{z_{n+1}} f_{n+1}(\\bm{z}_n, \\bm{z}_{n+1})\\beta_{n+1}(\\bm{z}_{n+1})\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tOur beliefs can be calculated by:\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\text{Variable belief } \\hspace{2mm} p(\\bm{z}_n,\\bm{X}|\\bm{\\theta}^{\\text{old}}) & = \\alpha_n(\\bm{z}_n)\\beta_n(\\bm{z}_n)\\\\\n\t\t\t\t\\text{Factor belief } \\hspace{2mm} p(\\bm{z}_{n-1}, \\bm{z}_n,\\bm{X}|\\bm{\\theta}^{\\text{old}}) & = \\mu_{f_{n-1}\\to z_{n-1}}(z_{n-1})\\mu_{f_{n+1}\\to z_n}(\\bm{z}_n)f_n(\\bm{z}_{n-1}, \\bm{z}_n)\\\\\n\t\t\t\t\\text{Normalization constant } \\hspace{2mm} p(\\bm{X}|\\bm{\\theta}^{\\text{old}}) & = \\sum_{\\bm{z}_n} \\alpha_n(\\bm{z}_n)\\beta_n(\\bm{z}_n)\\\\\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tFinally, we can calculate our sufficient statistics:\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\gamma(\\bm{z}_n) & = \\frac{p(\\bm{z}_n, \\bm{X}|\\bm{\\theta}^{\\text{old}})}{p(\\bm{X}|\\bm{\\theta}^{\\text{old}})} = \\frac{\\alpha_n(\\bm{z}_n)\\beta_n(\\bm{z}_n)}{p(\\bm{X}|\\bm{\\theta}^{\\text{old}})}\\\\\n\t\t\t\t\\zeta(\\bm{z}_{n-1},\\bm{z}_n) & = \\frac{p(\\bm{z}_{n-1},\\bm{z}_n,\\bm{X}|\\bm{\\theta}^{\\text{old}})}{p(\\bm{X}|\\bm{\\theta}^{\\text{old}})} = \\frac{\\alpha_{n-1}(\\bm{z}_{n-1})\\beta_n(\\bm{z}_n)p(\\bm{z}_n|\\bm{z}_{n-1}, \\bm{A}^{\\text{old}})p(\\bm{x}_n|\\bm{z}_n, \\bm{\\phi}^{\\text{old}})}{p(\\bm{X}|\\bm{\\theta}^{\\text{old}})}\t\t\t\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\t\n\t\t\\item[M-step] In the maximization step, we optimize $Q(\\bm{\\theta}, \\bm{\\theta}^{\\text{old}})$ regarding the parameters $\\bm{\\pi}$, $\\bm{A}$ and $\\bm{\\phi}$. Note that we need to add Lagrangian for the constraints on $\\bm{A}$ and $\\bm{\\pi}$:\n\t\t$$\\tilde{Q}(\\bm{\\theta}, \\bm{\\theta}^{\\text{old}}) = Q(\\bm{\\theta}, \\bm{\\theta}^{\\text{old}}) + \\lambda \\left(\\sum_{k=1}^{K} \\pi_k - 1\\right) + \\sum_{j=1}^{K} \\lambda_j \\left(\\sum_{k=1}^{K} A_{jk} - 1\\right)$$\n\t\tPerforming the maximization, we get:\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\pi_k^{\\text{new}} & =\\frac{\\gamma(z_{1k})}{\\sum_{j=1}^{K}\\gamma(z_{1j})}, \\hspace{5mm} A_{jk} = \\frac{\\sum_{n=2}^{N} \\zeta(z_{n-1,j}, z_{nk})}{\\sum_{l=1}^{K}\\sum_{n=2}^{N} \\zeta(z_{n-1,j}, z_{nl})}\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tSolving the same for the parameter $\\bm{\\phi}$ depends on the form of emission probability that was chosen. For example, if we have a Gaussian density $p(\\bm{x}|\\bm{\\phi}_k)$, the optimized parameters are:\n\t\t$$\\bm{\\mu}_k=\\frac{\\sum_{n=1}^{N}\\gamma(z_{nk})\\bm{x}_n}{\\sum_{n=1}^{N}\\gamma(z_{nk})}, \\hspace{5mm} \\bm{\\Sigma}_k = \\frac{\\sum_{n=1}^{N} \\gamma(z_{nk})(\\bm{x}_n - \\bm{\\mu}_{k})(\\bm{x}_n - \\bm{\\mu}_{k})^T}{\\sum_{n=1}^{N} \\gamma(z_{nk})}$$\n\t\tSimilarly, if we would have discrete observations and model it with a multinomial distribution, i.e. $p(\\bm{x}|\\bm{z},\\bm{\\mu}) = \\prod_{i=1}^{D}\\prod_{k=1}^{K} \\mu_{ik}^{x_i z_k}$, we would get as solution:\n\t\t$$\\mu_{ik} = \\frac{\\sum_{n=1}^{N}\\gamma(z_{nk})x_{ni}}{\\sum_{n=1}^{N}\\gamma(z_{nk})}$$\n\t\\end{description}\n\t\n\t\\item To conclude, the EM algorithm for Hidden Markov Models can be summarized as follows:\n\t\\begin{tcolorbox}[colback=white!85!gray,colframe=gray!75!black,title=EM for HMM]\n\t\t\\begin{enumerate}\n\t\t\t\\item Choose initial values $\\bm{\\theta}^{\\text{old}}$ with $\\bm{\\theta}=(\\bm{\\pi}, \\bm{A}, \\bm{\\phi})$\n\t\t\t\\item Iterate until $\\Delta \\bm{\\theta}^{(t)} < \\epsilon$\n\t\t\t\\begin{enumerate}\n\t\t\t\t\\item \\textbf{E-step}: Calculate $Q(\\bm{\\theta}, \\bm{\\theta}^{\\text{old}})$ by:\n\t\t\t\t\\begin{enumerate}\n\t\t\t\t\t\\item Run forward $\\alpha$ recursion to calculate $\\alpha(z_1),...,\\alpha(z_N)$\n\t\t\t\t\t\\item Run backward $\\beta$ recursion to calculate $\\beta(z_N),...,\\beta(z_1)$\n\t\t\t\t\t\\item Calculate sufficient statistics $\\gamma(z_n)$, $\\zeta(z_{n-1},z_n)$ for $n=1,..,N$, and normalization constant $p(\\bm{X}|\\bm{\\theta}^{\\text{old}})$\n\t\t\t\t\\end{enumerate}\n\t\t\t\t\\item \\textbf{M-step}: Calculate $\\bm{\\theta}^{\\text{new}}=\\arg\\max_{\\bm{\\theta}} \\tilde{Q}(\\bm{\\theta}, \\bm{\\theta}^{\\text{old}})$\n\t\t\t\t\\begin{itemize}\n\t\t\t\t\t\\item $\\pi_k^{\\text{new}}=\\gamma(z_{1k})/\\sum_{k=1}^{K}\\gamma(z_{1j})$\n\t\t\t\t\t\\item $A_{jk}=\\sum_{n=2}^{N} \\zeta(z_{n-1,j}, z_{nk})/\\sum_{l=1}^{K}\\sum_{n=2}^{N} \\zeta(z_{n-1,j}, z_{nl})$\n\t\t\t\t\t\\item $\\bm{\\phi}^{\\text{new}}$ depending on choice of emission probability\n\t\t\t\t\\end{itemize}\n\t\t\t\\end{enumerate}\n\t\t\\end{enumerate}\n\t\\end{tcolorbox}\n\\end{itemize}\n\\subsubsection{Viterbi Algorithm (Max-sum for HMMs)}\n\\begin{itemize}\n\t\\item Sometimes we might be interested in the latent variables $\\bm{Z}$ for a given fixed model as they can be interpreted, and thus we want to determine the most likely values\n\t\\item For this, we can use an adaptation of the max-sum algorithm where we use dynamic programming for the backward path\n\t\\item We use the same factor graph as in Figure~\\ref{fig:HMM_factor_graph}. For simplification, we denote the message from $f_n$ (or $h_1$ for $n=1$) to $\\bm{z}_n$ as $\\omega(\\bm{z}_n)$.\n\t\\item The whole algorithm can be then summarized as:\n\t\\begin{tcolorbox}[colback=white!85!gray,colframe=gray!75!black,title=Pseudocode of Viterbi algorithm]\n\t\t\\begin{algorithm}[H]\n\t\t\t\\SetAlgoLined\n\t\t\t\\tcp{\\textcolor{blue}{Forward pass}}\n\t\t\tSet initial message $\\omega(\\bm{z}_1) = \\ln p(\\bm{z}_1) + \\ln p(\\bm{x}_1|\\bm{z}_1)$\\;\n\t\t\t\\For{$n=1,...,N-1$}{\n\t\t\t\t$\\omega(\\bm{z}_{n+1})=\\ln p(\\bm{x}_{n+1}|\\bm{z}_{n+1}) + \\max_{\\bm{z}_n}\\left(\\ln p(\\bm{z}_{n+1}|\\bm{z}_n) + \\omega(\\bm{z}_n)\\right)$\\;\n\t\t\t\t$\\psi_n(\\bm{z}_{n+1})=\\arg\\max_{\\bm{z}_n}\\left(\\ln p(\\bm{z}_{n+1}|\\bm{z}_n) + \\omega(\\bm{z}_n)\\right)$\n\t\t\t}\n\t\t\t\\tcp{\\textcolor{blue}{Backward pass}}\n\t\t\t$\\bm{z}_N^{\\text{max}}=\\arg\\max_{\\bm{z}_N} \\omega(\\bm{z}_N)$\\;\n\t\t\t\\For{$n=N-1,...,1$}{\n\t\t\t\t$\\bm{z}_n^{\\text{max}} = \\psi_{n}(\\bm{z}_{n+1}^{\\text{max}})$\\;\n\t\t\t}\n\t\t\tReturn $\\bm{z}_1^{\\text{max}}, \\bm{z}_2^{\\text{max}},...,\\bm{z}_N^{\\text{max}}$\\;\n\t\t\\end{algorithm}\n\t\\end{tcolorbox}\n\t\\item Coming back to the discussion about the max-sum algorithm on trees, we have said that for multiple optima, we need to use the Viterbi algorithm. Assume that for any $n$, we have two solutions for $\\bm{z}_n^{\\text{max}} = \\psi_{n}(\\bm{z}_{n+1}^{\\text{max}})$ (i.e. we have two values for $z_n$ that maximize our objective). If we are just interested in one, arbitrary solution, we get pick any of them and continue. Otherwise, we select one of the optimums, continue the Viterbi algorithm, and afterwards go back to this step, and select the next optimum. In the end, we can combine all solutions to return a full set of all optima. \n\t\n\tWe prevent the problem of returning undesirable combinations of independent optima by conditioning them on each other (i.e. $\\bm{z}_n^{\\text{max}}$ depends on $\\bm{z}_{n+1}^{\\text{max}}$).\n\\end{itemize}\n\\subsection{Linear Dynamical Systems}\n\\begin{itemize}\n\t\\item As mentioned before, Linear Dynamical Systems use a continuous latent space $\\bm{z}_n\\in \\mathbb{R}^{d_z}$ instead of a discrete as in HMM. This also influences the models we take because, as we will see later, specific distribution properties help to simplify the calculations\n\t\\item One popular model is a \\textit{Linear-Gaussian}: all conditional distributions in the Bayesian network are Gaussians with means that depend linearly on its parents. This leads to the probabilities:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\text{Transition probability}\\hspace{2mm} p(\\bm{z}_n|\\bm{z}_{n-1}) & = \\mathcal{N}(\\bm{z}_n|\\bm{A}\\bm{z}_{n-1}, \\bm{\\Gamma})\\\\\n\t\t\t\\text{Emission probability}\\hspace{2mm} p(\\bm{x}_n|\\bm{z}_n) & = \\mathcal{N}(\\bm{x}_n|\\bm{C}\\bm{z}_n, \\bm{\\Sigma})\\\\\n\t\t\t\\text{Initial state}\\hspace{2mm} p(\\bm{z}_1)  & = \\mathcal{N}(\\bm{z}_1|\\bm{\\mu}_0, \\bm{V}_0)\n\t\t\\end{split}\n\t\\end{equation*}\n\twith parameters $\\bm{\\theta}=(\\bm{A}, \\bm{\\Gamma}, \\bm{C}, \\bm{\\Sigma}, \\bm{\\mu}_0, \\bm{V}_0)$ where $\\bm{\\Sigma}$ is the observation noise, and $\\bm{\\Gamma}$ the transition uncertainty.\n\t\\item We first consider inference in LDS which represents the E-step, and then complete the learning process with the M-step in the second subsection\n\\end{itemize}\n\\subsubsection{Inference in Linear Dynamical Systems}\n\\begin{itemize}\n\t\\item To find the marginal distributions for the latent variables, we use again message passing. The forward equations are denoted with the normalized marginal distributions $\\widehat{\\alpha}(\\bm{z}_n)$:\n\t$$p(\\bm{z}_n|\\bm{x}_1,...,\\bm{x}_n) = \\widehat{\\alpha}(\\bm{z}_n) = \\mathcal{N}(\\bm{z}_n|\\bm{\\mu}_{n}, \\bm{V}_n)$$\n\t\\item Similarly to the HMM, our forward propagation takes the form:\n\t$$c_n \\widehat{\\alpha}(\\bm{z}_n) = p(\\bm{x}_n|\\bm{z}_n) \\int \\widehat{\\alpha}(\\bm{z}_{n-1})p(\\bm{z}_n|\\bm{z}_{n-1})d\\bm{z}_{n-1}$$\n\twhere we now have an integral instead of the sum, and $c_n$ is a constant making sure of the right scale as $\\widehat{\\alpha}(\\bm{z}_n)$ is normalized.\n\t\\item Plugging in the Gaussian distributions, we get:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tc_n \\mathcal{N}(\\bm{z}_n|\\bm{\\mu}_n, \\bm{V}_n) &  = \\mathcal{N}(\\bm{x}_n|\\bm{C}\\bm{z}_n, \\bm{\\Sigma})\\int \\mathcal{N}(\\bm{z}_{n-1}|\\bm{\\mu}_{n-1}, \\bm{V}_{n-1})\\mathcal{N}(\\bm{z}_n|\\bm{A}\\bm{z}_{n-1}, \\bm{\\Sigma})d\\bm{z}_{n-1}\\\\\n\t\t\t& = \\mathcal{N}(\\bm{x}_n|\\bm{C}\\bm{z}_n, \\bm{\\Sigma}) \\mathcal{N}(\\bm{z}_n|\\bm{A}\\bm{\\mu}_{n-1}, \\underbrace{\\bm{\\Gamma}+\\bm{A}\\bm{V}_{n-1}\\bm{A}^T}_{\\bm{P}_{n-1}})\n\t\t\\end{split}\n\t\\end{equation*}\n\tHere we see the first point where the chosen distributions can make a difference. The integral can be calculated due to the Gaussian, and we can get $\\bm{\\mu}_n$ and $\\bm{V}_n$ by applying more mathematical tricks with Gaussians and matrices (which we will not detail here, but can be found in the lecture notes). The end result is:\n\t$$\\bm{\\mu}_N = \\bm{A}\\bm{\\mu}_{n-1} + \\bm{K}_n (\\bm{x}_n - \\bm{C}\\bm{A}\\bm{\\mu}_{n-1}), \\hspace{4mm} \\bm{V}_n=(\\bm{I} - \\bm{K}_n\\bm{C})\\bm{P}_{n-1}$$\n\tThe important thing here is that without observation, $\\bm{\\mu}_n$ would be simply $\\bm{\\mu}_{n-1}$ moved/shifted by $A$, but the second term corrects it for the observation.\n\t\\item For the EM algorithm, in the backward pass, we get our sufficient statistics:\n\t$$\\gamma(\\bm{z}_n)=\\mathcal{N}(\\bm{z}_n|\\hat{\\bm{\\mu}}_n, \\hat{\\bm{V}}_n),\\hspace{5mm}\\zeta(\\bm{z}_{n-1},\\bm{z}_n)=\\mathcal{N}\\left(\\begin{bmatrix}\n\t\\hat{\\bm{\\mu}}_{n-1} \\\\ \\hat{\\bm{\\mu}}_{n}\n\t\\end{bmatrix}, \\begin{bmatrix}\n\t\\hat{\\bm{V}}_{n-1} & \\bm{J}_{n-1}\\hat{\\bm{V}}_{n}\\\\ \\hat{\\bm{V}}_{n}\\bm{J}_{n-1}^T& \\hat{\\bm{V}}_{n}\n\t\\end{bmatrix}\\right)$$\n\\end{itemize}\n\\subsubsection{Learning in LDS using EM}\n\\begin{itemize}\n\t\\item Above we have seen the results of the E-step in Linear Dynamical Systems. Now we take a closer look at the M-step, where we want to optimize for the parameters $\\bm{\\theta}=(\\bm{A}, \\bm{\\Gamma}, \\bm{C}, \\bm{\\Sigma}, \\bm{\\mu}_0, \\bm{V}_0)$\n\t\\item The complete data likelihood which we want to optimize in expectation to the posterior, is:\n\t$$\\ln p(\\bm{X},\\bm{Z}|\\bm{\\theta}) = \\ln p(\\bm{z}_1|\\bm{\\mu}_0, \\bm{V}_0) + \\sum_{n=2}^{N} \\ln p(\\bm{z}_n|\\bm{z}_{n-1}, \\bm{A}, \\bm{\\Gamma}) + \\sum_{n=1}^{N} \\ln p(\\bm{x}_n|\\bm{z}_n,\\bm{C},\\bm{\\Sigma})$$ \n\t\\item If we now e.g. want to optimize for $\\bm{\\mu}_0$ and $\\bm{V}_0$, we need to derive $Q(\\bm{\\theta}, \\bm{\\theta}^{\\text{old}})$ with respect to these variables. The solution for those is:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\bm{\\mu}_0, \\bm{V}_0: Q(\\bm{\\theta}, \\bm{\\theta}^{\\text{old}}) & = -\\frac{1}{2}\\ln \\left|\\bm{V}_0\\right| - \\frac{1}{2}\\E_{\\bm{z}_1|\\bm{\\theta}^{\\text{old}}}\\left[\\left(\\bm{z}_1 - \\bm{\\mu}_0\\right)^T \\bm{V}_0^{-1}(\\bm{z}_1 - \\bm{\\mu}_0)\\right] + \\text{const}\\\\\n\t\t\t\\bm{\\mu}_0^{\\text{new}} & = \\E[\\bm{z}_1|\\bm{\\theta}^{\\text{old}}]\\\\\n\t\t\t\\bm{V}_0^{\\text{new}} & = \\E[\\bm{z}_1\\bm{z}_1^T|\\bm{\\theta}^{\\text{old}}] - \\E[\\bm{z}_1|\\bm{\\theta}^{\\text{old}}]\\E[\\bm{z}_1|\\bm{\\theta}^{\\text{old}}]^T\n\t\t\\end{split}\n\t\\end{equation*}\n\\end{itemize}"
  },
  {
    "path": "Machine_Learning_2/ml2_summary.tex",
    "content": "\\documentclass[a4paper]{article} \n\\addtolength{\\hoffset}{-2.25cm}\n\\addtolength{\\textwidth}{4.5cm}\n\\addtolength{\\voffset}{-3.25cm}\n\\addtolength{\\textheight}{5cm}\n\\setlength{\\parskip}{0pt}\n\\setlength{\\parindent}{0in}\n\n\\usepackage{blindtext} % Package to generate dummy text\n\\usepackage{charter} % Use the Charter font\n\\usepackage[utf8]{inputenc} % Use UTF-8 encoding\n\\usepackage{microtype} % Slightly tweak font spacing for aesthetics\n\\usepackage[english]{babel} % Language hyphenation and typographical rules\n\\usepackage{amsthm, amsmath, amssymb, amsfonts, nccmath} % Mathematical typesetting\n\\usepackage{float} % Improved interface for floating objects\n\\usepackage[final, colorlinks = true, \nlinkcolor = black, \ncitecolor = black]{hyperref} % For hyperlinks in the PDF\n\\usepackage{graphicx, multicol} % Enhanced support for graphics\n\\usepackage{xcolor} % Driver-independent color extensions\n\\usepackage{marvosym, wasysym} % More symbols\n\\usepackage{rotating} % Rotation tools\n\\usepackage{subcaption}\n\\usepackage{wrapfig}\n% \\usepackage{geometry}\n\\usepackage{censor} % Facilities for controlling restricted text\n\\newcommand{\\note}[1]{\\marginpar{\\scriptsize \\textcolor{red}{#1}}} % Enables comments in red on margin\n\\usepackage{bm}\n\\usepackage{blkarray}\n\\usepackage{enumitem}\n\\usepackage{pgfplots}\n\\usepackage{tikz}\n\\usetikzlibrary{bayesnet}\n\n\\usepackage{tcolorbox}\n\\usepackage[ruled,vlined]{algorithm2e}\n\n\\newcommand{\\pd}[2]{\\frac{\\partial #1}{\\partial #2}}\n\\newcommand{\\loss}[0]{\\mathcal{L}}\n\\newcommand{\\chain}[3]{\\frac{\\partial #1}{\\partial #2}\\frac{\\partial #2}{\\partial #3}}\n% \\newcommand{\\eq}[1]{\\begin{equation*}\\begin{split}#1\\end{split}\\end{equation*}}\n\\newcommand{\\TODO}[1]{\\textbf{\\textcolor{red}{#1}}}\n\\newcommand{\\E}[0]{\\mathbb{E}} % Expectation\n\\newcommand{\\R}[0]{\\mathbb{R}} % Real numbers\n\\newcommand{\\Cdo}[0]{\\textnormal{do}}\n\\newcommand\\independent{\\protect\\mathpalette{\\protect\\independenT}{\\perp}}\n\\def\\independenT#1#2{\\mathrel{\\rlap{$#1#2$}\\mkern2mu{#1#2}}}\n\\newcommand*{\\QED}{\\hfill\\ensuremath{\\blacksquare}}%\n\n\\definecolor{green}{RGB}{0,160,0}\n\\definecolor{blue}{RGB}{0,0,160}\n\\definecolor{red}{RGB}{160,0,0}\n\\definecolor{orange}{RGB}{200,160,0}\n\\definecolor{purple}{RGB}{170,0,200}\n\\definecolor{cyan}{RGB}{0,200,200}\n\\definecolor{lightred}{RGB}{200,50,50}\n\n\\setcounter{tocdepth}{2}\n% Title Page\n\\title{Summary Machine Learning 2}\n\\author{Phillip Lippe}\n\n\n\\begin{document}\n\\maketitle\n\\tableofcontents\n\\newpage\n\n\\input{ml2_exponential_family.tex}\n\\newpage\n\\input{ml2_graphical_models.tex}\n\\newpage\n\\input{ml2_variational_EM.tex}\n\\newpage\n\\input{ml2_sampling_methods.tex}\n\\newpage\n\\input{ml2_sequential_data.tex}\n\\newpage \n\\input{ml2_causality.tex}\n\\appendix\n\\newpage\n\\input{ml2_appendix.tex}\n\n\\end{document}"
  },
  {
    "path": "Machine_Learning_2/ml2_variational_EM.tex",
    "content": "\\section{Variational Expectation Maximization}\n\n\\begin{itemize}\n\t\\item The expectation maximization algorithm can be viewed from a different angle where we focus on the latent variables $z_n$\n\t\\item For each observed data point $\\bm{x}_n$, we create a latent variable $\\bm{z}_n$ which \\textit{explains} the observation by our underlying model\n\t\\begin{itemize}\n\t\t\\item For example, in case we have a mixture model, the latent variable $z_n$ indicates from which component $\\bm{x}_n$ was created\n\t\t\\item We create the model in such a way that $p(X,Z|\\theta)$ can be easily calculated\n\t\\end{itemize}\n\t\\item As a graphical model, we can represent it as follows:\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\tikz{ %\n\t\t\t\\node[obs] (x) {$\\bm{x}_n$} ; %\n\t\t\t\\node[latent, above=of x] (z) {$\\bm{z}_n$} ; %\n\t\t\t\n\t\t\t\\edge{z}{x};\n\t\t\t\n\t\t\t\\plate{xz}{(x)(z)}{$n=1,...,N$};\n\t\t\t\\node[const, left=of x] (theta) {$\\bm{\\theta}$};\n\t\t\t\\edge{theta}{x};\n\t\t}\n\t\\end{figure}\n\t\\item The objective we want to optimize is the log-likelihood $$\\ln p(\\bm{X}|\\bm{\\theta}) = \\ln \\left\\{\\sum_{\\bm{Z}}p(\\bm{X}, \\bm{Z}|\\bm{\\theta})\\right\\}$$\n\t\\item However, note that in the standard setting, we are not given $\\bm{Z}$ so that we need to work with the posterior $p(\\bm{Z}|\\bm{X},\\bm{\\theta})$ as knowledge over latent variables. Hence, we calculate the log-likelihood under expectation of our posterior:\n\t$$\\E_{\\bm{Z}\\sim p(\\bm{Z}|\\bm{X},\\bm{\\theta})}\\left[\\ln p(\\bm{X}, \\bm{Z}|\\bm{\\theta})\\right] = \\sum_{\\bm{Z}} p(\\bm{Z}|\\bm{X},\\bm{\\theta}) \\ln p(\\bm{X}, \\bm{Z}|\\bm{\\theta})$$\n\t\\item As the optimization problem for the posterior and the parameters has often no analytical closed-form solution, we optimize both sequentially, leading to the general idea of the EM algorithm\n\t\\begin{description}\n\t\t\\item[E-step] Find the posterior distribution $p(\\bm{Z}|\\bm{X},\\bm{\\theta}^{\\text{old}})$ where $\\bm{\\theta}^{\\text{old}}$ means that we fix the other parameters\n\t\t\\item[M-step] Optimize the log-likelihood with respect to parameters $\\bm{\\theta}$ while keeping the posterior fixed\n\t\t$$\\bm{\\theta}^{\\text{new}} = \\arg\\max_{\\bm{\\theta}} \\mathcal{Q}(\\bm{\\theta}, \\bm{\\theta}^{\\text{old}}) = \\arg\\max_{\\bm{\\theta}} \\sum_{\\bm{Z}} p(\\bm{Z}|\\bm{X},\\bm{\\theta}^{\\text{old}}) \\ln p(\\bm{X}, \\bm{Z}|\\bm{\\theta})$$\n\t\\end{description}\n\t\\item In case we want to find the MAP instead of the MLE, we simply have to add the prior term $\\ln p(\\bm{\\theta})$ to $\\mathcal{Q}(\\bm{\\theta}, \\bm{\\theta}^{\\text{old}})$ in the M-step \n\\end{itemize}\n\\subsection{Generalizing EM}\n\\begin{itemize}\n\t\\item We can further generalize the EM algorithm to a form, which was originally used to prove its optimization objective. \n\t\\item However, we can also look at it from a different perspective. It might be sometimes the case, that the posterior $p(\\bm{Z}|\\bm{X},\\bm{\\theta}^{\\text{old}})$ is hard to determine. Instead, we can introduce an approximation $q(\\bm{Z})$ for which we can choose the form ourselves (e.g. Gaussian, or softmax distribution over discrete states, etc.)\n\t\\item Using this approximation, we can calculate the log likelihood by:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\ln p(\\bm{X}|\\bm{\\theta}) & = \\sum_{n=1}^{N} \\sum_{\\bm{z}_n} q(\\bm{z}_n)\\ln \\frac{p(\\bm{x}_n, \\bm{z}_n|\\bm{\\theta})}{p(\\bm{z}_n|\\bm{x}_n, \\bm{\\theta})}\\\\\n\t\t\t & = \\sum_{n=1}^{N} \\sum_{\\bm{z}_n} q(\\bm{z}_n)\\ln \\frac{p(\\bm{x}_n, \\bm{z}_n|\\bm{\\theta})}{q(\\bm{z}_n)}\\frac{q(\\bm{z}_n)}{p(\\bm{z}_n|\\bm{x}_n, \\bm{\\theta})}\\\\\n\t\t\t & = \\sum_{n=1}^{N} \\left[\\E_{q_n}\\left[\\log p(\\bm{x}_n, \\bm{z}_n|\\bm{\\theta})\\right] + H(q_n) + \\underbrace{\\text{KL}\\left(q_n(\\bm{z}_n)||p(\\bm{z}_n|\\bm{x}_n, \\bm{\\theta})\\right)}_{\\geq 0}\\right]\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item We can define an ELBO for our objective function:\n\t$$\\ln p(\\bm{X}|\\bm{\\theta}) = \\mathcal{L}(\\bm{\\theta}) \\geq  \\sum_{n=1}^{N} \\left[\\E_{q_n}\\left[\\log p(\\bm{x}_n, \\bm{z}_n|\\bm{\\theta})\\right] + H(q_n)\\right] = \\mathcal{L}(\\bm{\\theta}, q)$$\n\t\\item Hence, for any set of distributions $\\left\\{q_n(\\bm{z}_n)\\right\\}_{n=1}^{N}$, we can define a lower bound optimization objective $\\mathcal{L}(\\bm{\\theta}, q)$, which is equal to $\\mathcal{L}(\\bm{\\theta})$ iff $q_n(\\bm{z}_n)=p(\\bm{z}_n\\vert \\bm{x}_n, \\bm{\\theta})$\n\t\\item During the E-step, we optimize $\\mathcal{L}(\\bm{\\theta}, q)=\\ln p(\\bm{X}|\\bm{\\theta})-\\text{KL}(q||p)$ regarding $q$. As $\\ln p(\\bm{X}|\\bm{\\theta})$ is independent of $q$, we minimize the KL-divergence, meaning that we push $q_n(\\bm{z}_n)$ to be similar to $p(\\bm{z}_n\\vert \\bm{x}_n, \\bm{\\theta})$\n\t\\item In the M-step, we increase $p(\\bm{X}|\\bm{\\theta})$ by optimizing the parameters $\\bm{\\theta}$. Note that as $q$ is fixed in this step, the KL-divergence will increase as well, due to $q$ being now sub-optimal for the new parameters. Hence, we perform another E-step again a.s.o.\n\t\\item Keep in mind that when optimizing $\\mathcal{L}(\\bm{\\theta}, q)$, we need to add Lagrangian for the constraint that $q_n$ is a valid PDF:\n\t$$\\tilde{\\mathcal{L}}(\\bm{\\theta}, q) = \\mathcal{L}(\\bm{\\theta}, q) + \\sum_{n=1}^{N} \\lambda_n \\left(\\sum_{\\bm{z}_n} q_n(\\bm{z}_n) - 1\\right)$$\n\tDepending on our model, we might need to add additional Lagrangian to $\\tilde{\\mathcal{L}}(\\bm{\\theta}, q)$ (e.g. over $\\bm{\\pi}$ in mixture models)\n\t\\item In summary, the variational EM algorithm can be summarized as follows:\n\t\n\t\\begin{tcolorbox}[colback=white!85!gray,colframe=gray!75!black,title=Variational EM algorithm]\n\t\t\\begin{enumerate}\n\t\t\t\\item Choose initial $\\bm{\\theta}^{(0)}$\n\t\t\t\\item Iterate until $\\Delta \\bm{\\theta}^{(t)} < \\epsilon$\n\t\t\t\\begin{enumerate}\n\t\t\t\t\\item \\textbf{E-step}: Given \\underline{fixed} $\\bm{\\theta}^{(t)}$,\n\t\t\t\t\\begin{itemize}\n\t\t\t\t\t\\item If the posterior can be determined, evaluate $q_n^{(t)}(\\bm{z}_n) = p(\\bm{z}_n|\\bm{x}_n, \\bm{\\theta}^{(t)})$\n\t\t\t\t\t\\item Otherwise, use Variational EM by increasing $\\tilde{\\mathcal{L}}(\\bm{\\theta}^{(t)}, q)$ over $q$, e.g. gradient ascend\n\t\t\t\t\\end{itemize}\n\t\t\t\t\\item \\textbf{M-step}: Given \\underline{fixed} $q^{(t)}$,\n\t\t\t\t\\begin{itemize}\n\t\t\t\t\t\\item Solve, if possible, $\\bm{\\theta}^{(t+1)} = \\arg\\max_{\\bm{\\theta}} \\tilde{\\mathcal{L}}(\\bm{\\theta}, q^{(t)})$ % \\sum_{n=1}^{N} \\E_{q_n}\\left[\\log p(\\bm{x}_n, \\bm{z}_n|\\bm{\\theta})\\right]\n\t\t\t\t\t\\item Otherwise, increase $\\tilde{\\mathcal{L}}(\\bm{\\theta}, q^{(t)})$ over $\\bm{\\theta}$, e.g. gradient ascend\n\t\t\t\t\\end{itemize}\n\t\t\t\\end{enumerate}\n\t\t\\end{enumerate}\n\t\\end{tcolorbox}\n\\end{itemize}\n\\subsubsection{Example: Mixture of multivariate Bernoulli's}\n\\begin{itemize}\n\t\\item As an example, we outline the EM algorithm to optimize a mixture of multivariate Bernoulli's here. Our model distribution looks like:\n\t$$p(\\bm{x}_n|\\bm{\\mu}, \\bm{\\pi}) = \\sum_{k=1}^{K} \\pi_k \\prod_{i=1}^{D} \\mu_{ki}^{x_{ni}} (1- \\mu_{ki})^{1-x_{ni}} \\hspace{4mm}\\text{where}\\hspace{2mm} \\sum_{k=1}^{K}\\pi_k = 1$$\n\t\\item Optimizing the parameters without the EM algorithm does not has a closed-form solution because of a sum in the log:\n\t$$\\log p(\\bm{X}|\\bm{\\mu},\\bm{\\pi}) = \\sum_{n=1}^{N} \\log \\sum_{z_n=1}^{K} \\pi_{z_n} p(\\bm{x}_n|\\bm{\\mu_{z_n}})$$\n\t\\item Our EM objective is instead:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\mathcal{L}(q,\\bm{\\mu}, \\bm{\\pi}) & = \\sum_{n=1}^{N} \\sum_{\\bm{z}_n} q_n(\\bm{z}_n) \\left\\{ \\left(\\log \\pi_{z_n} + \\sum_{i=1}^{D}\\left[x_{ni}\\log \\mu_{z_n,i} + (1 - x_{ni})\\log (1 - \\mu_{z_n,i})\\right]\\right)- \\log q_n(\\bm{z}_n)\\right\\}\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\tilde{\\mathcal{L}}(q,\\bm{\\mu}, \\bm{\\pi}, \\lambda, \\left\\{\\lambda_n\\right\\}) & = \\mathcal{L}(q,\\bm{\\mu}, \\bm{\\pi}) + \\lambda \\left(\\sum_{k=1}^{K} \\pi_k - 1\\right) + \\sum_{n=1}^{N} \\lambda_n \\left(\\sum_{\\bm{z}_n} q_n(\\bm{z}_n) - 1\\right)\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item The update equation we get by deriving $\\tilde{\\mathcal{L}}(q,\\bm{\\mu}, \\bm{\\pi}, \\lambda, \\left\\{\\lambda_n\\right\\})$ with respect to the according parameters\n\t\\begin{description}\n\t\t\\item[E-Step] Optimize $q$\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\frac{\\partial \\tilde{\\mathcal{L}}}{\\partial q_n(z_n)} & = \\log \\pi_{z_n} + \\left[\\sum_{i=1}^{D} x_{ni}\\log \\mu_{z_n,i}+(1-x_{ni})\\log (1-\\mu_{z_n,i})\\right] - \\log q_n(z_n) - 1 + \\lambda_n = 0\\\\\n\t\t\t\t\\implies q_n(z_n) & = \\exp(\\lambda_n-1)\\pi_{z_n}\\prod_{i=1}^{D} \\mu_{z_n,i}^{x_{ni}}(1 - \\mu_{z_n,i})^{1-x_{ni}}\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tBy solving for the Lagrangian $\\lambda_n$, we would see that $\\exp(\\lambda_n-1)=1/(\\sum_{z_n}q(z_n))$, and hence, being a normalization factor.\n\t\t\\item[M-step] Optimize parameters $\\bm{\\pi}$:\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\frac{\\partial \\tilde{\\mathcal{L}}}{\\partial \\pi_k} & = \\sum_{n=1}^{N} \\sum_{\\bm{z}_n }  q_n(\\bm{z}_n)\\frac{z_{nk}}{\\pi_k}  + \\lambda \\overset{!}{=} 0\\\\\n\t\t\t\t\\Leftrightarrow \\pi_k & = \\frac{1}{\\lambda} \\sum_{n=1}^{N} \\sum_{\\bm{z}_n}  z_{nk} q_n(\\bm{z}_n)\\\\\n\t\t\t\t\\frac{\\partial \\tilde{\\mathcal{L}}}{\\partial \\lambda} & = \\sum_{k=1}^{K} \\pi_k - 1 \\overset{!}{=} 0\\\\\n\t\t\t\t\\Leftrightarrow 1 & = \\sum_{k=1}^{K} \\frac{1}{\\lambda} \\sum_{n=1}^{N} \\sum_{\\bm{z}_n} z_{nk} q_n(\\bm{z}_n) \\\\ \n\t\t\t\t\\Leftrightarrow \\lambda & = \\sum_{k=1}^{K} \\sum_{n=1}^{N} \\sum_{\\bm{z}_n} z_{nk} q_n(\\bm{z}_n) = \\sum_{n=1}^{N} 1 = N\\\\\n\t\t\t\t\\implies \\pi_k & = \\frac{\\sum_{n=1}^{N} \\sum_{\\bm{z}_n} z_{nk} q_n(\\bm{z}_n)}{N}\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\tOptimize parameter $\\bm{\\mu}$:\n\t\t\\begin{equation*}\n\t\t\t\\begin{split}\n\t\t\t\t\\frac{\\partial \\tilde{\\mathcal{L}}}{\\partial \\mu_{ki}} & = \\sum_{n=1}^{N} q_n(k)\\left[\\frac{x_{ni}}{\\mu_{ki}} - \\frac{1-x_{ni}}{1-\\mu_{ki}}\\right]\\\\\n\t\t\t\t\\implies \\mu_{ki} & = \\frac{\\sum_{n=1}^{N} q_n(k)x_{ni}}{\\sum_{n=1}^{N} q_n(k)}\n\t\t\t\\end{split}\n\t\t\\end{equation*}\n\t\\end{description}\n\\end{itemize}\n\\subsection{Variational Inference: Variational Bayes}\n\\begin{itemize}\n\t\\item In standard EM, we treat $\\bm{\\theta}$ to be a parameter that we want to optimize with a single value. However, looking from a Bayesian perspective, we would also treat it as a (latent) random variable and calculate its posterior, leading to the following graphical model:\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\tikz{ %\n\t\t\t\\node[obs] (x) {$\\bm{x}_n$} ; %\n\t\t\t\\node[latent, left=of x] (theta) {$\\bm{\\theta}$} ; %\n\t\t\t\n\t\t\t\\edge{theta}{x};\n\t\t\t\n\t\t\t\\plate{xz}{(x)}{$n=1,...,N$};\n\t\t}\n\t\\end{figure}\n\t\\item Note that we could also include latent variables $\\bm{z}_n$ as in the EM algorithm, but for generality, we could simply include them in $\\bm{\\theta}$ as the calculations are exactly the same\n\t\\item We perform the same derivation as for variational EM, with the difference, that $q$ is now over all parameters $\\bm{\\theta}$, giving us the following objective:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tL=\\log p(X) & \\geq \\mathcal{L}(q)\\\\\n\t\t\t\\text{where}\\hspace{2mm}\\mathcal{L}(q) & = \\int q(\\bm{\\theta}) \\log \\left[p(X|\\bm{\\theta})p(\\bm{\\theta})\\right]d\\bm{\\theta} - \\int q(\\bm{\\theta})\\log q(\\bm{\\theta})d\\bm{\\theta}\\\\\n\t\t\t& = \\E_{q(\\bm{\\theta})}\\left[\\log \\left(p(X|\\bm{\\theta})p(\\bm{\\theta})\\right)\\right] + H(q)\n\t\t\\end{split}\n\t\\end{equation*}\n\tNote that we use integrals here as parameters are often continuous.\n\t\\item The posterior can be found by optimizing $q$:\n\t$$p(\\bm{\\theta}|\\bm{X}) = \\arg\\max_{q(\\bm{\\theta})} \\mathcal{L}(q)$$\n\t\\item In practice, we restrict $q(\\bm{\\theta})$ to be in a function family $Q$ for  which we can calculate the maximum of $\\mathcal{L}(q)$ analytically, or optimize by gradient ascend. \n\t\\item One example for $Q$ is to assume that all (or at least certain parts) of the parameters are independent of each other, hence:\n\t$$q(\\bm{\\theta})\\in Q=\\left\\{\\prod_{i=1}^{D}q_i(\\bm{\\theta}_i)\\right\\}$$\n\twhere $\\bm{\\theta}_i$ can be either a scalar or a vector.\n\t\\item For this case, we can simplify the objective:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\tilde{L}(q) & = \\int \\left(\\prod_{i=1}^{D} q_i(\\bm{\\theta}_i)\\right) \\log \\left[p(X|\\bm{\\theta})p(\\bm{\\theta})\\right]d\\bm{\\theta} - \\sum_{i=1}^{D}\\int q_i(\\bm{\\theta}_i)\\log q_i(\\bm{\\theta}_i)d\\bm{\\theta}_i + \\sum_{i=1}^{D} \\lambda_i \\left(\\int q_i(\\theta_i)d\\bm{\\theta}_i - 1\\right)\\\\\n\t\t\t\\frac{\\partial \\tilde{L}}{\\partial q_i(\\bm{\\theta}_i)} & = \\int \\left(\\prod_{j\\neq i} q_j(\\bm{\\theta}_j)\\right) \\log \\left[p(X|\\bm{\\theta})p(\\bm{\\theta})\\right]d\\bm{\\theta}_{\\setminus i} - \\log q_i(\\theta_i) - 1 + \\lambda_i\\\\\n\t\t\t\\Leftrightarrow q_i(\\bm{\\theta}_i) & = \\exp\\left(\\lambda_i - 1\\right) \\exp\\left\\{\\int \\left(\\prod_{i\\neq j} q_j(\\bm{\\theta}_j)\\right)\\log \\left(p(\\bm{X}|\\bm{\\theta})p(\\bm{\\theta})\\right) \\right\\}\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Hence, our approximated posterior over $\\bm{\\theta}_i$ is the expectation of $\\log p(X,\\bm{\\theta})$ over all other parameters:\n\t\\begin{equation*}\n\t\t\\tcbox[nobeforeafter]{\\(p(\\bm{\\theta}|\\bm{X})=\\prod_{i=1}^{D}q_i(\\bm{\\theta}_i), \\hspace{2mm}q_i(\\bm{\\theta}_i) = \\frac{1}{Z} \\exp\\left(\\E_{q_{\\setminus i}}\\left[\\log p(\\bm{X}, \\bm{\\theta})\\right]\\right)\\)}\n\t\\end{equation*}\n\tWe iterate this update for each $q$ until convergence.\n\t\\item Note that Variational Bayes is highly related to Gibbs sampling as there, we do not calculate the full probability distribution $q$, but simply alternate between sampling from the different $q_i$'s. By iterating until convergence/for a sufficient number of time, we also reach the true posteriors.\n\t\n\\end{itemize}\n\\subsubsection{Example: Gaussian with mean and variance as latent variables}\n\\begin{itemize}\n\t\\item We consider the following graphical model:\\\\\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\tikz{ %\n\t\t\t\\node[obs] (x) {$x_n$} ; %\n\t\t\t\\node[latent, above=of x] (mu) {$\\mu$} ; %\n\t\t\t\\node[latent, right=of mu] (tau) {$\\tau$} ; %\n\t\t\t\n\t\t\t\\edge{mu}{x};\n\t\t\t\\edge{tau}{x};\n\t\t\t\\edge{tau}{mu};\n\t\t\t\n\t\t\t\\plate{xz}{(x)}{$n=1,...,N$};\n\t\t}\n\t\\end{figure}\n\n\twhere\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tp(\\bm{X}|\\mu, \\tau) & = \\prod_{n=1}^{N} \\mathcal{N}(x_n|\\mu, \\tau^{-1})\\\\\n\t\t\tp(\\tau) & = \\text{Gamma}(\\tau|a_0, b_0)\\\\\n\t\t\tp(\\mu|\\tau) & = \\mathcal{N}(\\mu|\\mu_0, (\\lambda_0 \\tau)^{-1})\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Now, let's assume that we approximate the posteriors by $q(\\tau, \\mu)=q(\\tau)q(\\mu)$. Our objective (excluding the Lagrangian) is:\n\t$$\\mathcal{L}(q_{\\mu}, q_{\\tau}) = \\int q_{\\mu}(\\mu)q_{\\tau}(\\tau)\\left[\\log p(\\bm{X}|\\mu, \\tau)p(\\mu|\\tau)p(\\tau)\\right] + H(q_{\\mu}) + H(q_{\\tau})$$\n\t\\item When solving this, we will end up with:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tq_{\\mu}(\\mu) & = \\mathcal{N}\\left(\\mu\\Big\\vert \\frac{\\lambda_0\\mu_0 + N\\overline{x}}{\\lambda_0 + N}, \\left[(\\lambda_0 + N)\\E_{q_{\\tau}}[\\tau]\\right]^{-1}\\right)\\\\\n\t\t\tq_{\\tau}(\\tau) & = \\text{Gamma}\\left(\\tau\\Big\\vert a_0 + \\frac{N}{2}, b_0 + \\frac{1}{2}\\E_{q_{\\mu}}\\left[\\sum_n (x_n-\\mu)^2+\\lambda_0(\\mu-\\mu_0)^2\\right]\\right)\\\\\n\t\t\t\\Rightarrow p(\\mu, \\tau|\\bm{X}) &\\approx q_{\\mu}(\\mu)q_{\\tau}(\\tau)\n\t\t\\end{split}\n\t\\end{equation*}\n\\end{itemize}\n\\subsubsection{Combining Variational EM and Variational Bayes}\n\\begin{itemize}\n\t\\item The Variational EM algorithm and the variational Bayes are strongly related. In fact, we can generalize the framework to combine both of them\n\t\\item Our goal is to optimize the parameters $\\bm{\\theta}$, and marginalize over latent variables $\\bm{Z}$, leading to the objective:\n\t$$\\ln p(\\bm{X}) = \\E_{q(\\bm{Z}, \\bm{\\theta})}\\left[\\ln p(\\bm{X}, \\bm{Z}, \\bm{\\theta})\\right] + \\underbrace{H(q_Z) + H(q_{\\theta})}_{\\text{assume } q(\\bm{Z},\\bm{\\theta})=q_Z(\\bm{Z})q_{\\theta}(\\bm{\\theta})} + \\underbrace{\\text{KL}\\left(q_Z q_{\\theta} || p(\\bm{Z}, \\bm{\\theta}|\\bm{X})\\right)}_{\\geq 0}$$\n\t\\item Again, we can optimize an ELBO instead:\n\t$$\\ln p(\\bm{X}) \\geq \\E_{q_Z q_{\\theta}}\\left[\\ln p(\\bm{X}, \\bm{Z}, \\bm{\\theta})\\right] = \\mathcal{L}(q_Z, q_{\\theta})$$\n\t\\item Assuming $\\tilde{\\mathcal{L}}(q_Z, q_{\\theta})$ being the objective function including the Lagrangian, we have the following steps:\n\t\\begin{description}\n\t\t\\item[$\\E_{Z}$-step]: $\\max_{q_Z} \\tilde{\\mathcal{L}}(q_Z, q_{\\theta})$\n\t\t\\item[$\\E_{\\theta}$-step]: $\\max_{q_{\\theta}} \\tilde{\\mathcal{L}}(q_Z, q_{\\theta})$\n\t\\end{description}\n\t\\item As this is a generalization, we can find both the EM algorithm and variational Bayes in it. The EM algorithm is found if we choose $q_{\\theta}(\\theta)=\\delta(\\theta'-\\theta)$, where $\\theta'$ is optimized $\\Rightarrow$ equal to replacing $q_{\\theta}$ with MLE/MAP solution\n\t\\item For variational Bayes, we can simply ignore the $Z$ part as we fully focus on $q_{\\theta}$. The iteration over the $\\E_{\\theta}$-step is equal to variational Bayes\n\\end{itemize}\n\\subsection{Variational Auto-Encoder (VAE)}\n\\begin{itemize}\n\t\\item One implementation with neural networks of this variational framework are VAEs, where we model the likelihood by:\n\t$$p(\\bm{X}, \\bm{Z}|\\bm{\\theta}) = \\underbrace{p(\\bm{X}|\\bm{Z}, \\bm{\\theta}_2)}_{\\text{decoder}}\\underbrace{p(\\bm{Z}|\\bm{\\theta}_1)}_{\\text{encoder/prior}}$$\n\t\\item The aim is to find a lower-dimensional representation $Z$ of the data $X$ (\\textit{encode} $X$). For approximating the posterior, we use again $q(\\bm{Z}|\\bm{X},\\bm{\\lambda})\\approx p(\\bm{Z}|\\bm{X}, \\bm{\\theta}_1, \\bm{\\theta}_2)$, where $q$ is now a neural network\n\t\\item As a prior distribution, we use e.g. $p(\\bm{Z}|\\bm{\\theta}_1) \\sim \\mathcal{N}(0,1)$ which encourages the latents to be independent. Furthermore, by using this prior, we can treat VAEs as a non-linear version of PCA (in case we choose student-t distribution, we would get non-linear ICA)\n\t\\item Optimize the ELBO of the log-likelihood via SGD:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\mathcal{L}(\\bm{\\theta}) & = \\ln p(\\bm{X}|\\bm{\\theta}) = \\ln \\int p(\\bm{X}|\\bm{Z},\\bm{\\theta})p(\\bm{Z}|\\bm{\\theta})d\\bm{Z}\\\\\n\t\t\t\\mathcal{L}(\\bm{\\theta}, \\bm{\\lambda}) & = \\E_{q(Z|X,\\lambda)}\\left[\\ln p(\\bm{X}|\\bm{Z},\\bm{\\theta}_2) + \\ln p(\\bm{Z}|\\bm{\\theta}_1)\\right] + H(q(\\bm{Z}|\\bm{X}, \\bm{\\lambda}))\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item To prevent the integral, we can sample instead. To make these samples differentiable, we need to use the reparameterization trick:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tz^{(k)}\\sim q(z|x, \\lambda) & \\Rightarrow z=g_{\\lambda}(x, \\epsilon), \\epsilon\\sim p(\\epsilon)\n\t\t\\end{split}\n\t\\end{equation*}\n\tAs for a multivariate Gaussian with diagonal covariance, we have $g_{\\lambda}(\\bm{x}, \\bm{\\epsilon})=\\bm{\\mu}_{\\lambda}(x)+\\bm{\\sigma}_{\\lambda}\\odot \\bm{\\epsilon}, \\epsilon\\sim\\mathcal{N}(0,1)$\n\\end{itemize}"
  },
  {
    "path": "Natural_Language_Processing_1/nlp_bayesian.tex",
    "content": "% \\section{Foundations of Bayesian NLP}\n% \\textbf{Foundations of Bayesian NLP is not in the exam.}"
  },
  {
    "path": "Natural_Language_Processing_1/nlp_compositional_semantic.tex",
    "content": "\\section{Compositional semantics and discourse processing}\n% \\subsection{Compositional semantics}\n\\begin{itemize}\n\t\\item \\textbf{Principle of Compositionality}: meaning of whole phrase derivable from meaning of its parts\n\t\\item Sentence structure conveys some meaning as well\n\t\\begin{itemize}\n\t\t\\item Different syntactic structures may have the same meaning, but similar syntactic structures can also have different meanings\n\t\\end{itemize}\n\t\\item Not all phrases are interpreted compositionally (e.g. \\textit{kick the bucket}) but can be grouped together and viewed as one element\n\t\\item Meaning of a single word can depend on the composition (\\textit{fast} programmer vs. \\textit{fast} plane, metaphors,...)\n\\end{itemize}\n\\subsection{Compositional distributional semantics}\n\\begin{itemize}\n\t\\item Extending distributional semantics to phrases/sentences\n\t\\item Unsupervised model $\\Rightarrow$ general-purpose representations\n\t\\item Model composition in vector space. However, if we would model every sentence as independent, we would get an infinite dimensional space\n\\end{itemize}\n\\subsubsection{Vector mixture model}\n\\begin{itemize}\n\t\\item Combining the vectors of all words in the sentence\n\t\\item Mostly done either additive (adding all vector) or multiplicative (elementwise product of vectors)\n\t\\item Problem: does not consider word order and is therefore suitable for modelling content words (nouns, verbs, adjectives,...), but not for function words that require syntactic dependencies (pronouns, ...)\n\t\\item Is often used as baseline\n\\end{itemize}\n\\begin{figure}[ht]\n\t\\centering\n\t\\begin{subfigure}{0.3\\textwidth}\n\t\t\\includegraphics[width=\\textwidth]{figures/compositional_semantic_vector_mixture_model.png}\n\t\t\\caption{Vector mixture model}\n\t\\end{subfigure}\n\t\\begin{subfigure}{0.3\\textwidth}\n\t\t\\includegraphics[width=\\textwidth]{figures/compositional_semantics_lexical_function_models.png}\n\t\t\\caption{Lexical function model}\n\t\\end{subfigure}\n\t\\caption{Compositional distributional semantics}\n\t\\label{fig:compositional_semantic_vector_mixture_model}\n\\end{figure}\n\\subsubsection{Lexical function model}\n\\begin{itemize}\n\t\\item Discriminate between words that meaning is determined by its context/distribution (e.g. nouns), and function words that are applied on the represented words as \\textbf{lexical functions}\n\t\\item Example: $\\underbrace{\\textit{old}}_{\\text{functional}} \\underbrace{\\textit{dog}}_{\\text{distributional}} \\Rightarrow$ apply function of \\textit{old} on \\textit{dog}\n\t\\item Lexical functions are parameter matrices (i.e. $\\bm{A}_{\\textit{old}}$) which are multiplied with the vector representation of nouns\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/compositional_semantics_lexical_function_models_adjectives.png}\n\t\\end{figure}\n\t\\item Adjectives that does not change the meaning of a word are diagonal up to identity matrix $\\Rightarrow$ element captures how features interact with each other given the adjective\n\t\\item The matrices are learned by comparing the representation of plain nouns to combination of noun and adjective. Pseudo-code algorithm:\n\t\\begin{enumerate}\n\t\t\\item Obtain a distributional vector $n_j$ for each noun $n_j$ in the lexicon.\n\t\t\\item Collect adjective noun pairs $(a_i,n_j)$ from the corpus.\n\t\t\\item Obtain a distributional vector $p_{ij}$ of each pair $(a_i,n_j)$ from the same corpus using a conventional DSM.\n\t\t\\item The set of tuples $\\{(n_j,p_{ij})\\}_j$ represents a dataset $\\mathcal{D}(a_i)$ for the adjective $a_i$.\n\t\t\\item Learn matrix $\\bm{A}_i$ from $\\mathcal{D}(a_i)$ using linear regression by minimizing:\n\t\t$$L(\\bm{A}_i) = \\sum\\limits_{j\\in \\mathcal{D}(a_i)} ||p_{ij} - \\bm{A}_i n_j||^2$$\n\t\\end{enumerate} \n\t\\item Verbs can be represented as high-order tensors. If only subject is taken into consideration, it is a two-dimensional matrix. When also considering object, then it is three dimensional (or even higher)\n\t\\item \\textit{Polysemy} (different forms of a word) are mostly handled by a single representation. We assume that ambiguity can be handled as long as the context is given.\n\t\\item To identify \\textit{metaphors}, two separate senses of every adjective can be learned (literal and metaphorical). We then map from literal to metaphorical by a linear transformation. \n\\end{itemize}\n\\subsubsection{Compositional semantics in neural networks}\n\\begin{itemize}\n\t\\item Supervised learning framework $\\Rightarrow$ compositional representations are fine-tuned for specific application/task\n\t\\item Word representations are taken as input and processed within the network\n\t\\item Example tasks include sentiment classification, paraphrasing, machine translation, ...\n\t\\item Using recurrent and/or recursive networks (LSTMs, Tree-LSTMs, ...)\n\\end{itemize}\n\\subsection{Discourse structure}\n\\begin{itemize}\n\t\\item Most documents have a implicit (in news paper articles, first sentence is a summary) or explicit structure like sections and paragraphs\n\t\\item There are also relationships between sentences that need to be modeled as follow\n\\end{itemize}\n\\subsubsection{Rethorical relations}\n\\begin{itemize}\n\t\\item There are implicit relations between sentences. For example:\\\\\n\t\\texttt{Max fell. John pushed him.}\\\\\n\tcan be interpreted as \\textit{explanation} (Max fell because John pushed him), or as \\textit{narration} (Max fell and then John pushed him).\n\t\\item This relation is called \\textbf{discourse relation} or \\textbf{rhetorical relation}\n\t\\item \\textbf{Cue phrases} indicate what kind of relation it is. In the previous examples, the cue phrases were \\texttt{because} and \\texttt{and then}.\n\t\\item Analyzing a text for rhetorical relations mostly gives a binary structure: the main sentence is called \\textbf{nucleus}, and subsidiary phrase (explanation, justification, ...) is called \\textbf{satellite}\n\t\\item In a \\textit{narration} (cue phrase \\texttt{and}) both sentences have equal weight instead of nucleus vs satellite.\n\\end{itemize}\n\\subsubsection{Coherence}\n\\begin{itemize}\n\t\\item Discourses need to have connectivity/context to be coherent.\n\t\\item Otherwise, a sentence/small discourse might not make sense\n\t\\item However, this information is mostly missing (background/world knowledge)!\n\t\\item Assuming discourse coherence can affect interpretation. Especially when dealing with pronouns, th\n\\end{itemize}\n\\subsubsection{Overview of factors influencing discourse interpretation}\n\\begin{enumerate}\n\t\\item \\textit{Cue phrases} (\\texttt{because}, \\texttt{and}, ...)\n\t\\item \\textit{Punctuation and text structure} (\\texttt{Max fell (John pushed him), and Kim laughed.})\n\t\\item \\textit{Real world context} (\\texttt{Max was falling.} \\texttt{John pushed him as he lay on the ground.})\n\t\\item \\textit{Tense and aspects} (\\texttt{Max was falling.} \\texttt{John pushed him.})\n\\end{enumerate}\n\\begin{itemize}\n\t\\item Discourse parsing (understanding discourse structure) is a hard task\n\t\\item Mostly done by supervision (annotated data of about 8-10 discourses)\n\t\\item However, \\textit{surface techniques} (primitive algorithms that look at characteristic phrases, punctuation, ...) seem to work to some extent\n\\end{itemize}\n\\subsection{Referring expressions and anaphora}\n\\begin{itemize}\n\t\\item To fully process a discourse, co-references/referring expressions like pronouns need to be resolved\n\t\\item We can define the following entities for a referring expression:\n\t\\begin{itemize}\n\t\t\\item \\textit{referent} - a real world entity to which is referred\n\t\t\\item \\textit{referring expression} - part of speech that refers to an entity\n\t\t\\item \\textit{antecedent} - the text initially evoking a referent (where referent is named)\n\t\t\\item \\textit{anaphora} - the phenomenon of referring to an antecedent\n\t\t\\item \\textit{cataphora} - pronouns that appear \\textit{before} the pronoun (rare)\n\t\\end{itemize}\n\t\\item \\textbf{Pronoun resolution}\n\t\\begin{itemize}\n\t\t\\item Identifying the referents of pronouns\n\t\t\\item \\textit{Anaphora resolution}: in most cases, the task is limited to identifying referents that are mentioned before the actual pronoun/reference\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Algorithms for anaphora resolution}\n\\begin{itemize}\n\t\\item For anaphora resolution, we mostly apply a supervised training algorithm\n\t\\item The instances in the corpus are possible pairs of pronoun and antecedent (possible antecedent include all noun phrases in the current and last 5 sentences)\n\t\\item The classification is binary (true if pronoun refers to this specific antecedent, otherwise false)\n\t\\item Training data is annotated by humans\n\t\\item Beware that there are also pronouns in the text that might have no referent at all (\\textit{pleonastic pronouns})\n\t\\item Distinguishing between \\textit{hard} and \\textit{soft} constraints that must be fulfilled between pronouns and antecedent\n\t\\item \\textbf{Hard constraints} : Pronoun must match in terms of tense, singular/plural, gender, ...\n\t\\item \\textbf{Soft constraints/Salience}: \n\t\\begin{itemize}\n\t\t\\item \\textit{recency} -  more recent antecedents are preferred\n\t\t\\item \\textit{grammatical role} - subjects might be referred to more often than objects. Also, it is preferred that entity and pronoun has same role in sentence (subject, object, ...)\n\t\t\\item \\textit{repeated mention} - entities that have been mentioned more often are preferred\n\t\t\\item \\textit{coherence effect} - pronoun resolution might depend on discourse relation/semantic within the sentences\n\t\\end{itemize}\n\t\\item Based on the hard and soft constraints, we can define features for every pronoun-antecedent pair\n\t\\item Simple classification model takes these features as input and classifies the link as valid or not\n\t\\item Simplest evaluation matrix is link accuracy (number of correct links). However, it does not take into account pleonastic pronouns or a chain of references so that multiple metrics exist\n\\end{itemize}"
  },
  {
    "path": "Natural_Language_Processing_1/nlp_dialog_modelling.tex",
    "content": "\\section{Computational Dialog Modeling}\n\\subsection{Modular dialog systems}\n\\begin{itemize}\n\t\\item There are two main tasks in dialog modeling: either understand a conversation from outsider's view (summarizing), or the capability to take part in conversation $\\Rightarrow$ make a dialog agent\n\t\\item First approaches of modeling dialog agent were based on hand-specified patterns/transformation rules based on keywords to find an appropriate answer\n\t\\item Recently, the focus shifted towards data-driven methods by either retrieving existing information or generating new sentences by i.e. Encoder-Decoder architectures\n\t\\item Problems: hard to evaluate, such systems often show to just copy patterns in training dataset but don't generalize well.\n\t\\item Different approach: in dialogs, there is a tendency to ascribe goal and \\textbf{intentions}\n\t\\item However, intentions are not easy to recognize. That's why such methods are often used for \\textbf{task-oriented} dialog systems where the end-goal makes intentions tractable\n\t\\item The modular dialog system architecture: \n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/dialog_modeling_modules.png}\n\t\\end{figure}\n\t\\item \\textit{Language understanding}: NLP1 course. Morphology, POS tagging, lexical semantics and syntactic parsing, compositional semantics, ...\n\t\\item \\textit{Dialog management}: consists of two modules:\n\t\\begin{itemize}\n\t\t\\item \\textit{Dialog state tracker}: handle linguistic context (what has been said) and how relevant it is to the task. We can convert messages into slots with parameters (like \\texttt{request(name)}) to simplify the task. \n\t\t\\item \\textit{Dialog policy}: select what action to take next/model the next answer. Estimate probabilities for possible actions, and choose best ones. Training mostly done in a reinforcement learning way with simulator\n\t\\end{itemize}\n\t\\item \\textit{Extra-linguistic environment}: taking information into account which is not coming from this dialog itself (images, databases, ...)\n\\end{itemize}\n\\subsection{Visually grounded, task-oriented dialog}\n\\begin{itemize}\n\t\\item \\textbf{Visual dialog}: given an image and a history of human-dialog, answer a follow-up question \n\t\\item We can evaluate this task with the same metrics as for summarization and translation (BLEU, ROGUE)\n\t\\item However, in this task the agent is thrown at random into a conversation without being able to interact\n\t\\item \\textit{Image guessing game}: one agents ($Q$) sees the original image, and the other agent ($A$) sees the image with the object highlighted that $Q$ needs to guess\n\t\\item Current implementation is based on LSTMs with CNN encoders. Still, the models perform poorly compared to humans\n\\end{itemize}"
  },
  {
    "path": "Natural_Language_Processing_1/nlp_formal_grammars.tex",
    "content": "\\section{Formal grammars and syntactic parsing}\n\\begin{itemize}\n\t\\item Syntax: structure of sentence, parsing syntax to get (long-distance) dependencies of words\n\\end{itemize}\n\n\\subsection{Generative grammar}\n\\begin{itemize}\n\t\\item Formally specified grammar that can generate all and only acceptable sentences of a natural language\n\t\\item A phrase can be bracketed into its internal structure: \\textit{((the (big dog)) slept)}\n\t\\item Each subpart is a \\textbf{constituent}: group of words/phrase behaving as a single unit \n\t\\item Labels can be assigned to the internal structures (for instance, \\textit{the big dog} is a noun phrase)\n\\end{itemize}\n\\subsubsection{Phrases and substitutability}\n\\begin{itemize}\n\t\\item Words with the same POS tag can be replaced\n\t\\item Phrasal categories indicate which phrases can be substituted\n\t\\item Example phrasal categories include noun phrase (NP), verb phrase (VP), propositional phrase (PP), ...\n\t\\item Goal: capture substitutability at phrase level by phrasal categories\n\\end{itemize}\n\\subsection{Context-Free Grammars}\n\\begin{itemize}\n\t\\item Defining a grammar on rules of production, and basic lexicon\n\t\\item Basic elements of a context-free grammar (CFG):\n\t\\begin{enumerate}\n\t\t\\item Set of non-terminal symbols (e.g. S, VP)\n\t\t\\item Set of terminal symbols (i.e., the words)\n\t\t\\item Set of rules, where left-hand side is single non-terminal symbol, and right side combination of non-terminal and terminal. Examples:\\\\\n\t\t\\texttt{S -> NP VP}\\\\\n\t\t\\texttt{V -> fish}\n\t\t\\item A start symbol (here \\texttt{S}) which is a non-terminal\n\t\\end{enumerate}\n\t\\item Exclude empty productions, like \\texttt{NP->$\\epsilon$}\n\t\\item For rules of non-terminal to single word, the non-terminal represents the POS tag of this word\n\t\\item A context free grammar can be used for either generating sentences (start with \\texttt{S}, and choose rules), or for analyzing/assigning a structure to a given sentence\n\t\\item For analyzing, the bracketed notation or a parse tree is often used to represent the structure:\n\t\n\t\\texttt{(S (NP} \\textit{they}\\texttt{) (VP (V }\\textit{fish}\\texttt{)))}  (for parse tree, \\texttt{S} would be root node and \\texttt{NP} and \\texttt{VP} its children a.s.o.)\n\t\\item However, the grammar is not always unique \n\t\\begin{itemize}\n\t\t\\item \\textbf{Lexical ambiguity}: a word is more than once in the lexicon having different POS tag\n\t\t\\item \\textbf{Structural ambiguity}: multiple possible analysis because of multiple rules for same non-terminals\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Chart parsing with CFGs}\n\\begin{itemize}\n\t\\item Increase efficiency by recording all possible rules we could apply on a sentence\n\t\\item The \\textbf{chart} is a record of all substructures that have ever been built during the parsing / stores partial results of parsing in a vector\n\t\\item An \\textbf{edge} is a data structure that represents a rule application ,which includes:\n\t\\begin{itemize}\n\t\t\\item An id for referring to it\n\t\t\\item The outer left and right node in the sentence of the phrase on which the rule is applied\n\t\t\\item The \\textit{mother} symbol (non-terminal which is on the left side of the rule)\n\t\t\\item The \\textit{daughters} which are the symbols on the right side of the rule (words and/or non-terminals produced by previous edges and referred to by the id)\n\t\\end{itemize}\n\t\\item In conclusion, a full chart for the sentence \\texttt{they can fish} look like that:\n%\t$$\\begin{array}{ccccc}\n%\t\\text{id} & \\text{left} & \\text{right} & \\text{mother} & \\text{daugthers}\\\\\n%\t\\hline\n%\t1 & 0 & 1 & \\texttt{NP} & \\text{(they)}\\\\\n%\t2 & 1 & 2 & \\texttt{V} & \\text{(can)}\\\\\n%\t3 & 1 & 2 & \\texttt{VP} & \\text{(2)}\\\\\n%\t4 & 0 & 2 & \\texttt{S} & \\text{(1 3)}\\\\\n%\t\\end{array}$$\n\t$$\\begin{array}{ccccc}\n\t\\text{id} & \\text{left} & \\text{right} & \\text{mother} & \\text{daugthers}\\\\\n\t\\hline\n\t1 & 0 & 1 & \\texttt{NP} & \\text{(they)}\\\\\n\t2 & 1 & 2 & \\texttt{V} & \\text{(can)}\\\\\n\t3 & 1 & 2 & \\texttt{VP} & \\text{(2)}\\\\\n\t4 & 0 & 2 & \\texttt{S} & \\text{(1 3)}\\\\\n\t5 & 2 & 3 & \\texttt{V} & \\text{(fish)}\\\\\n\t6 & 2 & 3 & \\texttt{VP} & \\text{(5)}\\\\\n\t7 & 1 & 3 & \\texttt{VP} & \\text{(2 6)}\\\\\n\t8 & 0 & 3 & \\texttt{S} & \\text{(1 7)}\\\\\n\t9 & 2 & 3 & \\texttt{NP} & \\text{(fish)}\\\\\n\t10 & 1 & 3 & \\texttt{VP} & \\text{(2 9)}\\\\\n\t11 & 0 & 3 & \\texttt{S} & \\text{(1 10)}\\\\\n\t\\end{array}$$\n\twhere the sentence is structured as $._0$\\texttt{they}$._1$ \\texttt{can}$._2$ \\texttt{fish}$._3$ and rows of the chart are edges. The parsing is visualized in the figure below\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/chart_parsing_structure.png}\n\t\t\\caption{Example for chart parsing. The resulting chart data structure is shown below.}\n\t\t\\label{fig:chart_parsing_structure}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{Implementation of bottom-up parser}\n\\begin{itemize}\n\t\\item A bottom-up parser would start from the left, look at the first two connections points (first word) and check for a rule that can be applied to this word\n\t\\item If a rule has been found, it is added as edge into the chart, and we start looking for rules that can be applied to the new edge (recursion!). Once no more rule can be applied on this edge and the recursion stops, we go back to the word and continue our search for applicable rules on this word \n\t\\item After every rule was applied, the parser moves on to the next word on the right and check for rules that can be applied to this word, \\textbf{and} all other words/edges that have been processed beforehand. \n\t\\item Only if no more rules can be applied, the parser moves on to the next word, until all words are processed\n\t\\item The correct parse/grammar structure is this one that end with the start symbol \\texttt{S} from the first to the last node of the sentence\n\t\\item Important sub-technique: \\textbf{Packing}\n\t\\begin{itemize}\n\t\t\\item Due to multiple rules with same input, we can have two identical edges that are just based on different daughters\n\t\t\\item Every following rule is then applied on both edges which is very inefficient\n\t\t\\item Thus, with \\textit{packing} we change the daughter entries to a list of possible daughter lists \n\t\t\\item For example, the edge 7 and 10 from the previous example can be combined:\n\t\t$$\\begin{array}{ccccc}\n\t\t\\text{id} & \\text{left} & \\text{right} & \\text{mother} & \\text{daugthers}\\\\\n\t\t\\hline\n\t\t7 & 1 & 3 & \\texttt{VP} & \\left\\{\\text{(2 6), (2 9)}\\right\\}\\\\\n\t\t\\end{array}$$\n\t\t\\item If a new daughter list is added, no new recursion/rule application needs to be done  \n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Probabilistic parsing}\n\\begin{itemize}\n\t\\item For a single sentence with 20 or more words, we will get over 1000 analysis $\\Rightarrow$ how do we determine the best/most probable analysis?\n\t\\item The traditional approach is it to grammar rules handwritten but they tend to often fail when parsing new sentences\n\t\\item  Current approaches: probabilistic CFG (PCFG) where every rule is augmented with a probability\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.35\\textwidth]{figures/chart_parsing_prob_cfg.png}\n\t\t\\caption{Probabilistic CFGs. The probabilities are normalized over all rules with same \\textit{left} side.}\n\t\t\\label{fig:probabilistic_cfg}\n\t\\end{figure}\n\t\\item The probability of a parse tree is the product of the probabilities of all the grammar rules that are used in the sentence derivation\n\t\\item Probabilistic CFGs help for \\textit{disambiguation} as we can rank all analysis by probability and just pick the best (n) one(s)\n\t\\item Probabilities can also be used to speed up parsing (drop trees/substructures during parsing that already have a very low probability compared to other current substructures)\n\\end{itemize}\n\\subsubsection{Treebank PCFGs}\n\\begin{itemize}\n\t\\item Instead of specifying/tuning the grammar and its corresponding probabilities by our own, we can use a large dataset of sentences with annotated parse trees\n\t\\item This way, we implicitly get a grammar and the probabilities of each rule\n\t\\item A \\textbf{treebank} is therefore a collection of sentences annotated with constituent trees\n\t\\item To estimate the rule probabilities, we use the maximum likelihood:\n\t$$p(X\\to \\alpha) = \\frac{C(X\\to \\alpha )}{C(X)}$$\n\twhere $C(X\\to \\alpha)$ number of times the rule is used in corpus, and $C(X)$ the number of times the non-terminal symbol $X$ appears in treebank\n\\end{itemize}\n\\subsubsection{Why CFG and not finite state machines}\n\\begin{itemize}\n\t\\item Language often has centre-embeddings like $A\\to \\alpha A \\beta$ which cannot be captures by FSAs\n\t\\item However, humans limit the application of such centre-embeddings so that we can convert those into finite rules\n\t\\item The advantage of a FSA would be that we can model hierarchical structures (supported by the fact that we understand the semantic of a sentence, we need good internal structures like the hierarchy)\n\\end{itemize}\n\\subsection{Dependency structures}\n\\begin{itemize}\n\t\\item Context free grammars were based on phrase-structures in sentences\n\t\\item Another possible representation of parsing sentences is using directed/asymmetric binary grammatical relations that hold among the words \n\t\\item A relation consists of \n\t\\begin{itemize}\n\t\t\\item a head \\texttt{H} (central word)\n\t\t\\item a dependent \\texttt{D}\n\t\t\\item a label identifying the relation between \\texttt{H} and \\texttt{D}\n\t\\end{itemize}\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/dependency_parsing.png}\n\t\t\\caption{Dependency parsing. All relations are directed form head \\texttt{H} to dependent \\texttt{D}.}\n\t\t\\label{fig:dependency_parsing}\n\t\\end{figure}\n\t\\item It is important to prevent parsing errors as these can significantly change the semantic of a sentence\n\\end{itemize}"
  },
  {
    "path": "Natural_Language_Processing_1/nlp_lexical_distributional_semantics.tex",
    "content": "\\section{Lexical and distributional semantics}\n\\begin{itemize}\n\t\\item \\textbf{Compositional semantics}: meaning of phrase/sentence\n\t\\item \\textbf{Lexical semantics}: meaning of individual words\n\\end{itemize}\n\\subsection{Approaches for lexical meaning}\n\\begin{itemize}\n\t\\item How to represent a meaning of a word. Problem: no representation can fully capture language yet!\n\\end{itemize}\n\\subsubsection{Formal semantics}\n\\begin{itemize}\n\t\\item Based on set theory, describing the features of a word\n\t\\item Meaning postulates: $\\forall x \\left[\\text{bachelor}'(x)\\to\\text{man}'(x)\\wedge\\text{unmarried}'(x)\\right]$\n\t\\begin{itemize}\n\t\t\\item If a word is in the set \\textit{bachelor}, then it also is in \\textit{man} and \\textit{unmarried}\n\t\\end{itemize}\n\t\\item Problems:\n\t\\begin{itemize}\n\t\t\\item Limited, especially for special cases (i.e. what is the Pope?)\n\t\t\\item Very expensive for large corpus\n\t\t\\item For some words, it is almost impossible to find a good formalization\n\t\\end{itemize}\n\t\\item Alternative: \\textbf{Prototype theory}\n\t\\begin{itemize}\n\t\t\\item Notion of graded semantic categories with no clear boundaries\n\t\t\\item No requirement that a feature must be shared by all members\n\t\t\\item Certain members are more central or prototypical $\\Rightarrow$ \\textbf{Protoypes}\n\t\t\\item New members are added based on similarity to prototypes\n\t\t\\item Features and category memberships are graded\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Semantic relations}\n\\begin{itemize}\n\t\\item \\textbf{Hyponymy}: IS-A relation, forms a taxonomy (\\textit{Example}: dog is a hyponym of animal)\n\t\\begin{itemize}\n\t\t\\item Easier to construct for certain nouns, but especially hard for adjectives\n\t\\end{itemize}\n\t\\item \\textbf{Meronomy}: PART-OF relation (\\textit{Example}: arm is a meronym of body)\n\t\\item \\textbf{Synonymy}: Words can be exchanged without changing the meaning of a sentence/phrase\n\t\\item \\textbf{Antonymy}: Opposite meanings (\\textit{Example}: big vs. little)\n\\end{itemize}\n\\subsubsection{WordNet}\n\\begin{itemize}\n\t\\item Large-scale corpus for English resource\n\t\\item Handconstructed\n\t\\item Organized in synsets: sets of synonyms\n\t\\item Synsets are connected by semantic relations\n\t\\item Similarity of words is the similarity of synsets\n\\end{itemize}\n\n\\subsection{Polysemy and word sense disambiguation}\n\\begin{itemize}\n\t\\item A word can mean different things based on the sentence/context it is used in\n\t\\item Meaning of words is not fixed, but dynamically adapted by the context\n\t\\item \\textbf{Regular polysemy}: mechanisms to apply on words to fit into context\n\t\\begin{itemize}\n\t\t\\item \\textit{Zero-derivation}: verb $\\leftrightarrow$ noun without changing word. Example: ``\\textit{tango}''\n\t\t\\item \\textit{Metaphorical}: using words from a different domain to express similar meaning. Example: ``\\textit{swallow information}''\n\t\t\\item \\textit{Metonymy}: use an entity to actually refer to other aspects of it. \n\t\tExample: ``\\textit{drinking his glass}''\n\t\\end{itemize}\n\t\\item \\textbf{Word sense disambiguation}: derive meaning of word in context\n\t\\begin{itemize}\n\t\t\\item \\textit{Supervised} (most common) $\\to$ predefined list of senses (i.e. WordNet), and train model on \\textit{large} corpus. Problem: we have to learn a new classifier for every word!\n\t\t\\item \\textit{Semi-supervised} $\\to$ annotate small dataset, bootstrap from there. Might be helpful as some instances have no single/discrete meaning.\n\t\t\\item \\textit{Unsupervised} $\\to$ induce sense by clustering of word occurences. Usually, the result is either too fine-grained or coarse-grained for most word (only work great for frequent words)\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Semi-Supervised WSD: Yarowsky algorithm}\n\\begin{itemize}\n\t\\item Using bootstrapping which needs only a very small hand-labeled training set\n\t\\item Still, learns a classifier for every word $\\Rightarrow$ no generalization!\n\t\\item Also, define features (see notion of context) for every word to determine its meaning\n\t\\item \\textbf{Algorithm iteration}: \n\t\\begin{enumerate}[start=0]\n\t\t\\item Given a small initial seed set $\\Lambda_0$ of labeled instances of each sense, and a much larger unlabeled corpus $V_0$\n\t\t\\item Train classifier on $\\Lambda_0$\n\t\t\\item Use trained classifier to label $V_0$\n\t\t\\item Select the examples in $V_0$ that the classifier is most confident on\n\t\t\\begin{enumerate}\n\t\t\t\\item Reliability of a prediction defined as $\\log\\left(\\frac{p(a|w)}{p(b|w)}\\right)$ for word w and possible senses $a$ and $b$\n\t\t\t\\item Rank reliabilities of all predictions and choose $n$ best\n\t\t\\end{enumerate}\n\t\t\\item Remove chosen examples from $V_0\\Rightarrow V_1$, and add them to the training set $\\Rightarrow\\Lambda_1$\n\t\t\\item Repeat step 2) to 4) until:\n\t\t\\begin{enumerate}\n\t\t\t\\item Either $V_i$ is empty\n\t\t\t\\item Or the error rate on the training/validation is sufficient low\n\t\t\\end{enumerate}\n\t\\end{enumerate}\n\t\\item Reported accuracy of 95\\%, but on easy homonymous examples\n\t\\item \\textbf{One sense per discourse}\n\t\\begin{itemize}\n\t\t\\item Original algorithm uses \\textit{one sense per discourse} as second heuristic\n\t\t\\item If a word appears twice or more often in the same text, they are probably of the same meaning\\\\\n\t\t$\\Rightarrow$ Annotate these as well and use them as additional training examples\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Distributional semantics}\n\\begin{itemize}\n\t\\item Probabilistic models for semantics\n\t\\item Distributional hypothesis about word meaning: the meaning of a word is determined by its context $\\Rightarrow$ similar meanings have similar contexts\n\t\\item Thus, distributions are a good conceptual representation if you believe that ‘the meaning of a word is given by its usage’ $\\Rightarrow$ Corpus-dependent like different culture, domains, ...\n\t\\item Distributions can encode lexical- and world knowledge, but mostly only partial lexical semantics \n\t\\item Techniques: Count-based models and prediction models\n\\end{itemize}\n\\subsubsection{Count-based models}\n\\begin{itemize}\n\t\\item Vector spaced models in the semantic space, where every dimension corresponds to a possible context $\\Rightarrow$ features\n\t\\item Distribution can be seen as point in space\n\t\\item As a result, we get a feature matrix:\n\t$$\\begin{array}{c|cccc}\n\t& \\text{feature}_1 & \\text{feature}_2 & \\dots & \\text{feature}_n\\\\\n\t\\hline\n\t\\text{word}_1  & f_{1,1} & f_{2,1} & \\dots & f_{n,1}\\\\\n\t\\text{word}_2  & f_{1,2} & f_{2,2} & \\dots & f_{n,2}\\\\\n\t\\vdots  & \\vdots & \\vdots & \\ddots & \\vdots\\\\\n\t\\text{word}_m  & f_{1,m} & f_{2,m} & \\dots & f_{n,m}\\\\\n\t\\end{array}$$\n\t\\item Possible design choices in count-based models:\n\\end{itemize}\n\\begin{enumerate}\n\t\\item \\textbf{Notion of context}: how to define the context of a word\n\t\\begin{enumerate}\n\t\t\\item \\textit{Word windows}: n words on either side of the lexical item, and count occurrences of words\n\t\t\\item \\textit{Filtered word windows}: n words, but remove irrelevant words based on POS-tag or stop-list (don't need to extend window)\n\t\t\\item \\textit{Lexeme windowing}: word windows (filtered or unfiltered), but with using stemming (mostly lead to more robust models)\n\t\t\\item \\textit{(Syntactic) dependencies}: context with dependency structure it belongs to (directed link between heads and dependents). Can be used with different extends\\\\\n\t\tExample: ``The prime minister acknowledged the question'' \\\\\n\t\t- [prime\\_a 1, acknowledge\\_v 1] (a for adjectives, v for verbs)\\\\\n\t\t- [prime\\_a\\_mod 1, acknowledge\\_v\\_subj 1] (mod for modifiers, subj for verb in relation to subject)\\\\\n\t\t$\\Rightarrow$ Problem: complex context lead to sparse vectors\n\t\\end{enumerate}\n\t$\\Rightarrow$ Working best: small window sizes or short dependencies\n\t\\item \\textbf{Context weighting}: how to set the weights in the vector\n\t\\begin{enumerate}\n\t\t\\item \\textit{Binary model}: if $c$ co-occurs with word $w$, value of entry is 1, else 0\n\t\t\\item \\textit{Basic frequency model}: number of times $c$ co-occurs (probably normalized)\n\t\t\\item \\textit{Characteristic model}: weights express how characteristic a given context is for a word $w$\\\\\n\t\t\\item \\textit{Pointwise Mutual Information (PMI)}: example of characteristic model. Comparing probability of both words occur together compared to occurring alone.\\\\\n\t\t$$PMI(w,c)=\\log \\frac{P(w,c)}{P(w)P(c)} = \\log \\frac{P(c|w)}{P(c)} \\text{\\hspace{5mm} where \\hspace{5mm}}P(c) = \\frac{f(c)}{\\sum_k f(c_k)}, P(c|w) = \\frac{f(w,c)}{f(w)}$$\n\t\t$$\\Rightarrow PMI(w,c) = \\log \\frac{f(w,c)\\sum_k f(c_k)}{f(w)f(c)}$$\n\t\t$PPMI\\to$ only use positive values, $PPMI(w,c) =\\max\\left(PMI(w,c),0\\right)$\n\t\\end{enumerate}\n\t\\item \\textbf{Semantic space}: what are possible contexts\n\t\\begin{enumerate}\n\t\t\\item \\textit{Entire vocabulary}: every word represents a possible context.\\\\\n\t\t+ All info included (also the rare one) - Inefficient (large space \\& sparse), noisy\n\t\t\\item \\textit{Top n words with highest frequency}: \\\\\n\t\t+ More efficient, noise is filtered out - May miss out infrequent contexts\n\t\t\\item \\textit{Singular Value Decomposition}: dimension reduction by exploiting redundancies\\\\\n\t\t+ Very efficient, good generalization - Not interpretable (or very hard)\n\t\t\\item \\textit{Non-negative matrix factorization}: Similar to SVB, but performs factorization differently\n\t\\end{enumerate}\n\\end{enumerate}\n\\subsubsection{Prediction-based models}\n\\begin{itemize}\n\t\\item Train a model to predict plausible contexts for a word\n\t\\item Learn word representations in the training process\n\t\\item \\textbf{Short dense} embeddings with \\textbf{latent} dimensions \n\t\\begin{itemize}\n\t\t\\item Easier to use as features with machine learning\n\t\t\\item Better generalization than simple counting $\\Rightarrow$ capturing more complex relations like synonym\n\t\\end{itemize}\n\t\\item One example for prediction-based models is skip-gram, also known as word2vec (see later section)\n\\end{itemize}\n\\subsubsection{Similarity}\n\\begin{itemize}\n\t\\item Definition of similarity very broad. Can include synonym, antonyms, hyponyms, ...\n\t\\item Measuring similarity with \\textbf{Cosine} between vectors $v$ and $u$:\n\t$$\\cos\\left(\\theta\\right) = \\frac{\\sum_k v_k \\cdot u_k}{\\sqrt{\\sum_k v_k^2} \\cdot \\sqrt{\\sum_k u_k^2}}$$\n\t\\item Cosine measure calculates the angle between $v$ and $u$, and is length-independent (normalization). Important as frequent words can have longer vectors\n\t\\item Other measures include Jaccard, Euclidean distance (vectors need to be normalized!), ...\n\t\\item However, true-synonyms do not always get higher similarity scores than near-synonyms, and also to antonyms\n\t\\item Identifying antonyms by extra heuristics like checking words with high similarity that frequently appear together (\\textit{example}: ``we serve hot and cold drinks'')\n\t\n\\end{itemize}\n\\subsubsection{Distributional word clustering}\n\\begin{itemize}\n\t\\item Cluster words based on the contexts they occur\n\t\\item Predefine number of clusters and corpus (for instance 2000 nouns in 200 clusters)\n\t\\item \\textbf{Features} can represent different kinds of contexts\n\t\\begin{itemize}\n\t\t\\item Windows based context, parsed or unparsed, syntactic dependencies\n\t\t\\item Define notion of context, context weighting, and semantic space \n\t\t\\item Feature representation can significantly influence performance\n\t\\end{itemize}\n\t\\item Clustering algorithm: K-means\n\t\\begin{itemize}\n\t\t\\item Given dataset with $N$ points and task of $K$ clusters, minimize sum of squares of distance of each data point to its closest cluster mean $\\bm{\\mu}_i$:\n\t\t$$\\arg\\min_C \\sum\\limits_{i=1}^{K}\\sum\\limits_{\\bm{x}\\in C_i} ||\\bm{x} - \\bm{\\mu}_{i}||^2$$\n\t\\end{itemize}\n\t\\item Small context sizes (small windows, syntactic dependencies) lead to clusters of synonyms (words that can be replaced)\n\t\\item Large context sizes lead to \\textbf{topical similarity} (words belonging to the same topic)\n\\end{itemize}\n\n\\subsubsection{Skip-gram}\n\\begin{itemize}\n\t\\item Sometimes referred to as \\textit{word2vec} because it is implemented in this package\n\t\\item Given a word $w_t$, predict neighbouring words in a context windows of $2L$ words (for $L=2$: $w_{t-2}, w_{t-1}, w_{t+1}, w_{t+2}$) \n\t\\item In skip-gram, we learn two representations for every word $w_j \\in V$:\n\t\\begin{itemize}\n\t\t\\item \\textbf{word embedding} $v$ in word matrix $W$\n\t\t\\item \\textbf{context embedding} $c$ in context matrix $C$ (word in the role as context for other words)\n\t\\end{itemize}\n\t\\item To learn these embeddings, we take every word $w(t)$ in the corpus (index $j$ in vocabulary), and try to predict $w(t+1), ...$ where we denote this word with index $k$ in the vocabulary: \n\t$$p\\left(w_k | w_j\\right)$$\n\t\\item The idea in skip-gram is that we compute this probability by the similarity between the words $w_k$ and $w_j$ whereas we use the context matrix $C$ for $w_k$ and the word matrix $W$ for $w_j$ (see figure below)\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/skip_gram_matrices.png}\n\t\t\\caption{Skip gram overview}\n\t\t\\label{fig:skip_gram_matrices}\n\t\\end{figure}\n\t\\item Similar to the cosine similarity, we use the dot product for calculating this:\n\t$$\\text{Similarity}(c_k, v_j)\\propto c_k\\cdot v_j$$\n\t\\item To normalize and get probability distribution over contexts, we use the softmax function:\n\t$$p\\left(w_k|w_j\\right)=\\frac{\\exp\\left(c_k \\cdot v_j\\right)}{\\sum_{i\\in V} \\exp\\left(c_i \\cdot v_j\\right)}$$\n\t\\item For the learning process, we start with randomly initialized vectors, and try to maximize the log-likelihood of the dataset (by performing SGD or similar):\n\t$$\\arg\\max \\sum\\limits_{\\left(w_j, w_k\\right)\\in D} \\log p\\left(w_k|w_j\\right) = \\sum\\limits_{\\left(w_j, w_k\\right)\\in D}  \\left(c_k \\cdot v_j - \\log \\sum\\limits_{c_i \\in V} \\exp\\left(c_i \\cdot v_j\\right)\\right)$$\n\t\\item We can also represent skip-gram as a (neural) network:\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/skip_gram_nn.png}\n\t\t\\caption{Skip gram as neural network. The weights of the first layer represent the word embedding $W$, whereas the context embedding $C$ is in the second layer.}\n\t\t\\label{fig:skip_gram_nn}\n\t\\end{figure}\n\t\\item However, the problem here is that for a large vocabulary size, the denominator of the softmax is very expensive to calculate $\\Rightarrow$ \\textbf{negative sampling}\n\t\\begin{itemize}\n\t\t\\item Approximate denominator by sampling $k$ random words from vocabulary (probability of sampling for a word is mostly connected to its unigram probability/frequency in the corpus like $\\text{count}^{\\alpha}(w)$ with for example $\\alpha=0.75$)\n\t\t\\item Dataset consists therefore out of words pair which are either positive or negative examples for context + word (note that we do not distinguish between the probability calculation of $w_{t-2}$ and $w_{t-1}$ for example)\n\t\t\\item We convert the classification task into predicting whether a context pair is a positive or negative example from the corpus\n\t\t$$p\\left(+|w_j, w_k\\right) = \\sigma(c_k\\cdot v_j) = \\frac{1}{1 + \\exp(-c_k\\cdot v_j)}$$\n\t\t$$p\\left(-|w_j, w_k\\right) = 1 - p\\left(+|w_j, w_k\\right) = \\frac{1}{1 + \\exp(c_k\\cdot v_j)}$$\n\t\t$$\\Rightarrow \\arg\\max \\sum\\limits_{\\left(w_j, w_k\\right)\\in D_{+}} \\log p\\left(+|w_k,w_j\\right) + \\sum\\limits_{\\left(w_j, w_k\\right)\\in D_{-}} \\log p\\left(-|w_k,w_j\\right) $$\n\t\\end{itemize}\n\t\\item Embeddings capture \\textbf{analogies}: \\textit{a} is to \\textit{b} as \\textit{c} is to \\textit{d}\n\t\\begin{itemize}\n\t\t\\item Due to similarity, we can use the offsets to find the appropriate word $d$:\n\t\t$$a-b\\approx c-d \\Rightarrow d' = \\arg\\max_{d'_w \\in V} \\cos\\left(a-b, c-d'\\right)$$\n\t\\end{itemize}\n\t\\item Word2vec is often used as initialization/pretraining for other tasks. Reasons:\n\t\\begin{itemize}\n\t\t\\item Will help the model to start from an informed position\n\t\t\\item Only needs a plain text corpus without any annotation\n\t\t\\item Is very fast and pretrained versions are also available on the internet\n\t\t\\item Best performance can be achieved by fine-tuning the weights afterwards\n\t\\end{itemize}\n\\end{itemize}"
  },
  {
    "path": "Natural_Language_Processing_1/nlp_morphology.tex",
    "content": "\\section{Morphology and finite state techniques}\n\\begin{itemize}\n\t\\item Morphology concerns the \\textbf{structure of words}\n\t\\item \\textit{Morpheme}: minimal information carrying unit in a word. A word consists of morphemes\n\t\\item \\textit{Affix}: Morphemes that only occur in conjunction with other morphemes\n\t\\item \\textit{Stem}: a word is made up of a stem and zero or more affixes. Stems are therefore stand-alone morphemes\n\t\\item There are different forms of affixes that describe when (prefix, suffix, infix, ...)\n\t\\item An affix is productive if it applies in general and therefore also probably for new words\n\t\\item \\textbf{Inflectional morphology}\n\t\\begin{itemize}\n\t\t\\item Fills predefined slots in paradigm, as plural, tense,... (create different grammatical forms, but word stays the same)\n\t\t\\item Fully productive, except irregular forms\n\t\t\\item Inflectional affixes are not combined in English\n\t\\end{itemize}\n\t\\item \\textbf{Derivational morphology}\n\t\\begin{itemize}\n\t\t\\item Forming a new word through affix (also change of meaning possible)\n\t\t\\item May change POS tag\n\t\t\\item Examples include \\textit{anti-}, \\textit{re-}, \\textit{-ism}, \\textit{-ist} (``reset'' vs. ``set'')\n\t\t\\item Generally semi-productive (applies for only subset of words in language)\n\t\t\\item Include \\textit{zero-derivation}: word that is both verb and noun, e.g. ``text'' vs. ``(to) text (someone)''\n\t\\end{itemize}\n\t\\item Ambiguities in terms of morphemes (single stems or affixes are ambiguous like ``dog'') or structure (combination of affixes/stem like ``shorts'' vs ``short-s'')\n\t\\item \\textbf{Bracketing}\n\t\\begin{itemize}\n\t\t\\item Starting from the stem, find the combination of nearby affixes that still lead to a possible form\n\t\t\\item Example \\textit{un-ion-ise-ed}. Putting \\textit{un-} and \\textit{ion} together not possible as this forms a non-valid word (union would be different stem). $\\Rightarrow$ \\textit{un-(ion-ise)-ed}\n\t\t\\item Next, adding the \\textit{-ed} ending is valid, and finally concatenating it with \\textit{un-}: \\textit{(un-((ion-ise)-ed))}\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Applications of morphological processing in NLP}\n\\begin{itemize}\n\t\\item We can use morphology to create a full-form lexicon (lexicon with each form of every word in it). However, this tends to explode very fast (high redundancy) and is not scalable for new words\n\t\\item \\textbf{Stemming}: use rules to get the stem form of a word. This allows us to match words to a small set of base words\n\t\\item \\textbf{Lemmatization}: Only finding split of stems and affixes. Is the preprocessing step before parsing (understanding the word!)\n\t\\item Morphological process can either by analysis or generation\n\t\\item Possible aspects/steps of morphological processing\n\t\\begin{enumerate}\n\t\t\\item Surface/ground-word mapped to stem(s) and affixes. Either by declaring the affixes (\\textit{ping-ed}) or by explicitly saying which rule was applied (\\textit{ping} \\textit{PAST\\_VERB})\n\t\t\\item After knowing the affixes, analyze internal structure by bracketing\n\t\t\\item Finally, understand syntactic and semantic effects where parsing can filter results of previous stages\n\t\\end{enumerate}\n\t\\item Overall, we need a lexicon combining three aspects:\n\t\\begin{itemize}\n\t\t\\item affixes (with the associated information they carry)\n\t\t\\item irregular forms\n\t\t\\item stems (with syntactic categories)\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Spelling rules}\n\\begin{itemize}\n\t\\item English morphology is essentially concatenative\n\t\\item English spelling rules can be described independently of the particular stems and affixes involved. It simply looks at the affix boundaries.\n\t\\item Example spelling rule for e-insertion:\n\t$$\\epsilon \\to \\text{e}/\\left\\{\\begin{array}{c}\n\t\\text{s}\\\\\n\t\\text{x}\\\\\n\t\\text{z}\n\t\\end{array}\\right\\}\\hat{\\text{ }}\\textunderscore s$$\n\tHere, the formula is interpreted as ``an empty string maps to e if an s,x or z is followed by an s of the next affix'' (where e is inserted in the underscore space)\n\t\\item Finite state machines (or transducers also creating corresponding output while parsing) can be used to implement spelling rules\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/morphology_FST.png}\n\t\\end{figure}\n\t\\item Each transition corresponds to a pair of characters. \n\t\\item When the transducer is run in analysis mode, the system can detect an affix boundary (where we look up the stem and affix in the corresponding lexicon)\n\t\\item In generation mode, we can just put in our parsed version and generate the correct spelling\n\t\\item Morphology systems are usually implemented so that there is one FST per spelling rule and these operate in parallel\n\t\\item However, FST are not applicable for internal structures as for example no bracketing model is possible\n\t\\item A system which generates invalid output/accepts invalid derivations is said to \\textbf{overgenerate}.\n\\end{itemize}"
  },
  {
    "path": "Natural_Language_Processing_1/nlp_pos_tagging.tex",
    "content": "\\section{Language models and part-of-speech tagging}\n\\subsection{Probabilistic language modeling}\n\\begin{itemize}\n\t\\item The Naive Bayes approach considers words as independent. However, we can also model word sequences using statistical techniques (based on context/semantic and syntax)\n\t\\item \\textbf{Corpus}: text collected for some purpose (i.e. movie reviews)\n\t\\begin{itemize}\n\t\t\\item A corpus is \\textit{balanced} if it represents different genres (types of text/domain)\n\t\t\\item A \\textit{tagged} corpus has annotations regarding the POS tags (mostly POS tags are learned unsupervised, but with this data we enable supervised training methods)\n\t\\end{itemize}\n\t\\item Use of language modeling\n\t\\begin{itemize}\n\t\t\\item In speech recognition, it is hard to distinguish between words that sound similar. Thus, language modeling is used to rank the hypothesis of the recognizing system to make an estimation what phrase is most likely being said\n\t\t\\item Language modeling can also be used for word prediction, text entry, spelling correction, ...\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{N-gram models}\n\\begin{itemize}\n\t\\item Modeling a sequence of $n$ words\n\t\\item \\textbf{Bigram} ($n=2$): \n\t\\begin{itemize}\n\t\t\\item use only previous word $\\Rightarrow$ $p(w_n|w_{n-1})$\n\t\t\\item Probability of a sequence can be expressed by $p(w_1,...w_N)=\\prod\\limits_{n=1}^{N} p(w_n|w_{n-1})$\n\t\t\\item Still, we assume that words with distance of more than 1 are independent\n\t\t\\item Estimating the probabilities by the maximum likelihood solution: $$p(w_n|w_{n-1})=\\frac{c(w_n, w_{n-1})}{\\sum_{w_k} c(w_k, w_{n-1})}=\\frac{c(w_n, w_{n-1})}{c(w_{n-1})}$$\n\t\tThus, we normalize the counts over the next word $w_{n-1}$\n\t\\end{itemize}\n\t\\item \\textbf{Trigram} ($n=3$):\n\t\\begin{itemize}\n\t\t\\item The probability of a word is based on the two previous words $\\Rightarrow$ $p(w_n|w_{n-1},w_{n-2})$\n\t\t\\item Again, the probability of the sequence is $p(w_1,...w_N)=\\prod\\limits_{n=1}^{N} p(w_n|w_{n-1},w_{n-2})$\n\t\\end{itemize}\n\t\\item Problems with sparse data\n\t\\begin{itemize}\n\t\t\\item \\textbf{Smoothing}: for smoothing, we add a small extra probability for rare and unseen events to prevent probabilities of zero. E.g. for bigram:\n\t\t$$p(w_n|w_{n-1})=\\frac{c(w_n, w_{n-1}) + \\kappa}{c(w_{n-1}) + |V|\\cdot \\kappa }$$\n\t\tSimple to implement, but only suitable if having few unseen events (high $n$-gram have a lot)\n\t\t\\item \\textbf{Backoff}: If we have good evidence of a long phrase, we use a high $n$-gram model (for example trigram). Otherwise, if phrase was not seen yet, we go to the next smaller model (here bigram) and checker its probability. If also not known, go deeper until you reach unigram.\n\t\t\\item \\textbf{Interpolation}: combine the probability estimations of all models. For example we can use linear interpolation where we weight every model with a parameter:\n\t\t$$p(w_n|w_{n-1},w_{n-2}) = \\lambda_1 p(w_n) + \\lambda_2 p(w_n|w_{n-1}) + \\lambda_3 p(w_n|w_{n-1}, w_{n-2})$$\n\t\tThe parameters $\\lambda_i$ need to sum up to 1 and are optimized on small held-out training subset. \n\t\t\\item \\textbf{Unknown word tag}: using a unknown word tag which is also used in the training set. Replace all unknown words in the (test) text by this tag\n\t\\end{itemize}\n\t\\item Another limitation of $n$-gram models are long-term dependencies as these cannot be captured efficiently\n\t\\item \\textit{Evaluation} of $n$-gram models\n\t\\begin{itemize}\n\t\t\\item \\textbf{Intrinsic evaluation}: evaluate directly on test set designed for the task with a metric\n\t\t\\begin{itemize}\n\t\t\t\\item A suitable metric is for example \\textit{perplexity} which is the inverse probability of the test dataset normalized by number of words $N$:\n\t\t\t$$PP(W) = \\left(p(w_1,...,w_N)\\right)^{-1/N}$$\n\t\t\t\\item For bigram, this would be $PP(W)=\\left(\\prod_{n=1}^{N}p(w_n|w_{n-1})\\right)^{-1/N}$\n\t\t\t\\item The goal is to minimize perplexity (lower perplexity indicates better model)\n\t\t\t\\item However, perplexity strongly relies on the similarity of training and test dataset and is therefore not comparable across different datasets\n\t\t\\end{itemize}\n\t\t\\item \\textbf{Extrinsic evaluation}: evaluation in the context of external task, i.e. speech recognition or word prediction\n\t\t\\begin{itemize}\n\t\t\t\\item Better, but very time consuming\n\t\t\t\\item Hybrid approaches compare own models by perplexity, and apply the best model in extrinsic environment (external task)\n\t\t\\end{itemize}\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Part-of-speech tagging}\n\\begin{itemize}\n\t\\item Tag every word by what king of speech it is (verb, noun, ... $\\Rightarrow$ ambiguity)\n\t\\item The tags are taken from a tagset which uses standardized codes for fine-grained POS\n\t\\item \\textit{Benefits} of POS tagging\n\t\\begin{itemize}\n\t\t\\item First step towards syntactic analysis (is very fast, but simpler than full syntax parsing)\n\t\t\\item POS tags can be useful features for application\n\t\\end{itemize}\n\t\\item Problem of ambiguity: most high-frequency words have more than one POS tag. Language with rich morphology (significant affixes) tend to have less as the distinguish affixes better\n\\end{itemize}\n\\subsubsection{Tagging strategies}\n\\begin{itemize}\n\t\\item Simplest strategy: assign to each words its most common tag (also called unigram tagging). Already gives a strong baseline\n\t\\item \\textbf{Hidden Marcov models}\n\t\\begin{enumerate}\n\t\t\\item Start with untagged text\n\t\t\\item Assign to the words all their possible POS tags\n\t\t\\item Find the most probable sequences of tags given sequences of words\n\t\t$$\\hat{t}^{n} = \\arg\\max_{t^{n}} p(t^{n} | w^{n}) = \\arg\\max_{t^{n}} p(w^{n}|t^{n})\\cdot p(t^{n}) $$\n\t\\end{enumerate}\n\t\\item If we apply for example bigram in this model, we get:\n\t\\begin{equation*}\n\t\\begin{split}\n\tp(t^{n}) & \\approx \\prod_{i=1}^{n} p(t_i | t_{i-1})\\\\\n\tp(w^{n}|t^{n}) & \\approx \\prod_{i=1}^{n} p(w_i | t_i)\\\\\n\t\\hat{t}^{n} & = \\arg\\max_{t^{n}} \\prod_{i=1}^{n}p(w_i | t_i)p(t_i | t_{i-1})\n\t\\end{split}\n\t\\end{equation*}\n\t\\item Actual systems use trigrams. Smoothing and backoff are important (fewer unknown open class words)\n\t\\item Evaluation by percentage of correct tags (but using most common tag already gives 90\\% accuracy. With trigram about 97\\%)\n\t\\item Common errors\n\t\\begin{itemize}\n\t\t\\item Difference between country ``Turkey'' and bird ``turkey'' (it decides based on whether an \\textit{a} is in front of turkey or not)\n\t\t\\item Because of smoothing, we can get for the phrase ``have hope'' that both words are verbs although hope has no past tense which is antigrammatical!\n\t\\end{itemize}\n\\end{itemize}"
  },
  {
    "path": "Natural_Language_Processing_1/nlp_summarization.tex",
    "content": "\\section{Language generation and summarization}\n\\begin{itemize}\n\t\\item Most tasks/methods until now have concentrated on language analysis. Next coming: tasks where we have to generate text\n\t\\item Generation mostly has the starting point at semantic representation like distributional semantics or hidden representation for neural networks\n\t\\item We can also concentrate on \\textit{regeneration} where we convert input to another representation. Examples include summarization, translation, ...\n\t\\item For generation, there are various subtasks (e.g. content selection, discourse structuring, ...)\n\t\\item Approaches for generation include:\n\t\\begin{itemize}\n\t\t\\item \\textit{Templates}: fixed text that has slots that can be filled\n\t\t\\item \\textit{Statistical}: using machine learning\n\t\t\\item \\textit{Deep Learning}: using deep embeddings, especially for regeneration task\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Text Summarization}\n\\begin{itemize}\n\t\\item Task: generate short version of input text with important points\n\t\\item We distinguish between \\textbf{single-document summarization} (given a single document, produce summary with important points) and \\textbf{multi-document summarization} (given a set of documents, produce brief summary of combination)\n\t\\item Also, we differentiate between \\textbf{generic summarization} (identifying important parts by itself and present these) and \\textbf{query-focused summarization} (regarding a query/question from the user, find relevant parts in the document/s)\n\t\\item There are mostly main approaches:\n\t\\begin{itemize}\n\t\t\\item \\textbf{Extractive summarization}: extract important info from document by copying sentences and combine them into a summary\n\t\t\\item \\textbf{Abstractive summarization}: interpret content of document and generate completely new sentences (much harder task!)\n\t\\end{itemize}\n\t\\item Most approaches deal with extractive summarization as it is much easier to realize and achieves better results till now\n\\end{itemize}\n\\subsubsection{Extractive summarization}\n\\begin{itemize}\n\t\\item For extractive summarization, there are three main steps:\n\t\\begin{enumerate}\n\t\t\\item \\textbf{Content selection}: identify important parts/sentences from the document \n\t\t\\item \\textbf{Information ordering}: order the sentence within the summary\n\t\t\\item \\textbf{Sentence realization}: optimizing the text by e.g. sentence simplification\n\t\\end{enumerate}\n\t\\item Approaches for \\textit{content selection}\n\t\\begin{itemize}\n\t\t\\item \\textit{Unsupervised}: \n\t\t\\begin{itemize}\n\t\t\t\\item Take those words that are significantly more often used than in other documents in average $\\Rightarrow$ these are the ``informative'' words and mostly biased towards names/cities (pronoun resolution important to find these references as well)\n\t\t\t\\item Measured by metrics like \\texttt{tf-idf}\n\t\t\\end{itemize}\n\t\t\\item \\textit{Supervised}:\n\t\t\\begin{itemize}\n\t\t\t\\item Large training corpus with human summary needed\n\t\t\t\\item Sentences of summary are aligned with those in the original document, and features are extracted (position in document, sentence length, informative words, ...)\n\t\t\t\\item Based on these features, we train a binary classifier whether a sentence should be included in the summary or not\n\t\t\t\\item Problem: expensive to generate all this data, and the supervised approaches did not significantly outperform the unsupervised ones\n\t\t\\end{itemize}\n\t\\end{itemize}\n\t\\item Approaches for \\textit{information ordering}:\n\t\\begin{itemize}\n\t\t\\item For a single document, the sentences are mostly structured in the order they occur in the original document\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Query-focused multi-document summarization}\n\\begin{itemize}\n\t\\item For query-focused multi-document summarization, we need to extend the extractive summarization by two pre-processing steps:\n\t\\begin{enumerate}\n\t\t\\item Find a set of relevant documents\n\t\t\\item (Optionally) simplify sentences in the documents (to make the task of content selection easier)\n\t\t\\item \\textit{Content selection}: identify informative sentences in the documents (much harder than for the single-document task)\n\t\t\\item \\textit{Information ordering}: order the sentences in the summary\n\t\t\\item \\textit{Sentence realization}: modify sentences to get consistent summary\n\t\\end{enumerate}\n\t\\item Approaches for \\textit{sentence simplification}\n\t\\begin{itemize}\n\t\t\\item Parse sentences and apply hand-rules what parts of a sentence we might drop (initial adverbials as ``for example'', irrelevant attribute clauses, ...)\n\t\t\\item Also possible to train a classifier to identify satellites (non-informative parts on a nucleon phrase)\n\t\\end{itemize}\n\t\\item Approaches for \\textit{content selection} for multiple documents\n\t\\begin{itemize}\n\t\t\\item We can either combine all documents into one, or retrieve information from all documents separately and weight these documents\n\t\t\\item Estimate informativeness similarly to single-document\n\t\t\\item Then, start by adding the most informative sentences in summary (one by one) until the maximum length of the summary is reached\n\t\t\\item When adding new sentences, we need to make sure that not the same/very similar sentences from different documents are added $\\Rightarrow$ \\textbf{Maximum marginal relevance}\n\t\t\\begin{itemize}\n\t\t\t\\item Iterative method to determine best sentence to add to summary. Relies on two counter-part measures:\n\t\t\t\\item \\textit{Relevance to query}: high cosine similarity between a sentence and the query indicates a high relevance for the summary\n\t\t\t\\item \\textit{Novelty regarding the summary so far}: low cosine similarity between sentences and summary\n\t\t\t\\item Estimated score is calculated as follows (for query $Q$, summary $S$, documents $D$):\n\t\t\t$$\\hat{s} = \\arg\\max_{s_i \\in D} \\left[\\lambda \\text{sim}\\left(s_i, Q\\right) - \\left(1 - \\lambda\\right)\\max_{s_j \\in S}\\text{sim}\\left(s_i, s_j\\right)\\right]$$\n\t\t\\end{itemize}\n\t\\end{itemize}\n\t\\item Approaches for \\textit{sentence ordering}\n\t\\begin{itemize}\n\t\t\\item \\textit{Chronologically}: for example by date of document\n\t\t\\item \\textit{Coherence}: sentences that are similar/discuss same entity should be grouped together in the summary\n\t\t\\item \\textit{Topical ordering}: learns set of topics present in documents (by e.g. topic modeling), and then order the sentences by topic\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Summarization using neural networks}\n\\begin{itemize}\n\t\\item We can apply neural networks for the task of summarization\n\t\\item For extractive summarization, we train a RNN on word level creating a representation of words, and a RNN on sentence/document level that combines sentence embeddings\n\t\\item Apply classifier on output of all document-level RNN to decide whether to include sentence in summary or not (problem: still captures coarse-grained features)\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.3\\textwidth]{figures/summarization_rnn.png}\n\t\t\\caption{Summarization by RNNs}\n\t\t\\label{fig:summarization_rnn}\n\t\\end{figure}\n\t\\item Abstractive summarization can be realized by large newspaper datasets where for a small article, a headline must be predicted\n\t\\item We use an Encoder-Decoder architecture (seq2seq models) where the encoder generates fixed-size embedding, and the decoder generates word-by-word output given this representation (decoder is autoregressive as it takes own output back as input for next time step)\n\\end{itemize}\n\\subsubsection{Evaluating summarization models}\n\\begin{itemize}\n\t\\item Human judgments of quality is too expensive\n\t\\item Better, automatic method: \\textbf{ROUGE} (recall oriented understudy for gisting evaluation)\n\t\\item We compare a few human-generated summaries $R_1, ..., R_N$ with the system generated summary $S$ by computing the percentage of $n$-grams from the reference summaries $R_1,...,R_N$ that occur in $S$. Example: ROUGE-2 (using bigram):\n\t$$\\frac{\\sum_{R_i}\\sum_{bigram_j\\in R_i} \\text{count}_{\\text{match}}(j,S)}{\\sum_{R_i}\\sum_{bigram_j\\in R_i} \\text{count}(j, R_i)}$$\n\t\\item Note that summary length is not considered here \n\t\\item Example for calculating ROUGE metric:\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/summarization_rogue_example.png}\n\t\t\\label{fig:summarization_rogue_example}\n\t\\end{figure}\n\t% summarization_rogue_example.png\n\\end{itemize}"
  },
  {
    "path": "Natural_Language_Processing_1/nlp_summary.tex",
    "content": "\\documentclass[a4paper]{article} \n\\addtolength{\\hoffset}{-2.25cm}\n\\addtolength{\\textwidth}{4.5cm}\n\\addtolength{\\voffset}{-3.25cm}\n\\addtolength{\\textheight}{5cm}\n\\setlength{\\parskip}{0pt}\n\\setlength{\\parindent}{0in}\n\n\\usepackage{blindtext} % Package to generate dummy text\n\\usepackage{charter} % Use the Charter font\n\\usepackage[utf8]{inputenc} % Use UTF-8 encoding\n\\usepackage{microtype} % Slightly tweak font spacing for aesthetics\n\\usepackage[english]{babel} % Language hyphenation and typographical rules\n\\usepackage{amsthm, amsmath, amssymb} % Mathematical typesetting\n\\usepackage{float} % Improved interface for floating objects\n\\usepackage[final, colorlinks = true, \nlinkcolor = black, \ncitecolor = black]{hyperref} % For hyperlinks in the PDF\n\\usepackage{graphicx, multicol} % Enhanced support for graphics\n\\usepackage{xcolor} % Driver-independent color extensions\n\\usepackage{marvosym, wasysym} % More symbols\n\\usepackage{rotating} % Rotation tools\n\\usepackage{subcaption}\n\\usepackage{censor} % Facilities for controlling restricted text\n\\newcommand{\\note}[1]{\\marginpar{\\scriptsize \\textcolor{red}{#1}}} % Enables comments in red on margin\n\\usepackage{bm}\n\\usepackage{blkarray}\n\\usepackage{enumitem}\n\\usepackage{pgfplots}\n\\definecolor{colkeyword}{rgb}{0,0.4,0}\n\\definecolor{colname}{rgb}{0.4,0.4,0}\n\\definecolor{coltype}{rgb}{0.4,0,0.4}\n\\definecolor{coloperators}{rgb}{0,0,1.0}\n\\definecolor{colscopes}{rgb}{0.4,0,0}\n\n\\setcounter{tocdepth}{2}\n% Title Page\n\\title{Summary Natural Language Processing 1}\n\\author{Phillip Lippe}\n\n\n\\begin{document}\n\\maketitle\n\\tableofcontents\n\\newpage\n\n\\input{nlp_morphology.tex}\n\\input{nlp_pos_tagging.tex}\n\\input{nlp_formal_grammars.tex}\n\\input{nlp_lexical_distributional_semantics.tex}\n\\input{nlp_compositional_semantic.tex}\n\\input{nlp_textual_entailment_paraphrasing.tex}\n\\input{nlp_dialog_modelling.tex}\n\\input{nlp_summarization.tex}\n\\input{nlp_translation.tex}\n\\input{nlp_bayesian.tex}\n\\appendix\n% \\newpage\n% \\input{nlp_appendix.tex}\n\\end{document}"
  },
  {
    "path": "Natural_Language_Processing_1/nlp_textual_entailment_paraphrasing.tex",
    "content": "\\section{Textual Entailment and Paraphrasing}\n\\begin{itemize}\n\t\\item Textual entailment is defined as a directional relationship of text $T$ to hypothesis $H$\n\t\\item We say $T$ entails $H$ if the meaning of $H$ can be inferred from the meaning of $T$\n\t\\item Task of recognizing textual entailment aims for classifying a pair of sentences as whether they are an entailment or not (binary classifier). Can be used in different settings:\n\t\\begin{itemize}\n\t\t\\item \\textit{Question-Answering}: A question answering system generates $n$ candidate solutions. The textual entailment recognizer must now decide which candidate solution is correct\n\t\t\\item \\textit{Summarization}: A summarization system sequentially generates new sentences to add to the summary in progress. The textual entailment recognizer should now identify whether a new sentence contains information that is already in the summary or not (redundancy checker).\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Levels of Representation}\n\\begin{itemize}\n\t\\item Determining the equivalence of the meaning of $T$ and $H$\n\t\\item The representation of the $T$-$H$ pair is used to train a supervised model\n\t\\item There are different levels of representation that can be used (all having their own benefits and drawbacks)\n\t\\item \\textbf{Lexical level}\n\t\\begin{itemize}\n\t\t\\item Solely looking on the words used in $T$ and $H$ (basically BoW of both sentences)\n\t\t\\item Comparing the used words for similarity (are words of $H$ in $T$)\n\t\t\\item Problem: structure of $H$ and $T$ cannot be fully captured by BoW\n\t\\end{itemize}\n\t\\item \\textbf{Structural level}\n\t\\begin{itemize}\n\t\t\\item Build up syntactic structure (like parse tree from context-free grammar or dependency graph) for $T$ and $H$\n\t\t\\item If $T$ contains same structures as $H$ (i.e. certain dependency edges, subtrees, ...), we predict the texts to be entailed\n\t\t\\item However, it is hard to distinguish which edges should contribute to similarity and which not \n\t\\end{itemize}\n\t\\item \\textbf{Semantic level}\n\t\\begin{itemize}\n\t\t\\item The idea is to label words/phrases with semantic role in sentence\n\t\t\\item Words are group into \\textit{arguments} (such as person or place) and connected to \\textit{predicates} (mostly verbs)\n\t\t\\item Now we check whether semantic connections of $H$ are in $T$ or not\n\t\t\\begin{figure}[ht]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.4\\textwidth]{figures/text_entailment_semantic_level.png}\n\t\t\\end{figure}\n\t\\end{itemize}\n\t\\item \\textbf{Knowledge Acquisition for RTE}\n\t\\begin{itemize}\n\t\t\\item To answer some text entailments, background/world knowledge is required (which words are synonyms, what is connected to a certain noun as i.e. a person/place...)\n\t\t\\item Knowledge is mostly constrained to lexical-semantic between two words (synonym, hypnonymy, ...)\n\t\t\\item But we can also model more complex relations like $X$ causes $Y$ $\\implies$ $Y$ is a \\textit{symptom} of $X$\n\t\t\\item Such connections/knowledge can be retrieved from WordNet, Wikipedia, ...\n\t\t\\item This leads to the \\textbf{Extended Distributional Hypothesis}: if two paths occur in similar contexts, the meaning of the paths tend to be similar ($X$ \\textit{solves} $Y$ compared to $X$ \\textit{is a solution of }$Y$)\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{Recognizing Text Entailment Methods}\n\\begin{itemize}\n\t\\item RTE depend on the representation which is used for $T$ and $H$\n\t\\item Different approaches to model the classifier\n\t\\item \\textbf{Similarity-based approach}\n\t\\begin{itemize}\n\t\t\\item Pair with strong similarity score gets high entailment relation\n\t\t\\item Similarity is measured by for example WordNet (how many edges to traverse to get to other word) and string similarity (length or even single letters)\n\t\\end{itemize}\n\t\\item \\textbf{Alignment-based approaches}\n\t\\begin{itemize}\n\t\t\\item Use heuristics to align junk of words from $T$ to $H$\n\t\t\\item For example, match phrase \"\\textit{purchase of $X$ to $Y$}\" with \"\\textit{$Y$ acquired $X$}\"\n\t\t\\item However, we need a knowledge base to infer these relations\n\t\t\\begin{figure}[ht]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.4\\textwidth]{figures/text_entailment_alginment_based_methods.png}\n\t\t\\end{figure}\n\t\\end{itemize}\n\t\\item \\textbf{Formal Logic approaches}\n\t\\begin{itemize}\n\t\t\\item Finding proof by theorem prover that $H$ can be proofed by $T$ \n\t\t\\item Convert statements in $T$ and $H$ into formal logic\n\t\t\\item Problem: mostly the lack of background knowledge is the bottleneck, as the simplest mistakes/missing statements can stop this approach to get the correct result\n\t\\end{itemize}\n\t\\item \\textbf{Edit distance-based approaches}\n\t\\begin{itemize}\n\t\t\\item Sequence of transformations that need to be applied on $T$ to get to $H$\n\t\t\\item If the number of transformations is higher than specified threshold, classify relation as \\texttt{false}\n\t\t\\item Alternative for \\textit{expensive} theorem prover\n\t\\end{itemize}\n\t\\item Evaluation done on dataset with 1,600 $T$-$H$ pairs with accuracy as metric. Lexical baseline is at about 58\\%\n\\end{itemize}\n\\subsection{Current methods}\n\\begin{itemize}\n\t\\item RTE datasets are mostly very small which limits the application of complex systems\n\t\\item However, there are large Natural Language Inference datasets, where also neural networks can be trained on (different domains, for example image to text)\n\t\\item We need datasets over multiple domains as otherwise the algorithms generalize poorly\n\t\\item \\textbf{Neural networks}\n\t\\begin{itemize}\n\t\t\\item Specifying features by hand for the input\n\t\t\\item Using both hypothesis and text as input. Mostly, we classify then into classes \\textit{entailment}, \\textit{contradiction} and \\textit{neutral} (not enough info to decide)\n\t\t\\item Using various LSTM models with attention modules \n\t\t\\item Generative models create a hypothesis given the text and the class for which the hypothesis should be generated\n\t\t\\item However, networks show to overfit on noise in the data (contradiction mostly contains negative words, entailments biased on animals and so on)\n\t\\end{itemize}\n\\end{itemize}"
  },
  {
    "path": "Natural_Language_Processing_1/nlp_translation.tex",
    "content": "\\section{Machine Translation}\n\\subsection{Statistical Machine Translation}\n\\begin{itemize}\n\t\\item Given a sentence $f$ in foreign language, find most probable translation $\\hat{e}$:\n\t$$\\hat{e} = \\arg\\max_{e} P(e|f) = \\arg\\max_{e} \\underbrace{P(f|e)}_{\\text{channel}} \\underbrace{P(e)}_{\\text{source}}$$\n\t\\item The source is the \\textbf{language model} which makes sure that the grammatical structure in the text is correct\n\t\\begin{itemize}\n\t\t\\item It is also helpful for disambiguate the word decision in the translating language\n\t\t\\item This is very important if a word in the foreign language is ambiguous\n\t\\end{itemize}\n\t\\item The (noisy) channel is the \\textbf{translation model} which is responsible to translate the text (makes sure that $f$ are translations of $e$)\n\t\\item IBM-3 model:\n\t\\begin{itemize}\n\t\t\\item For every word:\n\t\t\\begin{itemize}\n\t\t\t\\item Choose a fertility $\\phi_i$ (number of words in goal language should be translated into in foreign language. E.g. ``did'' has fertility of 0, ``slap'' 3 in French)\n\t\t\t\\item Generate $\\phi_i$ foreign words\n\t\t\t\\item Generate spurious/default words that might be needed\n\t\t\\end{itemize}\n\t\t\\item Permute translated words based on the position a word was before, and language it was in before\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{Learning parameters of models}\n\\begin{itemize}\n\t\\item For learning the parameters of the language and translation model, we would need the word alignments in the translation which however require the parameters\n\t\\item Thus, we apply the Expectation-Maximization algorithm\n\t\\item Assume we have alignments, but for every sentences multiple ones. For example, we can have the following possible alignments for a two-word sentence:\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/translation_EM_alignment.png}\n\t\\end{figure}\n\t\\item Every alignment is assigned to a probability/fractional count it occurs. Initially, we set the probability of every alignment/word to a uniform distribution\n\t\\item We try to maximize the probability that a foreign word/phrase $f_j$ in our corpus is  a translation of $e_{a_j}$ where $a_j$ is the alignment of the foreign to translated language. When using Bayes rule (and looking at only 1to1 alignments), we maximize:\n\t$$P(a,f|e) = \\prod\\limits_{j=1}^{M}t(f_j|e_{a_j})$$\n\twhere $t$ are the fractional counts\n\t\\item EM algorithm:\n\t\\begin{enumerate}[label=Step \\theenumi:]\n\t\t\\item Compute $P(a,f|e)$ for every possible alignment and sentence\n\t\t\\item Normalize the alignments for the same foreign sentence. $P(a,f|e)\\to P(a|f,e)$\n\t\t\\item Collect the fractional counts $tc(x|b)$ by summing up the probabilities of all $P(a|f,e)$ where $b$ is aligned to $x$. \n\t\t\\item Normalize fractional counts by $b$ $\\Rightarrow$ revised parameters for next iteration\n\t\\end{enumerate}\n\t\\item Example: Given $t(x|b) = 1/4$, $t(x|c)= 3/4$, $t(y|b)= 1/2$, $t(y|c)= 1/2$.\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/translation_EM_step_1.png}\n\t\\end{figure}\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/translation_EM_step_2.png}\n\t\\end{figure}\n\t\\begin{figure}[ht]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/translation_EM_step_3_4.png}\n\t\\end{figure}\n\t\\item Note on EM: Optimization function is non-convex so that we might find a local minimum\n\\end{itemize}\n\\subsection{Phrase-based Statistical Machine Translation}\n\\begin{itemize}\n\t\\item Previously, translation was based on single words as atomic unit. However, we can also use phrases (few consecutive words) as unit \n\t\\item The advantage is that context can be taken into account for translation, and no more fertility, insertion and deletion of words are necessary to translate\n\t\\item We now have a phrase table where we have probabilities to translate a certain phrase into another\n\t\\item The translation model uses phrases instead of words, but also needs to consider to reorder the phrases:\n\t$$P(f|e) = \\prod\\limits_{i=1} \\phi(\\overline{f}_i | \\overline{e}_i) \\underbrace{d(\\text{start}_i - \\text{end}_i -1)}_{\\text{distance-based reordering}}$$\n\tNote that $\\text{start}$ and $\\text{end}$ are the positions in the foreign language, but $i$ is the index of the translated language!\n\t\\item Extract all phrases that are consistent with a word alignment $A$. A phrase is consistent if all words of $\\overline{f}'$ are only aligned to words in $\\overline{e}'$ and not any other words outside this phrase (and the other way round).\n\t\\item The \\textit{phrase translation probability} $\\phi$ is estimated by the relative frequency:\n\t$$\\phi(\\overline{f}, \\overline{e}) = \\frac{\\text{count}(\\overline{f}, \\overline{e})}{\\sum_i \\text{count}(\\overline{f}_i, \\overline{e})}$$\n\\end{itemize}"
  },
  {
    "path": "README.md",
    "content": "# Summaries of Master AI at UvA\n\nIn this repository, I collect all my summaries I created during my studies in the Master programme Artificial Intelligence at the University of Amsterdam (2018 - 2020). Feel free to use them, but keep in mind that small mistakes might be included.\n\n## Courses\n\nThe PDF versions of the summaries can be found in the folder _Final_versions_. For latex files and editing, please see the corresponding folders.\n\n### Machine Learning 1 (UvA, 2018/19)\n* Semester 1, Period 2\n* Lecturer: Dr. Rianne van den Berg\n\n### Natural Language Processing 1 (UvA, 2018/19)\n* Semester 1, Period 2\n* Lecturer: Dr. Ekaterina Shutova\n* Website: https://cl-illc.github.io/nlp1/\n\n### Information Retrieval 1 (UvA, 2018/19)\n* Semester 1, Period 3\n* Lecturer: Dr. Evangelos Kanoulas\n\n### Knowledge Representation (VU, 2018/19)\n* Semester 2, Period 1\n* Lecturer: Dr. Frank van Harmelen\n\n### Computer Vision 1 (UvA, 2018/19)\n* Semester 2, Period 1\n* Lecturer: Dr. Theo Gevers\n* Website: https://cv1-uva.github.io \n\n### Deep Learning (UvA, 2018/19)\n* Semester 2, Period 2\n* Lecturer: Dr. Efstratios Gavves\n* Website: http://uvadlc.github.io\n\n### Machine Learning for Quantified Self (VU, 2018/19)\n* Semester 2, Period 3\n* Lecturer: Dr. Mark Hoogendoorn\n* Website: https://ml4qs.org\n\n### Machine Learning 2 (UvA, 2019/20)\n* Semester 1, Period 1\n* Lecturer: Dr. Joris Mooij\n\n### Reinforcement Learning (UvA, 2019/20)\n* Semester 1, Period 1\n* Lecturer: Dr. Herke van Hoof\n\n"
  },
  {
    "path": "Reinforcement_Learning/rl_appendix.tex",
    "content": "\\section{Deep RL in practice}\n\\textit{This section reviews the lecture slides 10 (last half).}\n\\begin{itemize}\n\t\\item There are several things to keep in mind when performing experiments in RL in practice\n\t\\item If we have a research questions that we want to investigate, we need to design experiments for which we have to answer the following questions:\n\\end{itemize}\n\\begin{description}\n\t\\item[On which tasks?] We need to find environments which fit to the research question in mind. Things to consider are:\n\t\\begin{itemize}\n\t\t\\item Continuous control tasks lend themselves to actor critic methods\n\t\t\\item Pixel-based task can show whether complex input data can be handled\n\t\t\\item Highly complex tasks show whether a method scales with having lots of compute and training data available\n\t\t\\item Toy examples can point out difference between methods, so it is often good to have both a toy example, and a more complex, practical one\n\t\\end{itemize}\n\t\\item[Which parameters and architectures to test?] RL have been shown to be very sensitive to the selection of hyperparameters. Hence, you should also spend similar tuning efforts on \\underline{all} your experiments, including the baseline, to ensure a fair comparison.\n\t\\item[Does a random seed affect my experiments?] Due to the high variance of the RL methods, we need to average all runs over a sufficient amount of seeds. Furthermore, if we perform a gridsearch, we should always keep the seeds fixed for all hyperparameter settings, but in the final test, use a different set of random seeds to prevent overfitting on seeds.\n\t\\item[What to report?] Next to the mean and/or median performance, the spread of the result should be shown as well. Furthermore, it should represent what you want to show. For example, if we want to underline that a new method learns faster, we should show a plot over learning iterations instead of just the final performance.\n\\end{description}\n"
  },
  {
    "path": "Reinforcement_Learning/rl_introduction.tex",
    "content": "\\section{Introduction to Reinforcement Learning}\n\\textit{This section reviews the lecture slides 1 and 2 (until Monte Carlo). }\n\\begin{itemize}\n\t\\item Reinforcement Learning deals with the following question: \n\t\n\tHow to \\textcolor{blue}{sequentially interact} with an environment to \\textcolor{blue}{maximize a long-term objective}?\n\t\n\t\\item The general RL setting is visualized in Figure~\\ref{fig:rl_introduction_reinforcement_learning}. Thereby we have 4 variables that are passed around:\n\t\\begin{itemize}\n\t\t\\item The state $S_t$, in which the agent currently is, determined by the environment\n\t\t\\item The action $A_t$ which is chosen by the agent, based on the observation of the state $S_t$\n\t\t\\item Based on the interaction $A_t$, the agent ends up in a new state $S_{t+1}$ as well as getting a reward $R_{t+1}$. Both properties are determined by the environment, and can only be influenced by the agent through $A_t$\n\t\\end{itemize}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/rl_introduction_reinforcement_learning.pdf}\n\t\t\\caption{Illustration of the interaction of an agent with an environment in the reinforcement learning setting.}\n\t\t\\label{fig:rl_introduction_reinforcement_learning}\n\t\\end{figure}\n\n\t\\item The decision of which action to take at which state is called policy, and we denote it by $\\pi$. The goal is to find the optimal policy $\\pi$ which maximizes the reward we get from the environment\n\t\t\n\t\\item As we learn by interactions, we have major main challenges in contrast to standard supervised learning:\n\t\\begin{itemize}\n\t\t\\item The dataset is not static as in e.g. image classification. Every time we change our policy, new data has to be generated as the actions we take at certain states are different. Note that there are techniques to use the data more efficiently, and we will discuss them later.  \n\t\t\\item Instead of having i.i.d. data, we have sequential data which are highly correlated. Standard optimizers like SGD fail because they assume i.i.d. inside a batch. We will also discuss how we can tackle this problem.\n\t\\end{itemize}\n\\end{itemize}\n\\subsection{MDPs and $k$-armed bandits}\n\\begin{itemize}\n\t\\item A common used, simplified example for RL is a $k$-armed bandit. We can image it as $k$ slot machines with unknown pay-off distribution. Hence, we have a static environment where the mapping function between actions and rewards is independent of state and time step $t$\n\t\\item Our goal is to maximize the cumulative reward over time. There are two variants we can use:\n\t\\begin{itemize}\n\t\t\\item If we want to maximize our reward for a finite horizon $T$ (i.e. limited number of trials), our goal is to maximize $\\sum_{t=1}^{T} r_t$\n\t\t\\item If we assume that we can take as many actions as we want, we have the objective for infinite horizon: $\\sum_{t=1}^{\\infty} \\gamma^{t}r_t$ where $\\gamma\\in [0,1]$ is a discount factor. Note that if we would not have a discount factor, we end up with infinite reward for any action whose reward has a mean greater than zero.\n\t\t\\item For generalization, we call $G_t=\\sum_{k=0}^{\\infty} \\gamma^{k}R_{t+k+1}=R_{t+1}+\\gamma G_{t+1}$ the \\textbf{(discounted) return}, or cumulative reward, at time step $t$. Only if a episode terminates (meaning that we cannot play forever), a discount factor of $\\gamma=1$ is allowed.\n\t\\end{itemize} \n\t\\item A general trade-off in reinforcement learning is between \\textbf{exploration} (i.e. trying new actions) and \\textbf{exploitation} (i.e. taking best actions we know). If we perform too little exploration, we might overlook the best action, for example due to stochasticity of the reward. However, exploiting the best actions is likely to lead to the maximum rewards, so that with exploring, we ``lose'' possible rewards.\n\t\n\tA general rule of thumb is: if we have much time left or are very uncertain about our current estimates, do more exploration. If we are limited on time, or are certain about our estimates, we should exploit more. \n\t\n\tAlso $\\gamma$ can play a role as the higher $\\gamma$, the more we care about rewards in the future, and hence, should perform more exploration.\n\t\n\t\\item We now introduce a set of important functions which are used for finding the optimal policy:\n\t\\begin{itemize}\n\t\t\\item The \\textbf{state-action value function}, also called \\underline{$q$-function}, expresses the expected return of taking a certain action in a given state:\n\t\t$$q_{\\pi}(s,a)=\\E_{\\pi}[G_t|S_t=s,A_t=a]$$\n\t\tNote that $q$-value is always specific to a certain policy as $G_t$ is in expectation that all steps after $t$ are taken according to the policy $\\pi$\n\t\t\\item The \\textbf{state-value function} is similar to the $q$ function, but only takes the state into account, and considers the action under the expectation:\n\t\t$$v_{^\\pi}(s)=\\E_{\\pi}[G_t|S_t=s]$$\n\t\\end{itemize}\n\n\t\\item In the case of the k-armed bandit, we try to learn a $q$-function (as we want to find the best action) but assume that we stay in the same state $s$. To balance exploration and exploitation, there are different strategies possible, for example:\n\t\\begin{itemize}\n\t\t\\item \\textbf{$\\epsilon$-greedy} takes in $(1-\\epsilon)$ cases the optimal action, and with the chance of $\\epsilon$ selects an action randomly\n\t\t\\item An annealed softmax takes the estimated action value into account, and creates a distriubtion based on this with a temperature factor $\\tau$ (high $\\tau$ means more stochasticity):\n\t\t$$p(a)=\\frac{\\exp\\left(\\hat{q}(a)/\\tau\\right)}{\\sum_{a'}\\exp\\left(\\hat{q}(a')/\\tau\\right)}$$\n\t\t\\item We can use the current estimate $\\hat{q}$ in combination with the uncertainty we have a certain action. This leads to the Upper confidence bound, or we can alternatively initialize all $q$-values optimistically (guarantees certain level of exploration)\n\t\\end{itemize} \n\n\t\\item \\textbf{Markov Decision Process}: An agent chooses an action which only depends on the current state $s_t$, and is independent of the history $s_0,...,s_{t-1}$ given $s_t$. Formally, we can define a finite MDP by\n\t\\begin{itemize}\n\t\t\\item A finite set of states $\\mathcal{S}$\n\t\t\\item A finite set of actions for each state $\\mathcal{A}_s$ (often the same in all states)\n\t\t\\item A dynamics function $p(s',r|s,a)=\\Prob{S_t=s',R_t=r|S_{t-1}=s,A_{t-1}=a}$ which is often split into\n\t\t\\begin{itemize}\n\t\t\t\\item Transition function $p(s'|s,a)$\n\t\t\t\\item Reward function $p(r|s,a,s')$\n\t\t\\end{itemize}\n\t\t\\item A discount factor $\\gamma\\in[0,1]$\n\t\\end{itemize}\n\t\\item In this setting, the optimal action can be found by optimizing the policy $\\pi^{*}(s_t)$. In the rest of the course, we mostly focus on MDPs\n\\end{itemize}\n\n\\subsection{Dynamic Programming}\n\\begin{itemize}\n\t\\item For simple environments where we know the dynamics function of the MDP, we can apply approaches of dynamic programming\n\t\\item One thing to notice about the functions $v$ and $q$ are their relationships, namely:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tv(s) & = \\E_{\\pi}\\left[G_t|S_t=s\\right] = \\E_{a\\sim\\pi}\\left[\\E_{\\pi}\\left[G_t|S_t=s,A_t=a\\right]\\right] = \\E_{a\\sim\\pi} q_{\\pi}(s,a)\\\\[8pt]\n\t\t\tq(s,a) & = \\E_{\\pi}[G_t|S_t=s,A_t=a] = \\E_{\\pi}[R_{t+1}|S_t=s,A_t=a]+\\E_{s',\\pi}[\\gamma G_{t+1}|S_{t+1}=s'] \\\\& = \\E_{s',\\pi}[R_{t+1}+\\gamma v(s')|S_t=s,A_t=a,S_{t+1}=s']\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item A policy is optimal if there is no other policy for which the value of any state is larger than the current one: $v_{*}(s)=\\max_{\\pi} v_{\\pi}(s)$, $q_{*}(s,a)=\\max_{\\pi} q_{\\pi}(s,a)$\n\t\\item Again, we can write down the relations between the two functions, which are called \\textit{Bellman optimality equations} for the optimal case:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tv_{*}(s) & =\\max_{a}q_{*}(s,a)= \\max_a \\E\\left[R_{t+1}+\\gamma v_{*}(S_{t+1})|S_t=s,A_t=a\\right]\\\\\n\t\t\tq_{*}(s,a) & = \\E\\left[R_{t+1}+\\gamma \\max_{a'} q_{*}(S_{t+1},a')\\Big\\vert S_t=s,A_t=a\\right]\n\t\t\\end{split}\n\t\\end{equation*}\n\t\n\t\\item The first approach of finding the optimal policy is \\textbf{policy iteration}. It combines two steps:\n\t\\begin{itemize}\n\t\t\\item \\textit{Policy evaluation}: given a policy $\\pi$, we try to find the corresponding value function $v_{\\pi}$. We do this by performing the update $v(s)=\\E[R_{t+1}+\\gamma v(s')]$ until the values converge. Note that we can evaluate the expectation as we know $\\pi$ and the MDP dynamics $p(s',r|s,a)$\n\t\t\\item \\textit{Policy improvement}: given the value function $v_{\\pi}$, we try to find a new policy for which we know that $\\forall s, v_{\\pi'}(s)\\geq v_{\\pi}(s)$. We can do that by taking the argmax over actions in each state.\n\t\\end{itemize}\n\tPolicy iteration performs these two in a loop until the policy is not changed anymore in the improvement step. It is guaranteed to converge to the optimal policy $\\pi$.\n\t\n\tThe full algorithm is shown in Figure~\\ref{fig:rl_introduction_policy_iteration}.\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.7\\textwidth]{figures/rl_introduction_policy_iteration.png}\n\t\t\\caption{Policy iteration algorithm (Sutton book)}\n\t\t\\label{fig:rl_introduction_policy_iteration}\n\t\\end{figure}\n\t\\item The issue of policy iteration is that the policy evaluation step can take a long time until it fully converges, although slight changes might not influence the policy too much. An alternative is to stop policy evaluation after a single iteration, and directly optimize it. This leads to the \\textbf{value iteration algorithm}.\n\t\n\tWhen implementing it, we can efficiently combine the two steps of evaluation and improvement, which is actually the same as performing the Bellman optimality equation as an update step. See Figure~\\ref{fig:rl_introduction_value_iteration} for details.\n\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.7\\textwidth]{figures/rl_introduction_value_iteration.png}\n\t\t\\caption{Value iteration algorithm (Sutton book)}\n\t\t\\label{fig:rl_introduction_value_iteration}\n\t\\end{figure}\n\n\tThe drawback of value iteration is that it can lead to noisy updates as it only performs a single update step and hence, can give inaccurate estimates of $v$. In practice, what has been found to mostly work the best, is to perform a limited, small number of steps of policy evaluation.\n\t\n\t\\item Keep in mind that for all these algorithms we require to know the MDP dynamics $p(s',r|s,a)$. However, this is often not the case, especially for more complicated, real-world environments. There we can only sample data point $(s_i,a_i,r_i,s'_i)$ which we need to use effectively. \n\t\n\\end{itemize}\n\\subsection{Outline}\n\\begin{itemize}\n\t\\item In the next sections (and rest of the whole course), we will deal with different ways of learning the optimal policy when the dynamics of the MDP are unknown in advance. We can distinguish the approaches into three main groups (see Figure~\\ref{fig:rl_introduction_overview_leanring_techniques}):\n\t\\begin{itemize}\n\t\t\\item \\textbf{Value-based} methods try to learn the value functions $v(s)$ and $q(s,a)$ from interactions with the environment. Based on these, we can find the optimal policy $\\pi$.\n\t\t\\item \\textbf{Policy-based} methods try to directly learn the desired objective, namely the policy $\\pi$. While we prevent propagating errors to the policy from learning a value function, it is often harder to optimize.\n\t\t\\item In contrast to the previous techniques, \\textbf{model-based} RL is based on the idea of learning the dynamics of the MDP, namely $p(s',r|s,a)$. With this knowledge given, we can then again apply value-based or policy-based methods, but support them by either using the transition function directly (i.e. take all possible future states into account instead of sampling), or simulate new trajectories if this is expensive in the original environment. \n\t\\end{itemize}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/rl_introduction_overview_leanring_techniques.png}\n\t\t\\caption{Overview of different learning strategies in RL.}\n\t\t\\label{fig:rl_introduction_overview_leanring_techniques}\n\t\\end{figure}\n\t\\item The next sections 2 and 3 (lecture slides 3 to 6) deal with value-based methods. First, we discuss tabular-based techniques, meaning that we store e.g. $v(s)$ by a big table (i.e. every state has an entry in this table). However, these methods cannot be applied if the state space is continuous and/or high-dimensional (size increases exponentially). Thus, we look at approximations in section 3.\n\t\\item Section 4 (lecture slides 7 to 10) deals with policy-based RL introducing different techniques for approximating the optimal gradients in policy learning.\n\t\\item Model-based RL is discussed in section 5 (lecture slides 11 and 12), but in less details than the previous two.\n\t\\item The final chapter deals with partially-observable environments (Section 7, lecture 13), and how to take uncertainty into account.\n\\end{itemize}\n"
  },
  {
    "path": "Reinforcement_Learning/rl_learning_with_approx.tex",
    "content": "\\section{Value-based RL: Learning with approximation}\n\\label{sec:value_based_approximation}\n\\textit{This section reviews the lecture slides 5 and 6.}\n\\begin{itemize}\n\t\\item When we talk about approximating the value function, we mean that instead of implementing $v$ as look-up table, we view it as parameterized function $\\hat{v}(\\bm{w},s)\\approx v_{\\pi}(s)$ with $\\bm{w}\\in\\R^{d}$.\n\t\\item Commonly, we try to allow generalization over nearby states while trying to keep it compact. Hence, the size of the weights,  $d$, is mostly much smaller than the actual state size. This implies that a change in $\\bm{w}$ will affect many states, and hence, generalize.\n\t\\item Learning value functions is similar to supervised learning as we try to push a prediction closer to a target (similar to regression). The value error can be summarized as:\n\t$$\\overline{\\text{VE}}(\\bm{w}) = \\sum_{s\\in S}\\mu(s)\\left[v_{\\pi}(s)-\\hat{v}(s,\\bm{w})\\right]^2=\\E_{s\\sim\\mu(s)}\\left[\\left(v_{\\pi}(s)-\\hat{v}(s,\\bm{w})\\right)^2\\right]$$\n\twhere $\\mu(s)$ is a weighting factor for the states (which state is how important, distribution over those). This depends on the task we are aiming for.\n\t\n\tHowever, keep in mind that our overall goal is to find the optimal policy, and not the best value function. So, the VE error might not be optimal as we often converge to local optima. \n\\end{itemize}\n\\subsection{Types of function approximations}\n\\begin{itemize}\n\t\\item There are various function approximation techniques we can use. We will review here a few, practical/simple ones\n\t\\item In general, we distinguish between linear and non-linear function approximation. We call an approximation linear if the value function is linear with respect to the weight, namely:\n\t$$\\hat{v}(s,\\bm{w})=\\bm{w}^T\\bm{x}(s)$$\n\twhere $\\bm{x}(s)$ can be any (non-)linear functions. It can be also seen as a linear combination of static feature extractions. Some examples for $\\bm{x}$ are:\n\t\\begin{itemize}\n\t\t\\item \\textit{Polynomials}, as if we take enough (infinite), we would be able to approximate any function. However, this is not feasible so we mostly have many features (especially in higher dimensions because we have $\\left[1,s_1,s_2,s_1^2, s_2^2, s_1s_2, s_1^2s_2,...\\right]$), and hence less generalization. Furthermore, the behavior at 0 is rather static\n\t\t\\item \\textit{Aggregations} where we group multiple states into one. This can be seen as returning a one-hot vector for $\\bm{x}$ where the $1$ assigns a point to a certain state group. Figure~\\ref{fig:rl_approximate_value_based_aggregation} visualizes some examples.\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.5\\textwidth]{figures/rl_approximate_value_based_aggregation.png}\n\t\t\t\\caption{Simple types of aggregations on a 2D state space. Note that we can combine aggregations, meaning that we use both a vertical and a horizontal aggregation.}\n\t\t\t\\label{fig:rl_approximate_value_based_aggregation}\n\t\t\\end{figure}\n\t\t\\item \\textit{Radial basis functions} that take the distance to a mean in the state space, e.g. $||\\mu_i-s||$, as input features. We can model this by having multiple Gaussians, and weight their influence by $p(s)$. It enables us to have smoother transitions between close-by states, but might be problematic for far-away states. This is why it is often problematic in high-dimensional state spaces.\n\t\t\\item \\textit{Fourier basis} where we take different frequencies to model $s$. This can provide quite a flexible feature set.\n\t\\end{itemize}\n\tNote that tabular RL can also be expressed by linear function approximation where we simply use $\\bm{x}(s)=\\left[\\delta(s=s_1), \\delta(s=s_2),...\\right]$, and $\\bm{w}$ therefore contains one parameter per state.\n\t\n\tLinear function approximation is especially used when prior knowledge can be introduced in the system. Carefully selecting the features simplifies the learning objective of the model, and hence, let it converge faster.\n\t\n\t\\item In non-linear function approximation, we use $\\bm{w}$ in a non-linear fashion in $\\hat{v}$, such as in neural networks.\n\t\n\\end{itemize}\n\\subsection{Prediction objective for on-policy prediction}\n\\begin{itemize}\n\t\\item In the case that we perform an on-policy prediction (i.e. policy evaluation for a fixed policy), the state importance is based on the visit frequency of $\\pi$. To arrive at $\\mu$, we also have to distinguish between the tasks:\n\t\\begin{itemize}\n\t\t\\item If we have a continuing task (never ending), we get a stationary distribution at the point:\n\t\t$$\\mu_{\\pi}(s)=\\sum_{s'}\\sum_{a}p(s|s',a)\\pi(a|s')\\mu_{\\pi}(s')$$\n\t\twith the condition that we can reach every state from the start.\n\t\t\\item For episodic tasks, we need to consider the start frequency $h(s)$ as well. To guarantee that $\\mu(s)$ is a distribution, we can use a softmax:\n\t\t$$\\mu_{\\pi}(s)=\\frac{\\eta(s)}{\\sum_{s'}\\eta(s')}, \\hspace{4mm}\\eta(s)=h(s)+\\sum_{s'}\\sum_a p(s|s',a)\\pi(a|s')\\eta(s')$$\n\t\twhere the second part is pretty much the same as before.\n\t\\end{itemize} \n\t\\item In order to calculate the gradients $\\nabla_{\\bm{w}}\\overline{\\text{VE}}(\\bm{w})$, we would need to know $\\mu(s)$ which is not possible due to missing information of the environment dynamics ($p(s'|s,a)$). However, we can approximate it by Monte-Carlo samples such that:\n\t$$\\nabla_{\\bm{w}}\\overline{\\text{VE}}(\\bm{w})\\approx \\nabla_{\\bm{w}}\\left[G_t - \\hat{v}(S_t,\\bm{w})\\right]^2 = -2\\cdot \\left[G_t - \\hat{v}(S_t,\\bm{w})\\right] \\nabla_{\\bm{w}}\\hat{v}(S_t,\\bm{w})$$\n\tWhich leads us to the \\textbf{Gradient Monte Carlo} algorithm:\n\t$$\\bm{w}_{t+1}=\\bm{w}_{t}+\\alpha \\left[G_t - \\hat{v}(S_t,\\bm{w})\\right] \\nabla_{\\bm{w}}\\hat{v}(S_t,\\bm{w})$$\n\t\\item Alternatively, we could also think about using the bootstrapping estimate as target, which gives us the following update rule:\n\t$$\\bm{w}_{t+1}=\\bm{w}_t + \\alpha\\underbrace{\\left[R_{t+1} + \\gamma\\hat{v}(S_{t+1},\\bm{w}_t) - \\hat{v}(S_t,\\bm{w}_t)\\right]}_{\\text{TD error }\\delta}\\nabla_{\\bm{w}}\\hat{v}(S_t,\\bm{w}_t)$$  \n\tThis method is called \\textbf{Semi-gradient TD(0)}, and indicates by its name that it is not a true gradient. The reason for that is that we actually ignore the dependency of the target on $\\bm{w}$. We assume it to be fixed.  Nevertheless, experiments with the true gradient have shown that the semi-gradient works much better in practice. We will discuss it later in more detail.\n\t\n\tNote that as in the value-based, we can extend this approach to $n$-step if wanted.\n\t\\item When comparing Gradient MC and semi-gradient TD(0), we get the same arguments as for the tabular case in Section~\\ref{sec:value_based_tabular_difference_TD_MC} $\\Rightarrow$ TD has lower variance and learns usually faster, but can have a bias (see below)\n\t\\item A thing to keep in mind when using semi-gradient TD(0) is that it tries to minimize distance between close-by states, especially if we take approximation like aggregating multiple state into the same. This is because a small step can lead to a huge TD error which we try to minimize. \n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/rl_approximate_value_based_semi_gradient_td.png}\n\t\t\\caption{Problems of semi-gradient TD(0) updates on the random walk example. It prefers a value function with low changes between states, so that it gets a biased prediction.}\n\t\t\\label{fig:rl_approximate_value_based_semi_gradient_td}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{Discussion on convergence for different objectives}\n\\begin{itemize}\n\t\\item The advantages of linear function approximations are that the gradients are easy to calculate ($\\nabla_{\\bm{w}}\\hat{v}(s,\\bm{w})=\\bm{x}(s)$). Furthermore, it can be proven that all local optima are global optima, so that gradient MC converges to the minimum of $\\overline{\\text{VE}}$. This is not necessarily the case for semi-gradient TD but we can define a upper bound $\\overline{\\text{VE}}(\\bm{w}_{td})\\leq \\frac{1}{1-\\gamma}\\min_{\\bm{w}}\\overline{\\text{VE}}(\\bm{w})$ and guarantee that it converges. \n\t\n\tIn the non-linear case, we cannot guarantee convergence for semi-gradient TD (but for Gradient Monte Carlo), and we might end up in local optima. Nevertheless, linear features are much more restricted than non-linear as neural networks. Hence, non-linear methods can lead to better results, even if we might get stuck in local optima.\n\t\n\t\\item When learning via Gradient Monte Carlo or Semi-gradient TD, we have to select a step size $\\alpha$. This can be a bit more tricky here because we combine the values of many states into a single function. In case of linear function approximation, we can actually give a rule of thumb because the features are static, namely:\n\t$$\\alpha = (\\tau\\E[\\bm{x}^T\\bm{x}])^{-1}$$\n\twhere $\\tau$ is the number of experiences we expect to have for the same (or similar) feature vector to average over (as say the learning rate you would choose for the tabular setting would be $\\frac{1}{\\tau}$)\n\t\\item Alternatively we could try to find the fix point of semi-gradient TD(0). We can write the TD update rule for linear function approximation as:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\bm{w}_{t+1} & = \\bm{w}_t + \\alpha \\left(R_{t+1}+\\gamma \\bm{w}_t^T \\bm{x}_{t+1} - \\bm{w}_t^T\\bm{x}_t\\right)\\bm{x}_t\\\\\n\t\t\t& = \\bm{w}_t + \\alpha \\left(R_{t+1}\\bm{x}_t - \\bm{x}_t (\\bm{x}_t - \\gamma \\bm{x}_{t+1})^T \\bm{w}_t\\right)\\\\\n\t\t\t\\implies \\E[\\bm{w}_{t+1}|\\bm{w}_t] & = \\bm{w}_t + \\alpha \\left(\\underbrace{\\E[R_{t+1}\\bm{x}_t]}_{\\bm{b}} - \\underbrace{\\E[\\bm{x}_t (\\bm{x}_t - \\gamma \\bm{x}_{t+1})^T]}_{\\bm{A}} \\bm{w}_t\\right)\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\tThe fix point is given when we do not change our weights anymore, meaning $\\bm{w}_{t+1}=\\bm{w}_t$. This is the case if:\n\t$$\\bm{w}_{td}=\\bm{A}^{-1}\\bm{b}$$\n\tWe can approximate $\\bm{A}$ and $\\bm{b}$ by MC sampling:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\hat{\\bm{A}}_t & = \\sum_{k=0}^{t-1}\\bm{x}_k\\left(\\bm{x}_k - \\gamma\\bm{x}_{k+1}\\right)^T + \\epsilon\\bm{I}\\\\\n\t\t\t\\hat{\\bm{b}}_t & = \\sum_{k=0}^{t-1}R_{k+1}\\bm{x}_k\n\t\t\\end{split}\n\t\\end{equation*}\n\twhere $\\epsilon$ is a small constant ensuring that $\\bm{\\hat{A}}$ is always invertible. This solution is called \\textbf{least-squares temporal-difference (LSTD)}, and is usually more sample efficient because we do not have to perform iterative updates, and has the benefit of not requiring a step size. However, it is more computationally expensive (quadratic plus the invert of $\\bm{A}$), and we cannot adapt to a change in the environment over time (once performed, we fix our weights)\n\\end{itemize}\n\\subsection{Control with approximation}\n\\begin{itemize}\n\t\\item For learning a policy $\\pi$, we again change our objective to learning the $q$-values, which we now approximate with $\\hat{q}(s,a,\\bm{w})$. We will focus on episodic cases, but note that everything could be generalized to the continuous case as well.\n\t\\item In the \\textbf{on-policy} case, we can use methods like (episodic) semi-gradient SARSA, so that our update step is:\n\t$$\\bm{w}_{t+1}=\\bm{w}+\\alpha\\left[U_t - \\hat{q}(S_t,A_t,\\bm{w})\\right]\\nabla_{\\bm{w}}\\hat{q}(S_t,A_t,\\bm{w})$$\n\twhere $U_t$ is our target, which is for one-step SARSA $U_t = R_t + \\gamma \\hat{q}(S_{t+1},A_{t+1},\\bm{w})$.\n\tAs usual, we iterate over this update rule while setting our policy to $\\epsilon$-greedy on $\\hat{q}$.\n\t\\item In the \\textbf{off-policy} case, we experience more problems. As we have a behavior policy $b$ and target policy $\\pi$, we often need to use importance weight to correct the target of the update:\n\t$$\\bm{w}_{t+1}=\\bm{w}_{t}+\\rho \\alpha\\left[U_t - \\hat{q}(S_t,A_t,\\bm{w})\\right]\\nabla_{\\bm{w}}\\hat{q}(S_t,A_t,\\bm{w})$$\n\tAlthough it increases variance, it is sometimes necessary to guarantee unbiased, correct estimates. However, note that in cases like Q-learning, where $U_t$ is independent of $b$, we might not have to consider the importance weights as $\\rho=1$.\n\t\n\tThe second issue is that we need to take the changed state distribution, $\\mu_b$, into account. Consider for example the very simple MDP in Figure~\\ref{fig:rl_approximation_value_based_offpolicy_divergence} for which we just want to estimate $v$ (policy evaluation). We assume the reward for any action to be $0$, and start with an initial value of $w=10$.\n\t\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.2\\textwidth]{figures/rl_approximation_value_based_offpolicy_divergence.png}\n\t\t\\caption{Simple MDP where off-policy updates can diverge. Green indicate behavior policy, red the target. Every transition has a reward of $0$, meaning that the optimal $v$ are 0 at both states.}\n\t\t\\label{fig:rl_approximation_value_based_offpolicy_divergence}\n\t\\end{figure}\n\t\n\tFirst, consider the on-policy case where $\\pi=b$ (green action from second state). Then, we would alternate between the two update equations:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\\text{Left to right: }\\hspace{2mm}w_{t+1} & = w_t + \\alpha (\\gamma \\cdot 2w_t-w_t)\\nabla_w w_t = (1+\\alpha(2\\gamma-1)) w_t\\\\\n\t\t\\text{Right to left: }\\hspace{2mm}w_{t+1} & = w_t + \\alpha (\\gamma w_t-2w_t)\\nabla_w 2w_t = (1+2\\alpha(1-2\\gamma)) w_t\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\tOverall, we would converge to $w=0$ as the right to left update is twice as high as the other.\n\t\n\tNow, assume the behavior policy stays the same, but our target policy stays at the second state. Then, the importance weight for left to right is 1 (as both policies do that with probability 1), but from right to left is zero because we would not take this action with our target policy. So we end up with the update:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\text{Left to right: }\\hspace{2mm}w_{t+1} & = w_t + \\alpha (\\gamma \\cdot 2w_t-w_t)\\nabla_w w_t = (1+\\alpha(2\\gamma-1)) w_t\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\twhich makes $w_t$ head to infinity if $\\gamma>0.5$. This shows that off-policy prediction can diverge!\n\t\n\t\\item This divergence can occur when the following three methods are used together (\\textit{Deadly Triad}):\n\t\\begin{itemize}\n\t\t\\item Function approximation\n\t\t\\item Semi-gradient bootstrapping\n\t\t\\item Off-policy training\n\t\\end{itemize}\n\t\\item To overcome this issue, we need to consider alternatives to semi-gradients.\n\t\n\\end{itemize}\n\\subsubsection{Alternatives to semi-gradients}\n\\begin{itemize}\n\t\\item There are couple of objectives that we can use instead of semi-gradient. We visualize all of them in Figure~\\ref{fig:rl_approximation_value_based_different_errors}, and discuss them here in detail\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.6\\textwidth]{figures/rl_approximation_value_based_different_errors.png}\n\t\t\\caption{Geometry of linear value-function approximation. We show an approximation of a 3D state space by a two dimension weight vector.}\n\t\t\\label{fig:rl_approximation_value_based_different_errors}\n\t\\end{figure}\n\t\n\t\\item Before we start our discussion, we need to introduce some notation:\n\t\\begin{itemize}\n\t\t\\item First, we need to consider how we measure distance between two value functions. The standard euclidean norm is not sufficient, as we give importance to different states. This is why we take $\\mu$ into account:\n\t\t$$||v_1-v_2||_{\\mu}^2 = \\sum_{s\\in\\mathcal{S}} \\mu(s)\\left[v_1(s)-v_2(s)\\right]^2$$\n\t\t\\item Given the norm, we also want to define a \\textit{projection operator} which assign to an arbitrary $v$ (over whole state space) the closest value function based on the norm that can be represented:\n\t\t$$\\Pi v = v_{\\bm{w}}\\hspace{3mm}\\text{where}\\hspace{3mm}\\bm{w}=\\arg\\min_{\\tilde{\\bm{w}}}||v-v_{\\tilde{\\bm{w}}}||_{\\mu}^2 $$\n\t\t\\item The last notation we want to introduce is the Bellman operator, which maps a value function $v$ to its bootstrapping estimates:\n\t\t$$(B_{\\pi}v_{\\bm{w}})(s) = \\sum_a \\pi(a|s)\\sum_{s',r}p(s',r|s,a)[r+\\gamma v_{\\bm{w}}(s')] = v_{\\bm{w}}(s) + \\overline{\\delta}_{\\bm{w}}(s)$$\n\t\twith $\\overline{\\delta}_{\\bm{w}}(s)$ being the expected TD error for state $s$.\n\t\\end{itemize}\n\t\\item The value error $\\overline{\\text{VE}}$ is minimized if the norm is the lowest to $v_{\\pi}$: $$\\min_{\\bm{w}} \\overline{\\text{VE}}(\\bm{w}) = \\min_{\\bm{w}} ||v_{\\bm{w}}-v_{\\pi}||_{\\mu}^2 $$\n\tThe projected point, which we can actually reach, is $\\Pi v_{\\pi}$ which is the best point we can represent in our $\\bm{w}$-space. Gradient Monte Carlo methods converge to this point, but mostly quite slowly\n\t\\item Without the approximation, we could simply apply the Bellman operator  over and over again, and reach $v_{\\pi}$ (as in tabular TD(0) learning) which is the gray line above. However, we cannot represent the change so that we have to project $v$ after each step: $\\Pi B_{\\pi}v_{\\bm{w}}$. The step we take in between is the projected Bellman error $PBE=\\Pi\\delta_{\\bm{w}}$ \n\t\n\tSemi-gradient TD is converging to the point where $PBE=0$ as we reach a fix-point there. However, this does not have to be where the minimum Bellman error is reached because imagine $\\delta_{\\bm{w}}$ being orthogonal to $\\bm{w}$-subspace. Then, the projected bellman error is 0, but without projection, we would continue changing $\\bm{w}$, until we reach $\\min \\overline{\\text{BE}}$.\n\t\n\tAt the same time, even if we would reach $\\min \\overline{\\text{BE}}$, it would most likely not be a optimum (i.e. gradients greater than zero) because the gradients can point to outside the representable $\\bm{w}$-space (does not need to be orthogonal as before), and hence the projected Bellman error can be unequal to zero.\n\t\n\t\\item The last objective we consider here is the true-gradient TD error, meaning: $$\\overline{\\text{TDE}}(\\bm{w})=\\sum_{s\\in\\mathcal{S}}\\mu(s)\\E\\left[\\delta_t^2 |S_t=s,A_t\\sim \\pi\\right] = \\E_{b}[\\rho_t \\delta_t^2] \\hspace{4mm}\\text{(if we assume $\\mu$ is under $b$)}$$\n\tFollowing SGD updates, we get:\n\t$$\\bm{w}_{t+1}=\\bm{w}_t + \\alpha \\rho_t \\delta_t (\\nabla \\hat{v}(S_t,\\bm{w}_t) - \\gamma \\nabla \\hat{v}(S_{t+1},\\bm{w}_t))$$\n\t\n\t\\item Now, let's consider which of these objectives we can take as alternative to semi-gradient updates. \n\t\n\tA major drawback of TDE is that we also take the gradients regarding the next steps, which can push the value function in a wrong direction (tries to minimize distance between the steps, similar to Figure~\\ref{fig:rl_approximate_value_based_semi_gradient_td}). Consider the MDP in Figure~\\ref{fig:rl_approximation_value_based_TDE}, with on-policy evaluation of a uniform policy, and $\\gamma=1$.\n\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.2\\textwidth]{figures/rl_approximation_value_based_TDE.png}\n\t\t\\caption{Simple example where TDE gives an undesirable result.}\n\t\t\\label{fig:rl_approximation_value_based_TDE}\n\t\\end{figure}\n\n\tThe optimal/correct value function is obviously $v(A)=1/2, v(B)=1, v(C)=0$. The TD error is given by:\n\t$$\\delta_t=\\frac{1}{2}\\left(\\left[v(B)-v(A)\\right]^2 + \\left[1-v(B)\\right]^2\\right)+\\frac{1}{2}\\left(\\left[v(C)-v(A)\\right]^2 + \\left[0-v(C)\\right]^2\\right)$$\n\tfor which the optimal is actually $v(A)=1/2, v(B)=3/4, v(C)=1/4$ because we also minimize the distance between $v(A)$ and $v(B)$, and similarly between $v(A)$ and $v(C)$.\n\t\n\t\\item For calculating the Bellman error ($\\min \\text{BE}$), we need to calculate:\n\t$$\\overline{\\text{BE}}=||\\overline{\\delta}_{\\bm{w}}||_{\\mu}^2\\hspace{2mm}\\text{where}\\hspace{2mm}\\overline{\\delta}_{\\bm{w}}=\\E_{\\pi}\\left[\\delta_{\\bm{w}}|S_t=s,A_t\\sim\\pi\\right]$$\n\tAs we have the square in the error, to guarantee an unbiased estimate, we need to sample at least two times independently (otherwise we estimate $\\E[\\delta^2]$ instead of $\\E[\\delta]^2$). This is mostly not possible in interactions to obtain, or makes the algorithm rather slow.\n\t\n\t\\item The last remaining objective is the mean squared projected Bellman error $\\overline{\\text{PBE}}$. We can derive at the following gradient for PBE:\n\t$$\\nabla_{\\bm{w}} \\overline{\\text{PBE}}(\\bm{w}) = 2\\E[\\rho_t (\\gamma \\bm{x}_{t+1}-\\bm{x}_t)\\bm{x}_t^T]\\E[\\bm{x}_t\\bm{x}_t^T]^{-1}\\E[\\rho_t\\delta_t\\bm{x}_t]$$\n\tUsing the same samples for all the expectations gives the same bias as the one for the Bellman error. What we can do, however, is learning some factors from all the data, namely the last two, and denote it as $\\bm{v}_t=\\E[\\bm{x}_t\\bm{x}_t^T]^{-1}\\E[\\rho_t\\delta_t\\bm{x}_t]$. Then, we can perform SGD as:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\bm{v}_{t+1} & = \\bm{v}_t + \\beta \\rho_t (\\delta_t - \\bm{v}_t^T \\bm{x}_t)\\bm{x}_t\\\\\n\t\t\t\\bm{w}_{t+1} & = \\bm{w}_t + \\alpha \\left[\\rho_t (\\gamma \\bm{x}_{t+1}-\\bm{x}_t)\\bm{x}_t^T\\right]\\bm{v}_t\n\t\t\\end{split}\n\t\\end{equation*}\n\tThis algorithm is called GTD2 (gradient TD) which converges to the minimum PBE for linear features. The drawbacks are that we need an additional learning rate $\\beta$ (mostly greater than $\\alpha$), and need to store two parameter updates.\n\t\n\tNevertheless, as we have a guarantee of convergence for all settings, it makes GTD2 the preferred technique compared to Semi-gradient TD, except when we just want a simple method.\n\t\\item Overall, we have the following convergence properties:\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.6\\textwidth]{figures/rl_approximate_value_based_convergence_overview.png}\n\t\t\\caption{Overview of convergence properties of different optimization methods. The columns show the setting (on=on-policy, off=off-policy). \"No C.\" for semi-gradient TD means that we cannot guarantee its convergence. \"N.A\" stands for \"not applicable\", as LSTD is based on the assumption of linear features.}\n\t\\end{figure}\t\n\\end{itemize}\n\\subsubsection{Deep Q network}\n\\begin{itemize}\n\t\\item Another way of stabilizing off-policy control is by using many additional tricks, to make it more similar to supervised learning. One popular example of this is the DQN\n\t\\item Given a state as input, we try to learn a $q$-value for each output, so that we can perform a simple maximization step over the outputs to get the optimal policy\n\t\\item We use image as input. However, to detect movement, a couple of frames are stacked on top of each other\n\t\\item To guarantee i.i.d. samples within a batch, and use data more efficiently (look at an experience more than once), we use \\textbf{experience replay}:\n\t\\begin{itemize}\n\t\t\\item All the experiences we had from interacting with the environment are stored in a buffer (if limited size, use FIFO queue)\n\t\t\\item At every time step, randomly select $N$ experiences which form a batch for training\n\t\\end{itemize}\n\tNote that this is only possible because of off-policy training, as the collected experiences come from a different policy, namely an older one.\n\t\\item Another trick to stabilize learning is \\textbf{fixing the target}. As we use semi-gradient version of $q$-learning, which is:\n\t$$\\bm{w}_{t+1}\\leftarrow \\bm{w}_t + \\alpha \\left[R_{t+1}+\\gamma\\max_a \\hat{q}(S_{t+1}, a, \\bm{w}_t) - \\hat{q}(S_t, A_t, \\bm{w}_t)\\right]\\nabla \\hat{q}(S_t, A_t, \\bm{w}_t)$$\n\tTo fix the target, we copy the weights $\\tilde{\\bm{w}}$, and use this to calculate the target $\\gamma\\max_a \\hat{q}(S_{t+1}, a, \\bm{w}_t)$.\n\t\n\tFurthermore, it has been shown to work well to clip the TD error between a range of $[-1,1]$ to prevent any divergence issues. \n\t\\item If needed/wanted, we can overcome the maximization by using a double Q-learning approach\n\\end{itemize}\n\n\n\n\n\n"
  },
  {
    "path": "Reinforcement_Learning/rl_mcts_alpha_go.tex",
    "content": "%\\section{Monte-Carlo Tree Search and Alpha Go}\n%\\label{sec:MCTS_Alpha_Go}\n%\\textit{This section reviews the lecture slides 12.}\n%\\begin{itemize}\n%\t\\item In Section~\\ref{sec:value_based_approximation}, we have seen that to learn a value function for problems with very large state space, we can approximate our $q$-function by e.g. a neural network. However, these approximations will always contain a certain amount of noise/inaccuracy.\n%\t\\item An alternative approach is to learn $q(s,a)$ \\underline{online}. The simplest approach for this is to perform rollouts \n%\\end{itemize}\n"
  },
  {
    "path": "Reinforcement_Learning/rl_model_based.tex",
    "content": "\\section{Model-based Reinforcement Learning}\n\\label{sec:model_based}\n\\textit{This section reviews the lecture slides 11 and 12.}\n\\begin{itemize}\n\t\\item We have seen that given the environment dynamics, we can find the optimal policy by dynamical programming. All the methods after that purely learned from interactions. We now want to take a step in between and try to learn the model dynamics itself, $p(s'|s,a)r(s,a,s')$. If we have that, we could plan by simulating in our learned model.\n\t\\item There are several benefits of this approach:\n\t\\begin{itemize}\n\t\t\\item We don't require interactions with the environment, but can generate new data from simulation. This is especially helpful when real-time data is expensive (whether in time, computational resources, etc.) as in real-life robotic systems (takes a long time for a single rollout)\n\t\t\\item We can obtain probability distributions which tell us how likely we end up in a state when we take a certain action. This can be very helpful in some cases, as e.g. in the (slippery) cliff world example, we would know how likely it is that we actually fall of the cliff even if we take the right action. \n\t\\end{itemize}\n\tHowever, when these things are not required, model-free methods mostly work better and/or are computationally cheaper/simpler. Furthermore, we prevent any bias we might get when our model is inaccurate.\n\t\\item In general, we distinguish between three types of systems we can have that tries to imitate the real environment:\n\t\\begin{itemize}\n\t\t\\item A \\textbf{full} or \\textbf{distributional model} is a full description of all transition probabilities and rewards. \n\t\t\\item A \\textbf{sample} or \\textbf{generative model} can be viewed as a black-box simulator, where given any state $s$ and action $a$, it can sample a reward $r_t$ and a next state $s'$.\n\t\t\\item A \\textbf{trajectory} or \\textbf{simulation model} can simulate whole episodes, but is not able to start at any state and action. This is for example the case for a physical model where we cannot start with an arbitrary velocity.\n\t\\end{itemize}\n\tThese three models can be seen as generalization steps. The most limited implementation is the trajectory one. If we provide the ability of changing the start state to any arbitrary state, we arrive at the generative model. Adding the probabilities $p(s',r|s,a)$ gives us in the end the distributional model.\n\t\\item There are several ways of implementing this. We will consider here a simple method, called Dyna\n\\end{itemize}\n\\subsection{Dyna-Q}\n\\begin{itemize}\n\t\\item Dyna makes two assumptions of the environment:\n\t\\begin{enumerate}\n\t\t\\item Our environment is deterministic, meaning that any transition probabilities are either 1 or 0. \n\t\t\\item The state and action space is discrete and limited, so that we can store it in a tabular setting.\n\t\\end{enumerate}\n\tNote that we can relax the first requirement slightly by storing e.g. how often we came from one state-action pair to another state.\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/rl_model_based_dyna_Q.png}\n\t\t\\caption{Real experience is generated by the interaction of the agent (according to the policy) and the environment. This real data is used to update our policy (direct RL), but at the same time update our model, from which we can generate new samples to learn from (indirect RL).}\n\t\t\\label{fig:rl_model_based_dyna_Q}\n\t\\end{figure}\n\t\\item The general overview of the idea is shown in Figure~\\ref{fig:rl_model_based_dyna_Q}. We have two sources from which we train our policy and/or value function: direct and indirect. The samples from the real environment are used to perform \"direct Reinforcement Learning\" as we use the actual samples to learn. At the same time, we use the real samples to update our model, and can generate from there as many samples as we want.\n\t\\item Written down as an algorithm, we arrive at Figure~\\ref{fig:rl_model_based_dyna_Q_algorithm}. The parameter $n$ which specifies how often we train from the simulated/learned model compared to the real environment, is a hyperparameter and depends on the access of the environment (how expensive is it, etc.). However, it is usually $n\\gg 1$.\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.65\\textwidth]{figures/rl_model_based_dyna_Q_algorithm.png}\n\t\t\\caption{Example implementation of the algorithm of Dyna-Q. We use a tabular-based setting to store transition and rewards. Note that we will discuss the design choices in this part more in detail.}\n\t\t\\label{fig:rl_model_based_dyna_Q_algorithm}\n\t\\end{figure}\n\t\\item Looking at Figure~\\ref{fig:rl_model_based_dyna_Q_algorithm}, we see that there are still quite a few design decisions to make, which we will go through step-by-step:\n\t\\begin{enumerate}\n\t\t\\item Step (e) - How do we learn our model?\n\t\t\\item Step (f) - When should we use our simulated environment? Can we also use it somewhere else than the update step?\n\t\t\\item Step (f) loop - Which state and action should we choose to update?\n\t\t\\item Step (f) loop - How do we update e.g. our values or policy?\n\t\\end{enumerate} \n\\end{itemize}\n\\subsubsection{How to learn the model}\n\\begin{itemize}\n\t\\item As mentioned previously, Dyna uses a tabular setting to learn the model. Hence, we have a table over state and actions, where an entry contains the information of the next state and reward\n\t\\item In case we have stochastic transitions, we have to slightly adjust our table. We now create a table over $(s,a,s')$ where we store how often we experienced this transition, and what reward we got. In case we also have stochastic rewards, we need to extend the table further.\n\t\n\tWhen sampling, we first have to normalize the probabilities for every $s'$ (and $r$) to occur when $(s,a)$ is given, and finally sample from this distribution.\n\t\n\tAn example table is shown below\n\t\\begin{table}[ht!]\n\t\t\\centering\n\t\t\\begin{tabular}{c|cc}\n\t\t\t& \\textit{State 1} & \\textit{State 2}\\\\\n\t\t\t\\hline\n\t\t\t\\textit{State 1, Action 1} & $\\eta_{111}=1, r_{111}=2$ (50\\%) & $\\eta_{112}=1, r_{112}=-1$ (50\\%)\\\\\n\t\t\t\\textit{State 1, Action 2} & $\\eta_{121}=5, r_{121}=5$ (100\\%) & $\\eta_{122}=0, r_{122}=0$ (0\\%) \\\\\n\t\t\t\\textit{State 2, Action 1}  & $\\eta_{211}=4, r_{211}=-4$ (80\\%) & $\\eta_{212}=1, r_{212}=2$ (20\\%)  \\\\\n\t\t\t\\textit{State 2, Action 2} & $\\eta_{221}=4, r_{221}=-2$ (40\\%) & $\\eta_{222}=6, r_{222}=1$ (60\\%) \\\\\n\t\t\\end{tabular}\n\t\t\\caption{Example table for stochastic transitions and deterministic rewards.}\n\t\\end{table}\n\\end{itemize}\n\\subsubsection{What to update}\n\\begin{itemize}\n\t\\item To prevent that we spend too much computational effort on state-action pairs that are not relevant for the current/optimal policy, we should make smarter selections\n\t\\item One approach is \\textbf{prioritized sweeping} where we prefer these state-action pairs that lead to a state for which we just have experienced an update (whether with real or simulated experience)\n\t\\begin{itemize}\n\t\t\\item The priority in the queue is given by the TD error we would get at the time we add the state in the queue. This supports that states with high errors, i.e. wrong estimates, are updated first.\n\t\t\\item To limit the queue, we can define a threshold $\\theta$ over which the TD error has to be to add a state in the queue\n\t\t\\item In the simulation step (indirect RL), we perform updates based on the queue until either the queue is empty, or we reached a maximum of $n$ steps. If the queue is non-empty, it is kept for the next iteration as well.\n\t\t\\item For this model, we require at least a sample/generative model because we need to be able to start at any state-action pair\n\t\t\\begin{figure}[ht!]\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=0.5\\textwidth]{figures/rl_model_based_dyna_prioritized_sweeping.png}\n\t\t\t\\caption{Algorithm of Prioritized Sweeping.}\n\t\t\\end{figure}\n\t\\end{itemize}\n\t\\item An alternative is performing \\textbf{trajectory sampling} where we start from the start state (or sample one if multiple exist), and follow our current policy. \n\t\\begin{itemize}\n\t\t\\item While updating the more frequently visited states, we have the disadvantage of limited exploration because we highly focus on states of our distribution\n\t\t\\item Hence, if we have a (close to) deterministic environment, trajectory sampling might work well, but in a stochastic environment where we continuously have to explore, it might perform worse than uniformly sampling any state-action pair \n\t\t\\item For this method, we only require a trajectory model, making it less complex\n\t\\end{itemize}\n\\end{itemize}\n\\subsubsection{How to update}\n\\begin{itemize}\n\t\\item For updating our $q$/$v$/policy, we can use any of the methods we have discussed before. \n\t\\item However, remember that for some methods like dynamic programming, we can make use of the model dynamics. Nevertheless, this might not be the most efficient computation, especially when we have many possible next steps. Remember that we have to take \\underline{all} next states into account for dynamic programming, although some can be neglected if we have a small weight on them. \n\t\\item In addition, we would require the full/distributional model to perform these kind of updates, whereas the others either work with generative or trajectory models\n\\end{itemize}\n\\subsubsection{When to plan}\n\\begin{itemize}\n\t\\item Currently, we only use the environment to generate new samples for training\n\t\\item Knowing the system dynamics can however be valuable in more than this situation. For example, we can easily plan ahead by trying out different actions, and observing the reward in simulation. Afterwards, we take the actions in the real world which gave the best result in the simulation\n\t\\item This idea is used in Monte Carlo Tree Search algorithms, which we will discuss in more detail in Section~\\ref{sec:MCTS_Alpha_Go}. \n\\end{itemize}\n\\subsection{Model-based policy search}\n\\begin{itemize}\n\t\\item In the previous discussion, we mainly focused on value-based updates. However, we could of course use policy-based methods as well.\n\t\\item Again, the decision of whether to use policy-based or value-based methods is based on multiple decisions. For example, if we need to learn a stochastic policy, or we have continuous actions, then we might want to use policy-based methods. In the case that we have discrete actions and aim for learning a deterministic, greedy policy, value-based methods are more suited because for policy gradient we require the policy to be smoothly changeable/differentiable.\n\t\\item Let's assume that the reward is known for now (e.g. we have defined the reward for a problem by our own). Then, we could reformulate the transition function as:\n\t$$s_{t+1} = f(s_t, a_t) + w$$\n\twhere $f(s_t, a_t)$ is a deterministic function that maps a state-action pair to a new state, and $w$ is additive noise (e.g. Gaussian for continuous states). To ensure this formulation to work well, we would require a mostly deterministic environment, as otherwise $f(s_t, a_t)$ cannot model the different outcomes.\n\t\\item We can then train by\n\t\\begin{enumerate}\n\t\t\\item Use any model-free policy-based approach (e.g. TRPO or DDPG) to learn from the real world.\n\t\t\\item Use the extra information from the environment, e.g. for richer gradient information\n\t\t\\begin{itemize}\n\t\t\t\\item As we now model the transition function $s_{t+1}\\approx f(s_t, a_t)$, we know how the next state will change if we change our parameters.\n\t\t\t\\item This allows us to look at the gradients over $s'$ and $a$, and find the best action more easily\n\t\t\t\\item When we perform backpropagation through $v_{\\theta}(s_t)$, we can (instead of sampling) also derive rewards because they are a simple function depending on $s_{t+1}$, or in $f(s_t,a_t)$. Hence, we can write:\n\t\t\t$v_{\\theta}(s_t)=\\nabla_{\\theta} r_{t+1}+\\gamma \\nabla_{\\theta} r_{t+2}+...$\n\t\t\\end{itemize}\n\t\\end{enumerate}\n\\end{itemize}\n\\subsection{Monte-Carlo Tree Search and Alpha Go}\n\\label{sec:MCTS_Alpha_Go}\n\\begin{itemize}\n\t\\item In Section~\\ref{sec:value_based_approximation}, we have seen that to learn a value function for problems with very large state space, we can approximate our $q$-function by e.g. a neural network. However, these approximations will always contain a certain amount of noise/inaccuracy.\n\t\\item An alternative approach is to learn $q_{\\pi}(s,a)$ \\underline{online}. The simplest approach, when we have given our model, is to perform a couple of rollouts from the state $s_t$ with our current policy $\\pi$. $q_{\\pi}(s,a)$ can then be estimated by the mean of the experienced returns $G_t$.\n\t\n\tPlaying the best action based on this estimate (e.g. estimated $q_{\\pi}(s_t,a_1)$, $q_{\\pi}(s_t,a_2)$, etc.) is guaranteed to be at least as good as $a\\sim \\pi(s)$ as if $\\pi$ was the optimal policy, $a$ will also be the argmax of the estimate (in expectation).\n\\end{itemize}\n\\subsubsection{Monte-Carlo Tree Search}\n\\begin{itemize}\n\t\\item If we have given a full model description including the dynamics, we could simply expand the previous approach by taking all possible futures into account. However, this is less likely to work for games like Go because there are a huge number of possible outcomes (a full game tree has about $10^{170}$ different states). \n\t\\item Nevertheless, a lot of this computation might not be necessary. Instead, we can focus on the most likely subtree which only contains a small selection of possible outcomes. This leads us to the Monte-Carlo Tree Search algorithm\n\t\\item In MCTS, we build a tree incrementally by performing 4 steps for $n$ steps (limited by computational resources, time, etc.), visualized in Figure~\\ref{fig:rl_model_based_MCTS}:\n\t\\begin{enumerate}\n\t\t\\item \\textbf{Selection}\n\t\t\n\t\tGiven a subtree, we need to decide at which point we want to expand it. This is defined by our \\textit{tree policy} $\\pi_{\\text{tree}}$, and can be for example the upper confidence bound (similarly to choosing the next action in a bandit setting):\n\t\t$$\\pi_{\\text{tree}}(s)=\\arg\\max_{a} \\left[Q(s,a)+c\\sqrt{\\frac{\\ln N(s)}{N(s,a)}}\\right]$$\n\t\twith $N(s)$ as the number of visits in $s$, and $N(s,a)$ the number of times we took $a$ in $s$. We continue our policy until we end up at a leaf node.\n\t\t\\item \\textbf{Expansion}\n\t\t\n\t\tAfter deciding at which node we want to \"grow\" the tree, we need to expand it. This means that we add a new leaf, which is an action in case of $q$, or a state in case of $v$ (we always add a state-action pair, just ordering is different). We initialize it with the values $N(s)=0$, $N(s,a)=0$ in case of UCB.\n\t\t\\item \\textbf{Simulation}\n\t\t\n\t\tFrom the newly added node, we perform a rollout. This means that starting from the leaf node, we interact with the environment according to the current policy $\\pi$ until terminating. \n\t\t\\item \\textbf{Backup}\n\t\t\n\t\tAfter finishing simulation, we update our estimates based on the newly observed return. Note that we update the $q$/$v$-values for each node which led to the leaf, while taking a possible discount factor $\\gamma$ into account. In case of UCB, this means that we increase $N(s)$ and $N(s,a)$ by one, as well as adding a new point to $Q$ for averaging (e.g. use running average).\n\t\\end{enumerate}\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.6\\textwidth]{figures/rl_model_based_MCTS.png}\n\t\t\\caption{Visualization of the four steps in MCTS: Selection, Expansion, Simulation, Backup.}\n\t\t\\label{fig:rl_model_based_MCTS}\n\t\\end{figure}\n\t\\item Note that we have seen two requirements of MCTS along the way: (1) we need a generative model where we can start at any $(s,a)$. Alternatively, if our environment is deterministic, we can also make trajectory models working by playing the same actions again from the top. (2) we assume $q(s,a)$ to be storable in a table\n\t\\item If we want to use MCTS for planning, we can choose our our policy based on $\\pi(a|s)\\propto N(s,a)^{1/\\tau}$. Note that we don't take the policy according to the $q$ values because they are estimates, and potentially very noisy as we have different amount of samples for each action.\n\t\\item After taking a step, we can reuse the selected branch of the tree, and don't have to start from scratch again\n\\end{itemize}\n\\subsubsection{AlphaGo Zero}\n\\begin{itemize}\n\t\\item For estimating the $q$-values, MCTS uses full Monte Carlo samples. However, we know that we can also use TD learning for it, meaning we bootstrap our estimates. Using this idea, two separate networks were used in Alpha Go: a policy network $\\pi_{\\theta}(a|s)$, and a value network $v_{\\theta}(s)$\n\t\\item We can now look at the changes AlphaGo makes in the MCTS algorithm:\n\t\\begin{enumerate}\n\t\t\\item \\textbf{Selection}\n\t\t\n\t\tWe define our tree policy as:\n\t\t$$\\pi_{\\text{tree}}(a|s)=\\arg\\max_a \\left[Q(s,a)+cU(s,a)\\right], \\hspace{5mm}U(s,a)=\\frac{\\pi_{\\theta}(a|s)}{1+N(s,a)}$$\n\t\tNote that this is a solution found empirically as we sum $q$-values and probabilities.\n\t\t\n\t\t\\item \\textbf{Expansion}\n\t\t\n\t\tWhen reaching a leaf node, we evaluate the value network $v_{\\theta}(s)$ for this specific state, and expand the state by all its possible actions.\n\t\t\n\t\t\\item \\textbf{Simulation}\n\t\t\n\t\tIn the original AlphaGo, we randomly choose between using $v_{\\theta}(s)$, or simulating by performing a rollout. However, in the newer version AlphaGo zero, we fully rely on $v_{\\theta}(s)$.\n\t\t\n\t\t\\item \\textbf{Backup}\n\t\t\n\t\tUsing $v_{\\theta}(s)$, we update all $q$ values above.\n\t\\end{enumerate}  \n\t\\item Our policy network $\\pi_{\\theta}$ limits our search in width because we select values based on its prior. The value network $v_{\\theta}(s)$ limits the search in depth because we don't have to sample anymore.\n\t\\item This approach is working well, if we have (1) a discrete state space, (2) a fully observable environment, and (3) a deterministic environment.\n\t\\item We train the network by self-play. The policy network tries to predict the outcome of the tree search (how often will we choose action $a$ at state $s$), and the value network tries to predict the return we get after the full rollout. See Figure~\\ref{fig:rl_model_based_alphago_zero_selfplay} for a visualization of the self-play learning.\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/rl_model_based_alphago_zero_selfplay.png}\n\t\t\\caption{Self-play RL in AlphaGo zero.}\n\t\t\\label{fig:rl_model_based_alphago_zero_selfplay}\n\t\\end{figure}\n\t\\item Nevertheless, the training might not be 100\\% stable. In a small amount of times, it can happen that the network diverges. To prevent this, we evaluate the network every $n$ steps by playing against itself/an older version of itself. If the policy did not improve (i.e. losing more games than winning against older version), we throw away the new model and start again from the old weights. \n\\end{itemize}\n"
  },
  {
    "path": "Reinforcement_Learning/rl_partially_observable.tex",
    "content": "\\section{Partially observable environments and Bayesian methods}\n\\label{sec:partially_observable}\n\\textit{This section reviews the lecture slides 13.}\n\\begin{itemize}\n\t\\item Until now, we always assumed to have a fully observable environment. However, this is often not the case, also in real life (we cannot see what is happening behind walls, or 1000km away. Thus we are \\textit{living} in a partially observable environment).\n\t\n\tThis can happen if we have state aliasing (we see twice the same state although if it would be fully observed, it is clear that we are in two different states), or even simple noise.\n\t\\item First, let's consider a generalization of our environment. We define a latent state in the environment $x'_t$ which captures all information about the true state. From this latent state, we can make an observation $o_t$ per state, which can be seen as measurement of an unknown quantity.\n\t\n\tNote that in fully observable environments, we have $s_t=o_t=x'_t$.\n\t\\item A simple approach would be to consider an observation $o_t$ as features from the latent state, and hence use approaches from Section~\\ref{sec:value_based_approximation}. But this is usually not sufficient.\n\t\\item In most environments, we can infer information about the latent space by looking at the history $H_t=A_0,O_1,A_1,...,A_{t-1},O_t$, and we choose our next action based on the whole history $A_t=\\pi(H_t)$\n\t\\item However, using a full history is neither efficient nor practical (increases in size over time). Hence, we better use a lower dimensional feature representation of the history, $f(H_t)$, and use this as internal state of the policy $s_t = f(H_t)$ (note that $s_t$ has slightly different meaning here because it is the state which the policy sees, not what we get back from the environment).\n\t\\item The best function $f$ would be the one that summarizes all important information. We can define what this means as follows:\n\t$$f(H_t)=f(H'_t)\\implies \\Prob{O_{t+1}=o|H=H_t,A_t=a}=\\Prob{O_{t+1}=o|H=H'_t,A_t=a}$$\n\twhich means in textual form: if the representation of two histories are the same, then the expectation of the next observation is the same for both histories. Hence, we would also choose the same action, which leads to the conclusion, that the optimal policy can be found solely on $f$.\n\t\n\tA function that fulfills this condition is called \\textit{Markov function}. For any function which is not Markovian, we can only find an approximate optimal policy, but often not the optimal itself.\n\t\\item Which function is Markovian depends highly on the environment, and will be discussed next.\n\\end{itemize}\n\\subsection{Markov functions and histories}\n\\begin{itemize}\n\t\\item First, let's consider what we need to deal with a partially observable environment. Besides the environment and our policy-/value-based approach, we also have a state-update function (see Figure~\\ref{fig:rl_partially_observability_architecture}). Our goal is to find a state-update function which is Markovian and efficient/compact.\n\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/rl_partially_observability_architecture.png}\n\t\t\\caption{Overview of the dynamics in dealing with a partially observable environment.}\n\t\t\\label{fig:rl_partially_observability_architecture}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{Sample Markov functions}\n\\begin{itemize}\n\t\\item The simplest function is the identity, meaning $s_t=H_t$. However, this is neither compact, nor can we use it in any tabular policy setting efficiently (all possible sequences must be stored).\n\t\\item We can define a probability distribution over the latent space $X$, and try to do Bayesian inference (i.e. finding the posterior). This is done by calculating a \\textbf{belief state}:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\ts'(x')=p(x'|o',a,s)& =\\frac{p(o'|x',a,s)p(x'|a,s)}{p(o'|a,s)}\\\\\n\t\t\t& = \\frac{p(o'|x',a)p(x'|a,s)}{p(o'|a,s)}\\hspace{5mm}\\text{(removing $s$ as $x'$ is given as true state)}\\\\\n\t\t\t& = \\frac{p(o'|a,x')\\overbrace{\\sum_x p(x'|x,a)s(x)}^{=p(x'|a,s)}}{\\sum_{x'}p(o'|a,x')\\sum_x p(x'|x,a)s(x)}\n\t\t\\end{split}\n\t\\end{equation*}\n\twhere $s(x)$ is the old belief (i.e. belief over $x$ from last step). We can further define $p(o'|a,x')$ as the \\underline{observation model} (i.e. what do I see from the latent space), and $p(x'|x,a)$ as the \\underline{transition model} (i.e. how likely is it to move from one latent state to another). If we know these model dynamics by a full model description (or can estimate them), we have as state the probability distribution over latent state $x$.\n\t\n\tThis method is the classical approach for POMDPs, as it is compact, can be updated recursively and is easily interpretable by a human. However, the disadvantages are that we need the underlying model (not always given), and that it is only feasible for a discrete latent state (otherwise sums become integrals etc.).\n\t\\item As a last example, we can consider the obvious approach of determining all the observation probabilities:\n\t$$f(h)=\\begin{bmatrix}\n\tf_{o_1a_1}(h)\\\\ f_{o_1a_2}(h)\\\\\\vdots\\\\f_{o_2a_1}(h)\\\\\\vdots\\\\\n\t\\end{bmatrix}\\hspace{5mm}\\text{where}\\hspace{5mm}f_{oa}(h)=\\Prob{O_{t+1}=o|H=h,A_t=a}$$\n\tGiven enough data, we can learn this distribution. Furthermore, we can extend this to longer trajectories like $\\tau=a_0o_1a_1o_2a_2o_3$, and it can be proven that for a special set of \"core tests\" $\\tau_1,\\tau_2,...,\\tau_d$, we can create a Markov state. This is called a \\textbf{predictive state representation}.\n\t\n\tThe advantage is that it is as compact or even more than the belief states as we only have a probability over observations and not the latent space. However, it might be harder to interpret as we only have the probabilities of the \"core tests\", and it is still limited to the tabular setting.\n\\end{itemize}\n\\subsubsection{Approximations with non-Markov functions}\n\\begin{itemize}\n\t\\item Alternatively, we can also consider non-Markov functions which cannot guarantee to find the optimal policy, but at least an approximate one\n\t\\item The simplest method here is just using the last state, $S_t=O_t$. However, this might not contain all the information we need (e.g. in Atari games, movement cannot be captured), and is often not compact (still have the whole screen)\n\t\n\tA slight improvement is stacking a few observations, as in Atari games. This allows us to observe movement, but we still lose long-term dependencies.\n\t\\item We can also apply RNNs which take $O_t$ and $A_t$ as input including the last state $S_{t-1}$, and generate a new state $S_t$. This feature extractor can be learned end-to-end, and applied to a wide range of environments. However, the training might be a bit tricky in terms of hyperparameter tuning. \n\\end{itemize}\n\n\\subsection{Partial observability and exploration}\n\\begin{itemize}\n\t\\item We have seen that Markov functions rely on uncertainty of the latent state, which can be consider as trying to take the actions that make you most certain about the latent state (while maximizing the reward).\n\t\\item Hence, we can also consider this as a exploration strategy. If we assume to know the set of states and action of our environment, we can try to learn the transition probabilities as well by adding them to our state. This leads us to a \\textit{hyperstate}:\n\t$$x_{\\text{POMDP}} = (s_{\\text{MDP}}, \\text{transition}, \\text{rewards})$$\n\twhere we consider a fully observable environment as partially observable by adding the transition and reward distributions. Now, we can simply apply POMDP techniques as we have discussed before, where $p(x'|a,x)$ is now modeled by our transition parameters $\\theta$.\n\t\\item For example, consider a simple environment with 2 states and 2 actions each. Our transition probabilities can be defined as a vector:\n\t$$\\theta=(p_{11},p_{12},p_{21},p_{22})$$\n\twhere for a prior, we assume a uniform distribution. By interaction, we change our belief towards what we have observed by using Bayes:\n\t$$p(\\theta | x',x,a) = \\frac{p(x'|x,a,\\theta)p(\\theta)}{\\int_{\\theta} p(x'|x,a,\\theta)p(\\theta)d\\theta}$$\n\tSo, if we observe $x'=s_1$, $x=s_1$, $a=1$, our new belief over $p_{11}$ is (all other stay the same):\n\t$$p(p_{11}|x'=s_1,x=s_1,a=1)=\\frac{p(x'=s_1|x=s_1,a=1,p_{11})p(p_{11})}{\\int_{\\theta} p(x'=s_1|x=s_1,a=1,p_{11})p(p_{11})dp_{11}} = \\frac{p_{11} \\cdot 1}{1/2} = 2p_{11}$$\n\tHence, we expect to go in at least $2/3$ of the cases from $s_1$ to $s_1$ if we take action $1$. If we observe now again the same combination, we get:\n\t$$p(p_{11}|x'_2=s_1,x_2=s_1,a_2=1)=\\frac{p(x'=s_1|x=s_1,a=1,p_{11})p(p_{11}|x_1,x_1',a_1)}{\\int_{\\theta} p(x'_2=s_1|x_2=s_1,a_2=1,p_{11})p(p_{11}|x_1,x_1',a_1)dp_{11}} = \\frac{2p_{11} \\cdot p_{11}}{2/3} = 3p_{11}^2$$\n\t\\item Now, an optimal policy will take the uncertainty of the transition probabilities in account, and tries to maximizes the expected return. This leads to an optimal trade-off between exploration and exploitation.\n\t\\item However, keep in mind that we have $|\\mathcal{S}|^{|\\mathcal{A}|}$ transition probabilities to learn, which can be too large for certain environments. Hence, we might have to consider using approximations again.\n\\end{itemize}\n\\subsubsection{Bayesian Adaptive MDP and Meta-reinforcement learning}\n\\begin{itemize}\n\t\\item Our approach on partial observability can be seen as Bayesian because we use our posterior to estimate the expected reward given a prior of our beliefs (iterative posterior update)\n\t\\item If the prior assigns a non-zero probability to a certain model, then it can find the optimal strategy for it. The amount of samples needed, i.e. exploration, is based on the design of the prior. The closer the prior to a certain model, the less it will have to explore. But giving a model a higher chance in the prior than it should leads also to worse performance on other MDPs.\n\t\\item We can show that to find the optimal strategy for finding the best policy in a unknown MDP can be learned by sampling from the prior over MDPs, and use simple gradient estimates\n\t\\item Hence, with a prior over MDPs, optimal exploration can be phrased as greedy behaviour in an augmented MDP, where the hyperstates include the unknown transition and reward probabilities\n\t\\item Such techniques are investigated under the term \\textbf{Meta-reinforcement learning}. Here the agent is not told which exact MDP it gets, but has to learn patterns across MDPs, and find the optimal way of exploring.\n\\end{itemize}\n"
  },
  {
    "path": "Reinforcement_Learning/rl_policy_gradient_methods.tex",
    "content": "\\section{Policy gradient methods}\n\\label{sec:policy_learning}\n\\textit{This section reviews the lecture slides 7, 8, 9 and 10.}\n\\begin{itemize}\n\t\\item In this section we will discuss techniques for learning the policy directly. There are couple of advantages to it:\n\t\\begin{itemize}\n\t\t\\item We are able to deal with continuous actions\n\t\t\\item We are changing the policy \\textit{smoothly}, meaning that after an update, we only slightly change the probability distribution over actions. In case of $\\epsilon$-greedy on $q$-values, the policy heavily changes when best action becomes another one\n\t\t\\item Small errors in the value functions don't give a big error in $\\pi$ (we are directly optimizing the quantity of interest)\n\t\t\\item We are able to include prior knowledge, like \"don't fall of the cliff\", \"going left is potentially more interesting\", etc.\n\t\t\\item We are able to learn how much stochasticity is optimal for the given environment. \n\t\\end{itemize}\n\t\\item In case we have discrete actions, we can simply learn by a softmax over these (viewing them as different classes). For continuous, we can for example use a Gaussian, and learn to predict its mean and variance\n\t\\item The objective of a policy is always the same, namely optimize the expected return for its start state(s): $$J(\\theta)=v_{\\pi_{\\theta}}(s_0)=\\E\\left[\\sum_{t=0}^{T-1} r_{t+1}\\right]$$\n\tNote that we assume here $\\gamma=1$. We will use it throughout this section as it makes the derivations/discussion a bit easier, but we can change this term if necessary.\n\t\\item The simplest update is using \\textbf{finite difference}, meaning that we estimate the gradients by a small parameter change:\n\t$$\\nabla J(\\theta)\\approx \\frac{ J(\\theta+\\epsilon) - J(\\theta-\\epsilon)}{2\\epsilon}$$\n\tHowever, this means that for $n$ parameters, we would need at least $2n$ roll-outs for an estimate. For stochastic policies, this estimate is extremely noisy and hence, not really applicable.\n\\end{itemize}\n\\subsection{REINFORCE and the Policy Gradient Theorem}\n\\begin{itemize}\n\t\\item The Policy Gradient Theorem says that the gradients of $\\nabla_{\\theta}J(\\theta)$ are proportional to:\n\t$$\\nabla_{\\theta}J(\\theta) \\propto \\sum_s \\mu(s) \\sum_a \\nabla_{\\theta} \\pi_{\\theta}(a|s)q_{\\pi_{\\theta}}(s,a) $$\n\tNote that we are not interested in the constant proportionality factor because we will absorb it anyways in the learning rate\n\t\\item For deriving the REINFORCE algorithm, we follow the approach of the lecture slides. Let's define $\\tau$ as a trajectory that starts from $s_0$ and ends in an arbitrary terminal state. Then, the expected return is the expected return over these trajectories. Using this equality, we can derive the gradients as:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\nabla_{\\theta}J(\\theta) & =  \\nabla_{\\theta} \\E_{\\tau}[G(\\tau)]\\\\\n\t\t\t& = \\int \\nabla_{\\theta} p_{\\theta}(\\tau) G(\\tau)d\\tau\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\twhere the probability of a trajectory is defined as $p_{\\theta}(\\tau)=p(s_0)\\prod_{t=1}^{T}\\pi_{\\theta}(A_t|S_t)p(S_{t+1}|A_{t},S_t)$, and $G(\\tau)$ is the expected return from the initial state. Using the trick $\\nabla_{\\theta} p_{\\theta}(\\tau)=p_{\\theta}(\\tau)\\cdot \\nabla_{\\theta} \\ln p_{\\theta}(\\tau)$, we get:\n\t\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\nabla_{\\theta}J(\\theta) & = \\int \\nabla_{\\theta} p_{\\theta}(\\tau) G(\\tau)d\\tau\\\\\n\t\t\t& = \\E_{\\tau}[G(\\tau)\\nabla_{\\theta} \\ln p_{\\theta}(\\tau)]\\\\\n\t\t\t& = \\E_{\\tau}\\left[G(\\tau)\\sum_{t=1}^{T}\\nabla_{\\theta} \\ln p_{\\theta}(a_t|s_t)\\right]\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item Hence, to estimate the gradient, we can sample trajectories and approximate the expectation above. This estimate is unbiased but has some disadvantages. The efficiency of REINFORCE is low because of its high variance which comes from two points:\n\t\\begin{itemize}\n\t\t\\item First, REINFORCE can be seen as a Monte Carlo method of policy-based RL. Hence, the MC samples bring a certain level of noise with them\n\t\t\\item Suppose we are playing CartPole. Our reward is 1 for each time step, until we terminate. This leads to always positive gradients, which can be seen as \"supporting\" the last actions. Only if we sample the other action, we might experience an even higher return which pushes the policy towards the newly explored actions. \n\t\\end{itemize}\n\t\\item The second issue can be tackled by the usage of a \\textit{baseline} which is a constant subtracted from the return, that does not influence the gradients being unbiased:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\E_{\\tau}\\left[\\left(G(\\tau)-b\\right)\\sum_{t=0}^{T}\\nabla_{\\theta} \\ln p_{\\theta}(a_t|s_t)\\right] & = \\E_{\\tau}\\left[G\\left(\\tau\\right)\\sum_{t=0}^{T}\\nabla_{\\theta} \\ln p_{\\theta}(a_t|s_t)\\right]  - \\E_{\\tau}\\left[b\\sum_{t=0}^{T}\\nabla_{\\theta} \\ln p_{\\theta}(a_t|s_t)\\right] \\\\\n\t\t\t& = \\nabla J(\\theta) - b\\underbrace{\\int p_{\\theta}(\\tau)\\nabla_{\\theta} \\ln p(\\tau)d\\tau}_{=0}\\\\\n\t\t\t& = \\nabla J(\\theta) \n\t\t\\end{split}\n\t\\end{equation*}\n\t\\item A good baseline is the expected reward, which we for example can aggregate over the past.\n\t\\item However, there is one drawback which still remains. We assign each action of a trajectory the same credit, meaning that we punish every action equally no matter how much it actually was responsible for it. This is especially a problem when we punish actions for something in the past (e.g. if last 5 steps get reward of 10 each, but first got -100, we punish all of them equally). To prevent this, we move to G(PO)MDP\n\\end{itemize}\n\\subsubsection{G(PO)MDP}\n\\begin{itemize}\n\t\\item Gradient estimates for (Partially Observable) Markov Decision Processes\n\t\\item Let's reconsider the gradient estimate again, and try to split it into a part before $t$, and a part after $t$:\n\t\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\nabla_{\\theta}J(\\theta) & =  \\E_{\\tau}\\left[G(\\tau)\\sum_{t=1}^{T}\\nabla_{\\theta} \\ln p_{\\theta}(a_t|s_t)\\right]\\\\\n\t\t\t& = \\E_{\\tau}\\left[\\sum_{t=1}^{T} r_t \\sum_{t'=1}^{T}\\nabla_{\\theta} \\ln p_{\\theta}(a_{t'}|s_{t'})\\right]\\hspace{5mm}\\text{(Put in definition of return)}\\\\\n\t\t\t& = \\sum_{t=1}^{T} \\E_{\\tau_{1:t}}\\E_{\\tau_{t+1:T}}\\left[r_t \\sum_{t'=1}^{T}\\nabla_{\\theta} \\ln p_{\\theta}(a_{t'}|s_{t'})\\right]\\hspace{5mm}\\text{(Move sum out and split expectation)}\\\\\n\t\t\t& = \\sum_{t=1}^{T} \\E_{\\tau_{1:t}}\\left[r_t \\left(\\sum_{t'=1}^{t}\\nabla_{\\theta} \\ln p_{\\theta}(a_{t'}|s_{t'}) + \\underbrace{\\E_{\\tau_{t+1:T}}\\left[\\sum_{t'=t+1}^{T}\\nabla_{\\theta} \\ln p_{\\theta}(a_{t'}|s_{t'})\\right]}_{=\\E[\\int p(x)\\nabla\\log p(x)dx]=0}\\right)\\right]\\\\\n\t\t\t& = \\sum_{t=1}^{T} \\E_{\\tau_{1:t}}\\left[r_t \\sum_{t'=1}^{t}\\nabla_{\\theta} \\ln p_{\\theta}(a_{t'}|s_{t'}) \\right]\\\\\n\t\t\t& = \\E_{\\tau}\\left[\\sum_{t=1}^{T}  r_t \\sum_{t'=1}^{t}\\nabla_{\\theta} \\ln p_{\\theta}(a_{t'}|s_{t'}) \\right]\n\t\t\\end{split}\n\t\\end{equation*}\n\tWith this rewritten gradient, we give credit for $r_t$ only those actions that came \\textit{before} $t$\n\t\\item This reduces the variance from REINFORCE, and can be combined with baselines etc. However, keep in mind that we still have a Monte Carlo sample, so that there remains a significant variance\n\\end{itemize}\n\\subsubsection{Policy Gradients with Parameter-based Exploration (PGPE)}\n\\begin{itemize}\n\t\\item Even when sampling from stochastic policies during a rollout, the variation and exploration we get is mostly fairly limited. Furthermore, we end up with small perturbations (choose \"left\"-\"right\"-\"left\" in CartPole) which are less likely to be repeated by a deterministic policy, and can damage e.g. a robot in real-life situations\n\t\\item Instead, we rather \\textit{sample} a deterministic policy $\\pi_{\\theta}$ from a distribution $p(\\theta|\\nu)$.  The advantage is that if we now see a state twice, we can guarantee that our policy takes a same action although we are still exploring/stochastic. \n\t\\item So, instead of choosing $a$ at every randomly, we randomly choose the action for any state in the beginning (represented by $\\pi_{\\theta}$), and keep it fixed over the trajectory\n\t\\item Our gradient (which is now with respect to $\\nu$ as we want to learn $p(\\theta|\\nu)$) is:\n\t$$\\nabla_{\\nu} J(\\nu) = \\E_{\\theta}\\E_{\\tau|\\pi_{\\theta}}[G(\\tau) \\nabla_{\\nu}p(\\theta;\\nu)]$$\n\\end{itemize}\n\\subsection{Actor-critic Policy Gradient}\n\\begin{itemize}\n\t\\item Although we increased the stability of REINFORCE by the discussed improvements, the problems of Monte Carlo sampling remain: we have to wait until the end of the episode, and the samples have a high variance\n\t\\item First we take a look again at G(PO)MDP, where we slightly re-arrange the terms:\n\t$$\\E_{\\tau}\\left[\\sum_{t=1}^{T}  r_t \\sum_{t'=1}^{t}\\nabla_{\\theta} \\ln p_{\\theta}(a_{t'}|s_{t'}) \\right] = \\E_{\\tau}\\left[\\sum_{t'=1}^{T}\\nabla_{\\theta} \\ln p_{\\theta}(a_{t'}|s_{t'}) \\sum_{t=t'}^{T}  r_t\\right]$$\n\tIn our value-based methods, we previously learn the terms $\\sum_{t=t'}^{T}  r_t$ by the $v$/$q$-functions, as it is the expected value of those. Hence, we can also plug them in here:\n\t$$\\E_{\\tau}\\left[\\sum_{t'=1}^{T}\\nabla_{\\theta} \\ln p_{\\theta}(a_{t'}|s_{t'}) \\sum_{t=t'}^{T}  r_t\\right] = \\E_{\\tau}\\left[\\sum_{t'=1}^{T}\\nabla_{\\theta} \\ln p_{\\theta}(a_{t'}|s_{t'}) q_{\\pi}(s_{t'},a_{t'})\\right]$$\n\t\\item The question arises whether we can replace $q_{\\pi}(s_{t'},a_{t'})$ by an estimate $\\hat{q}_{\\bm{w}}(s_{t'},a_{t'})$ without introducing a bias. The answer is yes, but with two constraints on the function $\\hat{q}_{\\bm{w}}$:\n\t\\begin{enumerate}\n\t\t\\item The function has to be \\textit{compatible}, which means:\n\t\t$$\\nabla_{\\bm{w}}\\hat{q}_{\\bm{w}}(s,a) = \\nabla_{\\theta}\\ln \\pi_{\\theta}(a|s)\\hspace{5mm}\\text{like}\\hspace{2mm} \\hat{q}_{\\bm{w}}(s,a)=\\bm{w}^T \\nabla_{\\theta} \\ln \\pi_{\\theta}(a|s)$$\n\t\t\\item $\\hat{q}_{\\bm{w}}$ has to be fully converged, i.e.\n\t\t$$\\E\\left[(q_{\\pi}(s,a)-q_{\\bm{w}}(s,a)) \\frac{\\partial \\hat{q}_{\\bm{w}}(s,a)}{\\partial \\bm{w}}\\right]= 0$$\n\t\\end{enumerate}\n\t\\item We call the policy $\\pi$ the actor, while $\\hat{q}_{\\bm{w}}$ is the critic\n\t\\item Note that we can still add a baseline to stabilize learning further. For example, a good baseline is the value function so that our actual goal of $\\hat{q}_{\\bm{w}}$ should be to learn $\\hat{q}_{\\bm{w}}(s,a)\\approx q_{\\pi}(s,a)-v_{\\pi}(s)=A(s,a)$ which is also called the \\textbf{advantage}\n\t\\item The \\underline{benefits} of actor-critic methods is a lower variance as the target is not sampled anymore, and we can update our policy more frequently instead of waiting until the end of the episode.\n\t\n\tHowever, the \\underline{drawbacks} are that we have more hyperparameters to finetune (two learning rates etc.), and we require an stochastic policy (no full greedy policy possible). We will see later methods which can deal with deterministic ones.\n\\end{itemize}\n\\subsubsection{Generalized Advantage Estimation (n-step AC)}\n\\begin{itemize}\n\t\\item Let's reconsider the difference between actor-critic and actor-only approaches from a different perspective. Actor-critic bootstrap its estimate on the next value, which is very similar to TD(0) learning. Actor-only uses the sampled return, which is a Monte Carlo method. In value-based methods, we discussed that we can generalize TD(0) and Monte Carlo to $n$-step TD learning which we can do here similarly. This is again a trade-off between variance and bias. %  because as we have seen before, if $\\hat{q}_{\\bm{w}}$ has not fully converged (which is in practice mostly not the case), we get a biased estimate of our gradients.\n\t\\item The advantage for an $n$-step estimate is:\n\t$$\\hat{A}_t^n = r_t + \\gamma r_{t+1} + ... + \\gamma^n v(s_{t+n}) - v(s_t) = \\sum_{l=0}^{n-1}\\gamma^{l}\\delta_{t+l}$$\n\twhere $\\delta_{t}$ is the TD error for time step $t$. \n\t \n\tHowever, we can also take a smoother version of $n$-step, where we take a weighted average of all the advantages:\n\t$$\\hat{A}_t^{GAE} = (1-\\lambda)\\left(\\hat{A}_t^{(1)} + \\lambda \\hat{A}_t^{(2)} + \\lambda^2 \\hat{A}_t^{(2)} + ...\\right) = \\sum_{l=0}^{\\infty} (\\gamma \\lambda)^{l}\\delta_{t+l}$$\n\twhich is also known as TD($\\lambda$).\n\t\\item A lower $\\lambda$ reduces the variance but increases the bias (TD). Similarly, choosing a high $\\lambda$ gives a low bias but high variance (MC).\n\t\n\tHowever, a disadvantage of using $\\lambda$ instead of a fixed $n$ is that we need to run a full episode before we can calculate any advantage. We can overcome this issue by using eligibility traces where we update each state by its already known advantage factors of an episode, and continue doing so while following the trajectory.\n\t\\item This approach can again be used in combination with many different optimization techniques, like using TRPO (see next section). \n\\end{itemize}\n\\subsection{Higher-order Policy Search Methods}\n\\begin{itemize}\n\t\\item When updating our policy, we want to make sure that we don't change too much. The reason for that is that our samples come from $\\pi_{\\theta}$. The more we change $\\pi_{\\theta}$, the more our gradient estimate becomes inaccurate! Hence, we should limit our change in $\\pi_{\\theta}$.\n\t\\item A simple way of checking that is by taking the L2 norm over parameters, namely $d\\theta^T d\\theta$, and fix this norm to a certain value $c$\n\t\\item To find the next optimal value, we have to solve the following equation for the step we take, namely $\\theta^{*}-\\theta_0$ ($\\theta^{*}$ next value), such that $d\\theta^T d\\theta=c$:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\theta^{*}-\\theta_0 & = \\arg\\max_{d\\theta} J(\\theta_0 + d\\theta)\\\\\n\t\t\t& \\approx \\arg\\max_{d\\theta} J(\\theta_0) + (\\nabla_{\\theta} J(\\theta))^T d\\theta \\hspace{5mm}\\text{(Taylor expansion)}\\\\\n\t\t\t& \\propto \\nabla_{\\theta} J(\\theta)\n\t\t\\end{split}\n\t\\end{equation*}\n\tThis means that the previous, standard policy gradient methods maximize the Taylor expansion of $J$ such that the update is on the norm sphere (as $d\\theta^T d\\theta=c$)\n\t\\item However, there are many disadvantages and possible problems of this:\n\t\\begin{itemize}\n\t\t\\item The norm itself is highly sensitive to the parameterization of the model. For example, consider a Gaussian for which we want to learn the mean and the variance. We can achieve the same if we learn the standard deviation, or even the precision of the Gaussian. However, all these parameters have a different scale, and euclidean distance is not always the best distance measurement (e.g. $\\sigma=0.1$ and $\\sigma=0.2$ are more \\textit{different} than $\\sigma=10.1$ and $\\sigma=10.2$)\n\t\t\\item Another simple fail case is when we change the scale of the parameters. Suppose we express the mean by $\\mu=4\\cdot \\theta_1$ instead of $\\mu=\\theta_1$, while keeping $\\sigma=\\theta_2$. As we only look at the gradient norm of $\\theta_1$ and not $\\mu$, we take in the first case a four-times as big step than in the other case. This is clearly not desired because both parameterizations express the same model, with just different scales. Furthermore, this can lead to an issue for $\\sigma=\\theta_2$ as we do not change $\\sigma$ equally.\n\t\t\\item Furthermore, we ignore correlations between parameters. If the gradient of $\\theta_1$ and $\\theta_2$ are highly correlated, like if we would use $\\mu=\\theta_1+\\theta_2$, we update both as if they were independent.\n\t\\end{itemize}\n\t\\item So, we are not directly interested in the change of the parameters, but of the policy distribution. A better way of doing so is by using the KL divergence as difference:\n\t$$D_{\\text{KL}}(p||q)=\\int p(x)\\log\\frac{p(x)}{q(x)}dx$$\n\tThere are different algorithms that exploits this property, and we will discuss two of them: Natural Policy Gradient, and Trust region policy optimization\n\\end{itemize}\n\\subsubsection{Natural Policy Gradients}\n\\begin{itemize}\n\t\\item The first step for using the KL divergence as step size regulator, is by replacing the constant $c$ by the quadratic expansion of expected KL divergence over states:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tc & = \\E_{s}\\left[D_{KL}\\left(\\pi(a|s;\\theta_0)||\\pi(a|s;\\theta)\\right)\\right] = \\text{EKL}(\\theta)\\\\\n\t\t\t& \\approx \\underbrace{\\text{EKL}(\\theta_0)}_{=KL(p||p)=0} + d\\theta^T \\underbrace{(\\nabla_{d\\theta}\\text{EKL})(\\theta_0)}_{=0 \\text{ as }\\theta_0\\text{ optimum}} + \\frac{1}{2}d\\theta^T (\\nabla_{d\\theta}^2\\text{EKL})(\\theta_0)d\\theta\\\\\n\t\t\t& = \\frac{1}{2}d\\theta^T (\\nabla_{d\\theta}^2\\text{EKL})(\\theta_0)d\\theta\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n    \\item So, to obtain the optimal parameter $c$, we need to calculate the Hessian $\\nabla_{d\\theta}^2\\text{EKL}$ at point $\\theta_0$. This is also known as the Fisher information matrix (i.e. how much information of $\\pi$ is changed by $\\theta$), and can be calculated by:\n\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tF & = \\nabla_{d\\theta}^2\\text{EKL} = \\E_{s}\\left[\\nabla_{d\\theta}^2 D_{KL}\\left(\\pi(a|s;\\theta_0)||\\pi(a|s;\\theta)\\right)\\right]\\\\\n\t\t\t\\nabla_{d\\theta}^2 D_{KL} & = \\E_{a\\sim\\pi(a|s;\\theta_0)}[\\nabla_{d\\theta}^2 \\log \\pi(a|s;\\theta_0+d\\theta)]\\\\[10pt]\n\t\t\\end{split}\n\t\\end{equation*}\n\t$$\\implies F = \\begin{bmatrix}\n\t\\E_a\\left[\\left(\\nabla_{d\\theta_1} \\log \\pi_{\\theta}(a|s)\\right)^2\\right] & \\E_a\\left[\\nabla_{d\\theta_1} \\log \\pi_{\\theta}(a|s) \\cdot \\nabla_{d\\theta_2} \\log \\pi_{\\theta}(a|s)\\right] & ...\\\\\n\t\\E_a\\left[\\nabla_{d\\theta_2} \\log \\pi_{\\theta}(a|s) \\cdot \\nabla_{d\\theta_1} \\log \\pi_{\\theta}(a|s)\\right] & \\E_a\\left[\\left(\\nabla_{d\\theta_2} \\log \\pi_{\\theta}(a|s)\\right)^2\\right] & ...\\\\\n\t% \\E_a\\left[\\nabla_{d\\theta_1} \\log \\pi_{\\theta}(a|s)\\right]\\cdot \\E_a\\left[\\nabla_{d\\theta_3} \\log \\pi_{\\theta}(a|s)\\right] & \\E_a\\left[\\nabla_{d\\theta_2} \\log \\pi_{\\theta}(a|s)\\right]\\cdot \\E_a\\left[\\nabla_{d\\theta_3} \\log \\pi_{\\theta}(a|s)\\right] & ...\\\\\n\t\\vdots & \\vdots & \\ddots\n\t\\end{bmatrix}$$\n\t\n\t\\item Now, we reconsider our update step:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\theta^{*}-\\theta_0 & \\approx \\arg\\max_{d\\theta} J(\\theta_0) + (\\nabla_{\\theta} J(\\theta))^T d\\theta \\hspace{5mm}\\text{(Taylor expansion)}\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\tTo take the maximum, we now also need to consider our constraint as Lagrangian:\n\t$$\\max_{d\\theta}\\min_{\\lambda} J(\\theta_0) + (\\nabla_{\\theta} J(\\theta))^T d\\theta + \\lambda (d\\theta F d\\theta-c)$$\n\tSo that, when we solve it, we get:\n\t$$d\\theta  \\propto F^{-1}\\nabla_{\\theta}J(\\theta)$$\n\twhich we call the \\textit{natural gradient}\n\t\\item The update rule is now:\n\t$$\\theta_{t+1} = \\theta_t + \\alpha F^{-1}\\nabla_{\\theta_t}J(\\theta_t)$$\n\twhere for the vanilla gradient $\\nabla_{\\theta_t}J(\\theta_t)$, we can use any of the above methods.\n\t\\item We can show that for a sufficiently small step size, we will always improve by this update step \n\t\\item Figure~\\ref{fig:rl_policy_gradients_NPG} shows a visualization of the update difference between NPG and standard policy gradients. NPG allows us to find a better fit in the region of \"safe\" changes, so that we possibly can take larger steps, towards the right policy.\n\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/rl_policy_gradients_NPG.png}\n\t\t\\caption{Comparison of L2 norm update (left) and NPG (right). The orange background represents the safe changes, meaning the parameter changes for which our policy does not change greater than our defined threshold. While the L2 sticks with the unit sphere, NPG can represents ellipsoids so that we can take larger steps in parameter space towards increasing $J$ without changing the policy too much.}\n\t\t\\label{fig:rl_policy_gradients_NPG}\n\t\\end{figure}\n\n\tWe can also visualize the gradient direction grid, as in Figure~\\ref{fig:rl_policy_gradients_NPG_gradient_example}. Gradients that point in the wrong direction, give very slow convergence because we focus on parameters which are already close to optimal. \n\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/rl_policy_gradients_NPG_gradient_example.png}\n\t\t\\caption{Gradient direction over parameter space for finding the optimum at $(-1,0)$. While the NPG gives very smooth transitions, vanilla gradients point straight down (strong gradient for $\\theta_2$) for points like $(-2,0.1)$ where we clearly need to update $\\theta_1$ more. This leads to slow convergence.}\n\t\t\\label{fig:rl_policy_gradients_NPG_gradient_example}\n\t\\end{figure}\n\t\\item The advantages of NPG are therefore:\n\t\\begin{itemize}\n\t\t\\item Faster convergence and less training time\n\t\t\\item Is an adaptation on top of standard policy gradient, so we can use any of the previous methods with all additions/tricks we want\n\t\\end{itemize}\n\tHowever, the biggest drawback is that we have to calculate the Fisher information matrix, which is known for standard distributions like Gaussians, but might be harder to determine for other cases, especially if we want to use a neural network (can be approximated with conjugate gradient algorithm). It also keeps the disadvantages of the other policy gradient methods, namely high variance and still slower convergence compared to value-based methods.\n\\end{itemize}\n\\subsubsection{Trust region policy optimization}\n\\begin{itemize}\n\t\\item A problem of Natural Policy Gradient is that we approximated the KL divergence by a second-order Taylor expansion. The errors that we introduced there, might cause our initial KL constraint to break meaning that $d\\theta^T F d\\theta\\neq c$\n\t\\item TRPO takes a bit different view on the problem. The main concept of the algorithm is that we take as big steps as long as we can guarantee improvement. Hence, we have three steps:\n\t\\begin{enumerate}\n\t\t\\item Approximate the return function $J$\n\t\t\\item Apply a penalty term to yield lower bound on the exact function\n\t\t\\item Maximize lower bound (which guarantees improvement on exact function) by e.g. SGD again\n\t\\end{enumerate} \n\t\\item The region where we assume our approximation to be valid, is called \\textit{trust region}\n\t\\item Figure~\\ref{fig:rl_policy_gradients_TRPO_vs_NPG} compares the ideas of NPG and TRPO visually. NPG takes a linear approximation of $J$ at a point, and limits the step size by approximating the KL divergence. TRPO however designs a lower bound which is strictly lower than the \"true\" function. It is a combination of an approximation of $J$, and a penalty term for big changes in the policy \n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.55\\textwidth]{figures/rl_policy_gradients_TRPO_vs_NPG.png}\n\t\t\\caption{Comparing the optimization properties of NPG (left) and TRPO (right).}\n\t\t\\label{fig:rl_policy_gradients_TRPO_vs_NPG}\n\t\\end{figure}\n\t\\item We can calculate the return by:\n\t$$\\eta(\\theta)=\\E_{\\bm{s}\\sim \\mu_{\\pi_{\\theta},\\bm{a}\\sim \\pi_{\\theta} }(s)}[r(\\bm{s},\\bm{a})]$$\n\twhere we use samples from $\\pi_{\\theta}$ to approximate the expectation. However, as soon as we shift $\\theta$, we cannot use the same samples anymore because we would be biased. Hence, we can use importance weights:\n\t$$\\eta(\\theta)\\approx \\E_{\\bm{s}\\sim \\mu_{\\pi_{\\theta}},\\bm{a}\\sim \\pi_{\\theta'} (\\bm{s})}\\left[\\frac{\\pi_{\\theta}(\\bm{a}|\\bm{s})}{\\pi_{\\theta'}(\\bm{a}|\\bm{s})}r(\\bm{s},\\bm{a})\\Bigg\\vert \\theta' \\right]=L_{\\theta'}(\\theta)$$\n\tbut note that the state distribution is not changed (hence approximation!).\n\t\\item Now, let's consider how we get the lower bound based on our approximation. \n\t$$\\eta(\\theta)\\geq L_{\\theta'}(\\theta) - \\frac{2\\epsilon \\gamma}{(1-\\gamma)^2}\\cdot \\max_s D_{\\text{KL}}\\left(\\pi_{\\theta}(\\cdot|s)||\\pi_{\\theta'}(\\cdot|s)\\right)$$\n\twhere the factor in front of the penalty is environment/policy dependent. \n\t\\item Although this term above guarantees us a lower bound, in practice, we might run into multiple issues:\n\t\\begin{itemize}\n\t\t\\item We need to take the maximum KL divergence over all states. However, in environments with many and/or continuous states, this is often not possible. So, we approximate it by taking the average instead.\n\t\t\\item The penalty is usually very high so that we cannot make big steps. So, the average can already help to reduce the penalty, but we can also consider the KL divergence rather as a constraint than a penalty. This leads us to a similar approach as for NPG\n\t\\end{itemize}\n\t\\item So, what we do instead is maximizing the approximation as in Natural Policy Gradient, but dynamically set the step size based on a maximum KL divergence that we allow between policies. Meaning, we solve the following equation for step size $\\beta$:\n\t$$D_{\\text{KL}} \\approx \\beta^2 d\\theta^T F_s d\\theta / 2$$\n\twith $d\\theta$ being in the same direction as NPG. One way of (approximately) solving it is by starting with an initial $\\beta_0$, and for a couple of steps, increase it if constraint is fulfilled. Otherwise, reduce until we find a valid, sufficiently high $\\beta$.\n\t\n\t\\item In Figure~\\ref{fig:rl_policy_gradients_TRPO_vs_NPG}, we now are at the left image again but the step size is adjusted by the KL.\n\t\\item The advantage of TRPO is that we can take bigger steps than the standard NPG, while in theory, having the guarantee of converging. It has been shown to work well with neural controllers where we approximate $F$ by the conjugate gradients. \n\t\n\tThe disadvantages are however, that it still requires many steps, and the return is still a Monte Carlo sample (high variance). The guarantee of convergence is actually broken by all the approximations we took.\n\\end{itemize}\n\n\\subsection{Deep Policy Search}\n\\begin{itemize}\n\t\\item When using deep neural networks for policy search, we might need to consider a few additional tricks because of the high non-linearity of the networks.\n\t\\item To discuss this, we take deterministic policy gradient as an example, and explain the tricks that are used here\n\\end{itemize}\n\\subsubsection{Deterministic policy gradients}\n\\begin{itemize}\n\t\\item All policy gradient methods we have discussed so far considered stochastic policy. However, it is sometimes preferred to learn a deterministic policy (e.g. remember Q-learning)\n\t\\item This means that we will also learn off-policy (behavior policy $b$ with target/actor $\\pi$). All other methods were discussed from the on-policy perspective but could be adjusted for off-policy with some minor modifications like importance sampling\n\t\\item When using a different policy for sampling, we change our state distribution from $\\mu_{\\pi}$ to $\\mu_{\\beta}$, so that our return is:\n\t$$J_{\\beta}(\\pi_{\\theta}) = \\int_{\\mathcal{S}} \\mu^{\\beta}(s)Q^{\\pi}\\left(s,\\pi_{\\theta}(s)\\right)ds$$\n\tIn terms of gradients, we end up with:\n\t$$\\nabla_{\\theta} J_{\\beta}(\\pi_{\\theta})=\\E_{s\\sim\\mu_{\\beta}}\\left[\\nabla_{\\theta}\\pi_{\\theta}(s)\\nabla_a Q^{\\pi}(s,a)|a=\\pi_{\\theta}(s)\\right] $$\n\tNote that this requires $\\pi_{\\theta}(s)$ to be differentiable, hence returning continuous actions.\n\t\n\tAlthough our samples are slightly off/biased ($\\mu_{\\beta}$ instead of $\\mu_{\\pi}$), this is usually not a problem as $\\beta$ is chosen to be $\\pi$ with some additional noise (like Gaussian).\n\t\\item Our update equations are as follows:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\text{TD error }\\hspace{2mm}\\delta_t & = r_t + \\gamma Q^{w}\\left(s_{t+1}, \\pi_{\\theta}\\left(s_{t+1}\\right)\\right) - Q^{w}\\left(s_{t}, a_t\\right)\\\\\n\t\t\t\\text{Update of $Q$ }\\hspace{2mm}w_{t+1} & = w_{t} + \\alpha_{w}\\delta_t \\nabla_{w} Q^{w}(s_t,a_t)\\\\\n\t\t\t\\text{Update of $\\pi$ }\\hspace{2mm}\\theta_{t+1} & = \\theta_{t} + \\alpha_{\\theta} \\nabla_{\\theta} \\pi_{\\theta}(s_t) \\nabla_{a} Q^{w}(s_t,a_t)|_{a=\\pi_{\\theta}(s)}\n\t\t\\end{split}\n\t\\end{equation*}\n\twhere we illustrate the gradients in Figure~\\ref{fig:rl_policy_gradients_DPG}.\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.5\\textwidth]{figures/rl_policy_gradients_DPG.png}\n\t\t\\caption{Illustrating the gradients in DPG. We have the combination of how much we change the action by changing $\\theta$, and how this action change influences the action value $Q^{\\pi}$.}\n\t\t\\label{fig:rl_policy_gradients_DPG}\n\t\\end{figure}\n\\end{itemize}\n\\subsubsection{Deep DPG}\n\\begin{itemize}\n\t\\item When using neural networks, we again have to consider the same issues as in the DQN approach\n\t\\item To use the collected data more efficiently and break the dependency between elements in a batch, we apply \\textit{experience replay}\n\t\\item For stabilizing the TD updates, we don't fix the target network, but create a second one that slowly tracks the learned $Q$ values\n\t\\item For ensuring a similar scale of features, we apply \\textit{batch normalization} within the network\n\t\\item One aspect of exploration that we have discussed in PGPE before is that independent noise on the actions do not explore well. As a simple improvement, the Deep DPG paper correlates the noise by endorsing to use the same random decision as the time step before\n\\end{itemize}\n\\subsection{Summary}\n\\begin{itemize}\n\t\\item To wrap up policy-based reinforcement learning, we want to put all discussed algorithms into perspective.\n\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\begin{subfigure}{0.45\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=\\textwidth]{figures/rl_policy_gradient_summary_1.png}\n\t\t\t\\caption{Exploration vs Evaluation}\n\t\t\\end{subfigure}\n\t\t\\hspace{5mm}\n\t\t\\begin{subfigure}{0.45\\textwidth}\n\t\t\t\\centering\n\t\t\t\\includegraphics[width=\\textwidth]{figures/rl_policy_gradient_summary_2.png}\n\t\t\t\\caption{Actor-only versus Value-only methods}\n\t\t\\end{subfigure}\n\t\t\\caption{Comparing algorithms across two dimensions. (a) Pointing out the main difference between simple methods and advanced policy gradients (see text for more explanation). (b) Setting policy gradient methods into perspective with value-based.}\n\t\\end{figure}\n\n\t\\item When discussing the first algorithms of policy gradient, we could distinguish the methods on two dimensions:\n\t\\begin{itemize}\n\t\t\\item \\textit{Exploration}: One key point in the discussion of PGPE was the exploration. Methods like REINFORCE explore by sampling an action at each time step independently, hence their exploration is step-based. PGPE however samples a new policy once in the beginning. This is episode-based because within the episode, we follow a deterministic policy and do not add noise per step. DDPG can be considered as in-between because it adds noise correlations between steps, and hence has not a purely step-based exploration strategy anymore. \n\t\t\n\t\tIn general, it is hard to say which of both is preferred, and possibly depends on the environment. Independent noise as in the step-based methods have been shown to explore worse (which is why DDPG added the correlation). However, PGPE is more complex to implement and to learn because we have to learn a distribution over parameters $\\theta$ which do not one-to-one correspond to distribution over different policies (as discussed in NPG, relation between $\\theta$ and $\\pi$ might be quite complex).\n\t\t\n\t\t\\item \\textit{Evaluation}: Across our discussion, we have seen that some algorithms evaluate their actions step-wise and others per episode. We prefer methods that we can evaluate step-wise because they usually don't need a full sample until the end of an episode (except GPOMDP and other non-Actor-Critic methods), and give every step individual credit assignment. REINFORCE performs episode-based evaluations because the first steps reward influences the last steps update (which we tried to prevent in the other algorithms)\n\t\\end{itemize}\n\t\\item Another part is to consider the different sub-groups of policy-based methods with respect to value-based techniques. REINFORCE, G(PO)MDP and PGPE are all actor-only methods, meaning that they only learn a policy $\\pi$. We have seen that we can extend most approaches by introducing a critic that learns $q_{\\pi}(s,a)$.\n\t\n\tNPG and TRPO can be applied whether with or without actor-critic. Furthermore, we are free to choose how we arrive at $\\nabla J$, but note that in the theoretical motivation of TRPO, we use the lower bound so that it is, strictly speaking, not a policy gradient method\n\\end{itemize}\n"
  },
  {
    "path": "Reinforcement_Learning/rl_summary.tex",
    "content": "\\documentclass[a4paper]{article} \n\\addtolength{\\hoffset}{-2.25cm}\n\\addtolength{\\textwidth}{4.5cm}\n\\addtolength{\\voffset}{-3.25cm}\n\\addtolength{\\textheight}{5cm}\n\\setlength{\\parskip}{0pt}\n\\setlength{\\parindent}{0in}\n\n\\usepackage{blindtext} % Package to generate dummy text\n\\usepackage{charter} % Use the Charter font\n\\usepackage[utf8]{inputenc} % Use UTF-8 encoding\n\\usepackage{microtype} % Slightly tweak font spacing for aesthetics\n\\usepackage[english]{babel} % Language hyphenation and typographical rules\n\\usepackage{amsthm, amsmath, amssymb, amsfonts, nccmath} % Mathematical typesetting\n\\usepackage{float} % Improved interface for floating objects\n\\usepackage[final, colorlinks = true, \nlinkcolor = black, \ncitecolor = black]{hyperref} % For hyperlinks in the PDF\n\\usepackage{graphicx, multicol} % Enhanced support for graphics\n\\usepackage{xcolor} % Driver-independent color extensions\n\\usepackage{marvosym, wasysym} % More symbols\n\\usepackage{rotating} % Rotation tools\n\\usepackage{subcaption}\n\\usepackage{wrapfig}\n% \\usepackage{geometry}\n\\usepackage{censor} % Facilities for controlling restricted text\n\\newcommand{\\note}[1]{\\marginpar{\\scriptsize \\textcolor{red}{#1}}} % Enables comments in red on margin\n\\usepackage{bm}\n\\usepackage{blkarray}\n\\usepackage{enumitem}\n\\usepackage{pgfplots}\n\\usepackage{tikz}\n\\usetikzlibrary{bayesnet}\n\n\\usepackage{tcolorbox}\n\\usepackage[ruled,vlined]{algorithm2e}\n\n\\newcommand{\\pd}[2]{\\frac{\\partial #1}{\\partial #2}}\n\\newcommand{\\loss}[0]{\\mathcal{L}}\n\\newcommand{\\chain}[3]{\\frac{\\partial #1}{\\partial #2}\\frac{\\partial #2}{\\partial #3}}\n% \\newcommand{\\eq}[1]{\\begin{equation*}\\begin{split}#1\\end{split}\\end{equation*}}\n\\newcommand{\\TODO}[1]{\\textbf{\\textcolor{red}{#1}}}\n\\newcommand{\\E}[0]{\\mathbb{E}} % Expectation\n\\newcommand{\\R}[0]{\\mathbb{R}} % Real numbers\n\\newcommand{\\Cdo}[0]{\\textnormal{do}}\n\\newcommand{\\Prob}[1]{\\Pr\\left\\{#1\\right\\}} % Real numbers\n\\newcommand\\independent{\\protect\\mathpalette{\\protect\\independenT}{\\perp}}\n\\def\\independenT#1#2{\\mathrel{\\rlap{$#1#2$}\\mkern2mu{#1#2}}}\n\\newcommand*{\\QED}{\\hfill\\ensuremath{\\blacksquare}}%\n\n\\definecolor{green}{RGB}{0,160,0}\n\\definecolor{blue}{RGB}{0,0,160}\n\\definecolor{red}{RGB}{160,0,0}\n\\definecolor{orange}{RGB}{200,160,0}\n\\definecolor{purple}{RGB}{170,0,200}\n\\definecolor{cyan}{RGB}{0,200,200}\n\\definecolor{lightred}{RGB}{200,50,50}\n\n\\setcounter{tocdepth}{2}\n% Title Page\n\\title{Summary Reinforcement Learning}\n\\author{Phillip Lippe}\n\n\n\\begin{document}\n\\maketitle\n\\tableofcontents\n\\newpage\n\n\\input{rl_introduction.tex}\n\\newpage\n\\input{rl_tabular_methods.tex}\n\\newpage\n\\input{rl_learning_with_approx.tex}\n\\newpage\n\\input{rl_policy_gradient_methods.tex}\n\\newpage \n\\input{rl_model_based.tex}\n\\newpage\n\\input{rl_partially_observable.tex}\n\\newpage\n\\appendix\n\\newpage\n\\input{rl_appendix.tex}\n\n\\end{document}"
  },
  {
    "path": "Reinforcement_Learning/rl_tabular_methods.tex",
    "content": "\\section{Value-based RL: Tabular Methods}\n\\textit{This section reviews the lecture slides 2 (Monte Carlo), 3 and 4.}\n\\subsection{Monte Carlo}\n\\begin{itemize}\n\t\\item We can try to estimate the value function by simply sampling from the expectation, meaning we generate episodes, and evaluate the cumulative reward:\n\t$$v(s_t)=\\E_{\\pi}\\left[\\sum_{k=0}^{\\infty} \\gamma^k R_{t+k+1}\\Big\\vert S_t=s\\right]\\approx \\frac{1}{N}\\sum_{n=1}^{N}\\sum_{k=0}^{T_{n}}\\gamma^k R_{t+k+1}^{(n)}$$\n\tNote that this requires the task to be \\textbf{episodic}, meaning that an episode always ends. Otherwise, we are not able to sample \n\t\\item We can get into the situation that we visit the same state twice in a trajectory. In the update, we can either just take the first time into account that we visited $s$ (\\textit{first-visit MC}), or we can consider all of them as different points (\\textit{every-visit MC}). Both approaches are very similar, and converge to the same optimum. However, every-visit MC leads to slightly biased estimates if number of samples is low.\n\t\\item If we want to use Monte-Carlo for learning the optimal policy, we need to slightly adjust our algorithm. First, note that we rather want to learn the $q$-value as we can determine the optimal policy from them by greedifying: $\\pi_{*}(s)=\\arg\\max_a q_{*}(s,a)$\n\t\\item To guarantee that every state-action pair is visited, we can either:\n\t\\begin{itemize}\n\t\t\\item Perform ``exploring starts'', meaning that we randomly sample our start state $(S_0,A_0)$. However, note that this requires an environment where we can set the agent to any position, which is not always possible (e.g. in physical systems, hard to initialize velocity or acceleration)\n\t\t\\item Use policy that visits every state and actions with non-zero probability. \n\t\\end{itemize}\n\t\\item As the random starts are often not possible, we mostly choose to integrate exploration into our policy. We can either do this by updating our policy \\textit{towards} the greedy one, but not match it exactly (called \\textbf{on-policy}). Or we sample from a non-greedy behavior policy but update with respect to our greedy one (called \\textbf{off-policy}).\n\\end{itemize}\n\\subsubsection{On-policy MC}\n\\begin{itemize}\n\t\\item To ensure that every action is taken with a non-zero probability, we can use policies like $\\epsilon$-greedy. Every time we update our $q$-value, we can update our policy by making it greedy on $q$, and adding $\\epsilon$ as probability for choosing a random action.\n\t\\item We can show that choosing our $\\epsilon$-greedy policy by that actually leads to the optimal $\\epsilon$-greedy policy, as:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tq_{\\pi}(s,\\pi'(s)) & =\\frac{\\epsilon}{|\\mathcal{A}(s)|}\\sum_a q_{\\pi}(s,a) + \\left(1-\\epsilon\\right)\\max_a q_{\\pi}(s,a)\\\\\n\t\t\tv_{\\pi}(s) & = \\frac{\\epsilon}{|\\mathcal{A}(s)|} \\sum_a q_{\\pi}(s,a) + (1-\\epsilon)\\left[\\sum_{a}\\frac{\\pi(a|s)-\\frac{\\epsilon}{|\\mathcal{A}(s)|}}{1-\\epsilon} q_{\\pi}(s,a)\\right]\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\tTo show that we improve, we need to show that $v_{\\pi'}(s)\\geq v_{\\pi}(s)$. As the stochastic part $\\frac{\\epsilon}{|\\mathcal{A}(s)|} \\sum_a q_{\\pi}(s,a)$ is unaffected by $\\pi$, we only have to compare the greedy part:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\sum_{a}\\frac{\\pi'(a|s)-\\frac{\\epsilon}{|\\mathcal{A}(s)|}}{1-\\epsilon} q_{\\pi}(s,a) \\geq \\sum_{a}\\frac{\\pi(a|s)-\\frac{\\epsilon}{|\\mathcal{A}(s)|}}{1-\\epsilon} q_{\\pi}(s,a)\\\\\n\t\t\\end{split}\n\t\\end{equation*}\n\twhere we can put in $\\pi'$ as the greedy policy:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\max_a q_{\\pi}(s,a) \\geq \\sum_{a}\\frac{\\pi(a|s)-\\frac{\\epsilon}{|\\mathcal{A}(s)|}}{1-\\epsilon} q_{\\pi}(s,a)\n\t\t\\end{split}\n\t\\end{equation*}\n\tThis is obviously true because there is no actions besides the greedy one which gives higher reward. \n\t\\item Hence, when we updating the policy, we either improve or stay equally optimal. Note that this only holds for $\\epsilon$-soft policies, meaning policies for which every action has at least $\\epsilon/|\\mathcal{A}(s)|$ probability of being selected\n\\end{itemize}\n\\subsubsection{Off-policy MC}\n\\begin{itemize}\n\t\\item In off-policy, we have a behavior policy $b$ from which we sample the trajectories, and our greedy target policy $\\pi$. The only constraint on $b$ is that for any action where $\\pi(a|s)>0$, $b$ also needs to be $b(a|s)>0$. We can ensure this by any $\\epsilon$-soft policy\n\t\\item When sampling, we need to correct for the fact that we use samples from $b$ to evaluate an expectation over $\\pi$. One way of doing so is importance sampling:\n\t$$v_{\\pi}(s) \\approx \\frac{1}{N}\\sum_{n=1}^{N} \\frac{p(\\tau^{n}_{t}|s,A_t\\sim\\pi)}{p(\\tau^{n}_{t}|s,A_t\\sim b)} G(\\tau_{t}^{n})$$\n\twhere $\\tau^{n}_{t}$ is the $n$-th trajectory starting from time step $t$ till the end. We can rewrite the importance weights as $\\rho_{t:T-1}=\\frac{\\prod_{k=t}^{T-1}\\pi(A_k|S_k)}{\\prod_{k=t}^{T-1}b(A_k|S_k)}$. Note that the transition probabilities between $S_{t}$ and $S_{t+1}$ cancel out as they are the same for $\\pi$ and $b$.\n\t\\item When using these importance weight, there are two ways we can average over them:\n\t\\begin{itemize}\n\t\t\\item \\textbf{Ordinary} importance sampling averages by taking the number of trajectories into account:\n\t\t$$v_{\\pi}(s)=\\frac{\\sum_{t\\in\\mathcal{T}(s)}\\rho_{t:T(t)-1}G_t}{|\\mathcal{T}(s)|}$$\n\t\tWhile this gives us an unbiased estimate, also for small sample sizes, it suffers from high variance when $\\rho_{t:T(t)-1}$ varies a lot (e.g. $b$ and $\\pi$ quite different)\n\t\t\\item \\textbf{Weighted} importance sampling averages by summing over weights:\n\t\t$$v_{\\pi}(s)=\\frac{\\sum_{t\\in\\mathcal{T}(s)}\\rho_{t:T(t)-1}G_t}{\\sum_{t\\in\\mathcal{T}(s)}\\rho_{t:T(t)-1}}$$\n\t\tThis approach reduces the variance because we take into account whether we mostly have big or small values of $\\rho_{t:T(t)-1}$, but gives an biased estimate. Suppose we have a single sample, then the importance weight cancels out, meaning we estimate $v_{\\pi}(s)\\approx v_{b}(s)$. The more samples we get, the lower this bias gets.\n\t\\end{itemize}\n\tNote that while both give the same correct result for $N\\to\\infty$, they differ for cases with limited sample size. In practice, the lower variance is mostly preferred so that weighted importance sampling is usually applied.\n\t\\item For implementing this, we take an incremental approach as we can calculate importance weights by $\\rho_{t:T(t)-1}=\\frac{\\pi(A_t|S_t)}{b(A_t|S_t)}\\rho_{t+1:T(t)-1}$ which we denote by $W$ in Figure~\\ref{fig:rl_tabular_methods_offpolicy_MC_control}. Hence, given a trajectory, we should start with the last state, and iterate to the start state.\n\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.6\\textwidth]{figures/rl_tabular_methods_offpolicy_MC_control.png}\n\t\t\\caption{Incremental implementation of Off-policy Monte-Carlo control.}\n\t\t\\label{fig:rl_tabular_methods_offpolicy_MC_control}\n\t\\end{figure}\n\t\n\tFurthermore, to perform the averaging efficiently, we need to keep track of the normalization constant which is the sum of all importance weights for a certain state-action pair. This we will store in $C$. \n\t\n\tIn case we use a greedy policy for $\\pi$, we can simplify the algorithm further. In the importance weight, $\\pi(A_t|S_t)$ is 1 if $A_t$ was the greedy action, otherwise, we have a factor of 0 (which gives $\\rho=0$ for all previous actions). If we update our states in a reverse manner, this means that we can stop the loop as soon as we hit a sub-optimal action. \n\t\n\tNote that $W$ can get fairly large for long trajectories, as $b(A|S)$ is always smaller than 1. We need to use weighted importance sampling instead of ordinary as otherwise, the variance is too high. In addition, as only the tails of the episode are updated frequently, we might have an insufficient amount of samples for the states close to the start, which makes the algorithm inefficient.\n\t\n\t\n\\end{itemize}\n\\subsubsection{On-policy versus Off-policy control}\n\\label{sec:value_based_tabular_on_off_policy}\n\\begin{itemize}\n\t\\item After reviewing two alternative ways of learning a policy for a given environment, we can consider what are the advantages and drawbacks of each of the methods\n\t\\item In general, we can note that off-policy is actually a generalization of on-policy because if we set the behavior policy equal to our target policy, i.e. $b=\\pi$, then off-policy becomes on-policy\n\t\\item Commonly, on-policy converges faster as it uses the samples from the same policy it updates for. Off-policy can introduce variance by correcting the estimates for the target policy, as e.g. the importance weight can vary a lot if $\\pi$ and $b$ are quite different. This can lead to slow convergence.\n\t\\item A benefit of off-policy is however that we can learn from already recorded data, eventually from another source, as only our updates are based on the current policy $\\pi$, and not the actual samples as in on-policy. This means that we could use the same data to evaluate multiple policies, which especially helps for limited data/interactions with the environment.\n\t\\item Another point to consider is that off-policy methods allow us to learn the actual greedy policy. This is not possible in the on-policy setting because for guaranteeing the convergence to the correct $q$-values, we need to give every state-action pair a chance greater than zero to be visited. Using an $\\epsilon$-soft policy in the on-policy method and greedifying it afterwards can lead to a good approximation, but we also need to keep in mind that we then learn the optimal $\\epsilon$-soft policy, and not the strictly the optimal greedy policy. Hence, our final moves might be sub-optimal (remember Cliff-World, Sutton Book Example 6.6, page 132)\n\\end{itemize}\n\n\\subsection{Temporal-Difference Learning}\n\\begin{itemize}\n\t\\item Combining the ideas of Monte Carlo and Dynamic Programming, we arrive at a different type of methods, called Temporal difference learning. Remember that we can define the value function as a recursive function: $v(s)=\\E[R_{t+1}+\\gamma v(S_{t+1})|S_t=s]$. We can use this equality as a target instead of $G_t$, leading to the following update rule:\n\t$$v(s_t)\\leftarrow v(s_t)+\\alpha\\underbrace{\\left[R_{t+1}+\\gamma v(s_{t+1}) - v(s_t)\\right]}_{\\text{TD error } \\delta_t}$$\n\tInstead of waiting for a full episode to finish, we can perform this update after \\textit{a single action} taken in the environment. This method is called TD(0), and we will see later that this is a special case of TD($\\lambda$), or $n$-step TD\n\t\\item TD(0) is a \\textit{bootstrapping} method as it uses its own estimates as targets. \n\t\\item There are two main approaches for learning policies with Temporal Difference, namely \\textbf{SARSA} and \\textbf{Q-learning}, which we will now discuss in detail. Both learn the $q$-function, but have slightly different update rules.\n\t\n\tNote that TD learning can of course also be used for policy evaluation by simply performing the update rule above on samples from the original policy until our value function converges\n\\end{itemize}\n\\subsubsection{SARSA}\n\\begin{itemize}\n\t\\item The update rule of SARSA is as follows:\n\t$$Q(S_t,A_t)\\leftarrow Q(S_t,A_t) + \\alpha\\left[R_{t+1}+\\gamma Q(S_{t+1},A_{t+1})-Q(S_t,A_t)\\right]$$\n\twhere $A_{t+1}$ is selected by the policy $\\pi$ based on $S_{t+1}$. Note that $Q(S_{t+1},A_{t+1})=0$ for the terminal state.\n\t\\item The method got its name from using $\\bm{S}_t$, $\\bm{A}_t$, $\\bm{R}_{t+1}$, $\\bm{S}_{t+1}$, $\\bm{A}_{t+1}$ in its update rule.\n\t\\item Note that SARSA is a \\textbf{on-policy} method, meaning that it learns the $q$-values of the policy $\\pi$ (see Section~\\ref{sec:value_based_tabular_on_off_policy} for discussion of benefits and drawbacks)\n\t\\item Instead of just using the next sample to estimate $Q(S_{t+1},A_{t+1})$, we could also take our policy into account as we can calculate the expectation operator over it instead of simply sampling:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\tQ(S_t,A_t) & \\leftarrow Q(S_t,A_t) + \\alpha\\left[R_{t+1}+\\gamma \\E_{\\pi}[Q(S_{t+1},A_{t+1})|S_{t+1}]-Q(S_t,A_t)\\right]\\\\\n\t\t\t& \\leftarrow Q(S_t,A_t) + \\alpha\\left[R_{t+1}+\\gamma \\sum_{a} \\pi(a|S_{t+1}) Q(S_{t+1},a)-Q(S_t,A_t)\\right]\n\t\t\\end{split}\n\t\\end{equation*}\n\tThis method is also called \\textbf{expected SARSA}\n\t\\item Note that we can perform \\textbf{off-policy} control with expected SARSA, where we use a different behavior policy $b$ to sample, but learn the $q$-values of $\\pi$. A special case of this is when we choose $\\pi$ to be the greedy policy, which leads to the \\textbf{Q-learning algorithm}\n\\end{itemize}\n\n\\subsubsection{Q-Learning}\n\\begin{itemize}\n\t\\item As mentioned before, Q-learning applies a greedy policy in expected SARSA. This simplifies the update rule to:\n\t$$Q(S_t,A_t)\\leftarrow Q(S_t,A_t) + \\alpha\\left[R_{t+1}+\\gamma \\max_a  Q(S_{t+1},a)-Q(S_t,A_t)\\right]$$\n\t\\item It can be shown that Q-learning converges to the optimal $q_{*}$ under the condition, that the learning rate $\\alpha$ goes to zero (but not too fast), and every state-action pair is visited infinite amount of times when we have infinite number of steps.\n\t\\item However, there are also disadvantages of using the greedy policy. Suppose you have multiple actions with the same value $q(s,a)=0$ as ground truth. When learning it, we will have a certain amount of noise on it, so that some are slightly lower and other slightly above 0. When we now take the maximum, $\\max_a q(s,a)$, we get a positive value although the GT is zero. Hence, we have a positive bias, to which we also refer to as \\textbf{maximization bias}.\n\t\\item This bias can occur when we use a maximum operator in our update step. Hence, it is also the case for SARSA if it uses a $\\epsilon$-greedy policy\n\t\\item With infinite number of samples it might become less relevant, but we are usually limited in computational resources/time. Take for example the environment in Figure~\\ref{fig:rl_tabular_methods_maximization_bias}. The action of going left has a obviously lower expected reward, but due to the maximization bias, we will have for some actions from $B$ positive rewards (due to a high variance), and hence, we prefer going left. If we limit our number of samples, it is likely that we didn't get a accurate estimate of each of the actions in $B$ yet, and hence, still prefer to go left.\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.3\\textwidth]{figures/rl_tabular_methods_maximization_bias.png}\n\t\t\\caption{Example environment where the maximization bias can lead to a suboptimal policy.}\n\t\t\\label{fig:rl_tabular_methods_maximization_bias}\n\t\\end{figure}\n\n\t\\item To overcome this bias, we have to prevent to take the maximum of the estimates as an estimate of the maximum of the true values. So, intuitively, we need to determine the maximizing action from somewhere else than our estimates we are trying to update.\n\t\\item A simple method of doing so is \\textbf{Double Q-Learning}. Instead of learning a single $q$-function, we learn two, almost independently. Now, we can update $Q_1$ by using the maximum operator over $Q_2$, and vice versa. By that, we overcome the positive bias as even if $Q_1$ and $Q_2$ are biased themselves, they are positively biased on different actions. \n\t\\item The general update rule is then:\n\t\\begin{equation*}\n\t\t\\begin{split}\n\t\t\t\\text{Either update }Q_1: \\hspace{2mm} Q_1(S_t,A_t) & \\leftarrow Q_1(S_t,A_t) + \\alpha\\left[R_{t+1}+\\gamma Q_2\\left(S_{t+1},\\arg\\max_a Q_1(S_{t+1},a)\\right)-Q_1(S_t,A_t)\\right]\\\\\n\t\t\t\\text{Or update }Q_2: \\hspace{2mm} Q_2(S_t,A_t) & \\leftarrow Q_2(S_t,A_t) + \\alpha\\left[R_{t+1}+\\gamma Q_1\\left(S_{t+1},\\arg\\max_a Q_2(S_{t+1},a)\\right)-Q_2(S_t,A_t)\\right]\n\t\t\\end{split}\n\t\\end{equation*}\n\twhere we randomly assign a sample either to $Q_1$ or $Q_2$ (but not both, because we otherwise get the same bias).\n\\end{itemize}\n\n\\subsubsection{N-step TD learning}\n\\begin{itemize}\n\t\\item As we will discuss in Section~\\ref{sec:value_based_tabular_difference_TD_MC} in detail, both MC and TD have certain advantages and drawbacks. However, we can actually interpolate between these two, which we call $n$-step TD (a generalization of TD(0))\n\t\\item Instead of bootstrapping on the next state, we could use the reward of the $R_{t+2}$ as well, and then bootstrap on $v(S_{t+2})$. This leads to $2$-step TD. It gets obvious, that if we bootstrap always on the terminal state, meaning $\\infty$-TD, we arrive at MC as the value of a terminal state is zero. So, we approximate the return $G_{t:t+n}$ for $n$-step TD by:\n\t$$G_{t:t+n} = R_{t+1}+\\gamma R_{t+2} + ... + \\gamma^{n-1}R_{t+n} + \\gamma^n v_{t+n-1}(S_{t+n})$$\n\tThe update rule is hence:\n\t$$v_{t+n}(S_t) = v_{t+n-1}(S_t) + \\alpha \\left[G_{t:t+n}- v_{t+n-1}(S_t)\\right]$$\n\t\\item Which $n$ works best, depends on the environment (see Section~\\ref{sec:value_based_tabular_difference_TD_MC} for more detailed discussion)\n\t\\item To enable off-policy learning, we would require importance sampling as in MC which can introduce additional variance. However, there is an alternative in $n$-step, namely the $n$-step \\textbf{Tree Backup} algorithm\n\t\\item In $n$-step tree backup, we take the next $n$ steps into account, but at each action decision, we also look at all other actions. As visualized in Figure~\\ref{fig:rl_tabular_methods_n_step_tree_backup}, we now have multiple leaf nodes. At each of the leaves, we use our estimates.\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.1\\textwidth]{figures/rl_tabular_methods_n_step_tree_backup.png}\n\t\t\\caption{3-step tree backup as a backup diagram (see Section~\\ref{sec:value_based_tabular_backup_diagram}).}\n\t\t\\label{fig:rl_tabular_methods_n_step_tree_backup}\n\t\\end{figure}\n\t\\item Even if we sample from a behavior policy $p$, we can re-weight each of the leaves' contribution by the target policy $\\pi$. For example, at $S_{t+1}$, the leaves are weighted by $\\pi(A_1|S_{t+1})$ and $\\pi(A_3|S_{t+1})$, while the main sample trajectory gets a factor $\\pi(A_2|S_{t+1})$. The next leaves then have $\\pi(A_1|S_{t+2})\\pi(A_2|S_{t+1})$ etc.\n\t\\item For a 1-step tree backup, we get the same return estimate as for expected SARSA:\n\t$$G_{t:t+1}=R_{t+1}+\\sum_a \\pi(a|S_{t+1})Q_t(S_{t+1},a)$$\n\tIf we have $n$ steps, we can calculate our return estimate recursively:\n\t$$G_{t:t+n}=R_{t+1}+ \\gamma\\Bigg[\\underbrace{\\sum_{a\\neq A_{t+1}} \\pi(a|S_{t+1})Q_{t+n-1}(S_{t+1},a)}_{\\text{leaf contributions}} + \\underbrace{\\pi(A_{t+1}|S_{t+1})G_{t+1:t+n}}_{\\text{samples at }t+1}\\Bigg]$$\n\t\\item Our behavior policy therefore influences where we generate longer updates, but in expectation (due to the re-weighting), we still get the correct value estimates. Hence, we can use it on off-policy data without needing importance weights ($b$ influences just the depth of certain updates)\n\\end{itemize}\n\n\\subsection{Comparing tabular-based methods}\n\\begin{itemize}\n\t\\item We have seen now various methods and all take a slightly different approach to Reinforcement Learning\n\t\\item In this section, we compare and review all methods, and put things into perspective\n\\end{itemize}\n\\subsubsection{Difference of TD learning and Monte Carlo}\n\\label{sec:value_based_tabular_difference_TD_MC}\n\\begin{itemize}\n\t\\item TD can be implemented in an online, fully incremental fashion. In contrast, MC has to wait for the whole episode to finish which can delay learning in applications with very long episodes.\n\t\\item TD learning is more strongly influenced by the initial values we give the $v$/$q$-values (and hence by \\underline{biased}), which can slow down training. Especially in cases where we try to learn a policy (like Q-learning for TD), we can focus our exploration in the wrong direction for a long time before finding another, optimal case.\n\t\\item In general, MC suffer from \\underline{high variance}. This is because $G_t$ is approximated by a single sample, which is often not enough for a sufficient estimate. For example, assume we have a uniform policy over two actions, and an episode is always exactly 10 steps long. Then, the expectation would give each of the $2^{10}=1024$ possibilities a weight factor of $\\frac{1}{1024}$ but in MC, we simply pick one of them which clearly is inaccurate.\n\t\\item As usual, there is a trade-off version of both, which is $n$-step TD learning. Which $n$ to choose is highly dependent on your environment. For the extreme case of only having a single action to take, MC is clearly preferred because there is no variance in the updates. However, in the case where we can have many different outcomes from the same state, TD learning might be the better option. A common rule of thumb is that we need a lower learning rate for larger $n$ due to the increase of variance\n\t\\item Another interesting difference is that (batch) MC finds the estimates that minimizes the \\underline{mean-squared error} on the training set, whereas (batch) TD(0) finds the optimal estimates for a \\underline{maximum-likelihood model} of the Markov process. This means that it estimates the transition probability from state $i$ to $j$ as the fraction of observed transitions from $i$ to $j$, and its expected reward is the average of the rewards observed for this transition. Thus, TD learning exploits the Markov property of the environment while MC neglects it. Note that this property can make a big difference if we observe an insufficient number of data points for each state.\n\\end{itemize}\n\\subsubsection{Backup diagrams}\n\\label{sec:value_based_tabular_backup_diagram}\n\\begin{itemize}\n\t\\item Another way to visualize the difference between TD and MC is the usage of \\textit{backup diagrams}\n\t\\item A backup diagram visualizes a sequence of actions (black dots) and states (white nodes). We always go from a state to a action, which leads us to a next state. One example diagram was already given in Figure~\\ref{fig:rl_tabular_methods_n_step_tree_backup}.\n\t\\item Now, consider Figure~\\ref{fig:rl_tabular_methods_final_comparison} where we visualize the different extreme cases. TD learning has a shallow update (bootstraps on the next state), while Monte Carlo has infinite depth (i.e. going until the end). The trade-off here is obviously the variance/bias so that $n$-step TD is in between those.\n\t\n\t\\begin{figure}[ht!]\n\t\t\\centering\n\t\t\\includegraphics[width=0.4\\textwidth]{figures/rl_tabular_methods_final_comparison.png}\n\t\t\\caption{Putting all methods so far into perspective.}\n\t\t\\label{fig:rl_tabular_methods_final_comparison}\n\t\\end{figure}\n\t\n\tTo the right, we get \\textit{wider} updates, i.e. move from sampled updates towards expected updates. This means that we consider more actions and states we can end up in. Dynamic programming uses the whole environment dynamics, hence it takes all possible actions and next states into account. In between TD and DP, we can consider expected SARSA or Q-learning as they look at the next actions, but use bootstrapping. \n\t\n\tIf we make those methods deeper, we arrive at the $n$-step tree backup algorithm. Making dynamic programming deeper means that we consider \\textbf{all} possible outcomes from a given state, which includes possible actions we can get, and possible next states we can end up in. This obviously ends up in a huge, intractable graph for most environments (if we even have given the dynamics) so that it is mostly not feasible to perform.\n\\end{itemize}\n\\subsubsection{Limitations so far}\n\\begin{itemize}\n\t\\item To summarize the tabular-based methods, we want to review their limitations so far.\n\t\\item First of all, as the name indicates, for all the previous methods we store the $q$- and $v$-functions as a table. This is however not always possible. Suppose we have a continuous state space. Then we cannot create a table for that. Alternatively, imagine we want to play an Atari game. Having a screen resolution of $256\\times 256$, we would get $256\\times 256\\times 3\\times 256$ different frames (last two factors are channels and 8-bit values of channels) making it infeasible to store. However, it would be also extremely inefficient because similar frames mostly relate to similar actions to take. This leads us to approximate value-based learning methods which we will review in Section~\\ref{sec:value_based_approximation}.\n\t\\item Currently, we have to choose the (behavior) policy ourselves with which we explore the environment. Furthermore, if we want to learn an optimal stochastic policy, we also need to set $\\epsilon$ in $\\epsilon$-soft policies, or the temperature for softmax distributions. But not only for exploration we want randomness, as in partially observable states, we also have uncertainty which we have to take into account. The question arises whether we cannot learn the optimal stochasticity in the algorithm itself, which we will discuss in Section~\\ref{sec:policy_learning} and \\ref{sec:partially_observable}.\n\t\\item Until we have learned what the effect of our actions are, it takes quite some time for TD and MC to learn. If we want to take sample efficiency into account, we might want to consider model-based approaches as will be discussed in Section~\\ref{sec:model_based}.\n\\end{itemize}\n"
  }
]