Bayesian model comparison"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will do an example of model comparison in the Bayesian framework. To do so, we'll simulate some data with a quadratic relation and fit both linear and quadratic functions to it to see if we can prefer one model to another. As you might expect, the ability to distinguish may depend on the uncertainties in the data points.\n",
" \n",
"In the Bayesian framework, we compare two models with the posterior odds ratio:\n",
"$${P(M_1 | D) \\over P(M_2|D)} = {P(D|M_1) P(M_1)\\over P(D|M_2) P(M_2)}$$\n",
"In the case where there is no prior preference for one model over the other, $P(M_1) = P(M_2)$, this reduces to the Bayes factor:\n",
"$$BF = {P(M_1 | D) \\over P(M_2|D)} = {P(D|M_1)\\over P(D|M_2)}$$\n",
"For a model with parameters:\n",
"$$P(\\theta|M,D) = {P(D|\\theta,M) P(\\theta|M)\\over P(D|M)}$$\n",
"To compute the Bayes factor, we need to compute the evidence, $P(D|M)$ for each of the two models. This is given by:\n",
"$$P(D|M) = \\int P(D|\\theta,M) P(\\theta|M) d\\theta$$\n",
"Equivalently, this is the integral over the posterior:\n",
"$$P(D|M) = \\int P(\\theta|M,D) d\\theta$$\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's simulate a data set with a second order polynomial, but with significant uncertainties, and see if we can distinguish between a linear and a quadratic fit. Use numpy.polyval() to generate the data, and add noise assuming a normal distribution."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Text(0, 0.5, 'y')"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAVR0lEQVR4nO3df4zc913n8ecbx75sS2HjxgnJhp4TLrKAi5oNe2mCAZUE6rRXtSZqSRA6DERKEdeD6u6c2oAqQEBcIn6eECS0cLlTCQ7GcaIKSKO4BV0vdbqpk7glNU7T1MR27W0Tt0ANTtZv/pjvlvF6Zr3emc/Md+f7fEirmfnOdzxvfz1+7Xc+8/m8JzITSVJzfMOwC5AkDZbBL0kNY/BLUsMY/JLUMAa/JDXMecMuYDEuvPDCXLt27bDLkKRl5YknnvhSZq6Zv31ZBP/atWuZnp4edhmStKxExBc6bXeoR5IaxuCXpIYpFvwRsS4inmz7+WpEvCciVkfEIxFxoLq8oFQNkqQzFQv+zNyfmVdn5tXAdwFfAx4AtgCPZuaVwKPVbUnSgAxqqOdG4HOZ+QXg7cC91fZ7gY0DqkGSxOCC/1bgvur6xZl5BKC6vGhANUiSGEDwR8Qq4G3An53j426PiOmImJ6ZmSlTnCQ10CDO+N8MfCozj1a3j0bEJQDV5bFOD8rMezJzKjOn1qw5Y/2BJGmJBhH8P8K/DfMAPARsqq5vAh4cQA2StKzccvdj3HL3Y0X+7KLBHxGvAn4Q2Nm2eRvwgxFxoLpvW8kaJEmnK9qyITO/Brx23rYv05rlI0kaAlfuSlLDGPyS1DAGvyQ1jMEvSQ1j8EtSwxj8ktQwBr8kNYzBL0k1s2vvIfYePM6ez7/I+m272bX3UF//fINfkmpk195DbN25j5OzpwA4dPwEW3fu62v4G/ySVMBSe+3c9fB+Trw8e9q2Ey/PctfD+/tVmsEvSZ2UbJK2kMPHT5zT9qUw+CWpRi4dHzun7Uth8EtSjWzesI6xlStO2za2cgWbN6zr23MU7c4pSTo3GycnALhjx9OcnD3FxPgYmzes+/r2fjD4JalmNk5OcN/jBwHY/q7r+/7nO9QjSQ1j8EtSwxj8ktQwBr8k9Vnplgu9MvglaZ5egnsQLRd6ZfBLUpteg3sQLRd6ZfBLUpteg3sQLRd65Tx+SWrTa3BfOj7GoQ77nmvLhRLz9+d4xi9JbXrtlTOIlgu9MvglqU2vwb1xcoI7b76KVSta8ToxPsadN1/V15YLvXKoR5La9KNXTumWC70y+CVpnroHd68c6pGkhjH4JalhDH5JahiDX5IaxuCXpIYx+CWpYYoGf0SMR8SOiPhsRDwTEddHxOqIeCQiDlSXF5SsQZJ0utLz+H8H+KvMfEdErAJeBfwc8GhmbouILcAW4L2F65Ckgarz/P/IzDJ/cMQ3AU8BV2Tbk0TEfuCNmXkkIi4BPpaZC66Fnpqayunp6SJ1StKoiognMnNq/vaSQz1XADPAH0fE3oj4QES8Grg4M48AVJcXdSn49oiYjojpmZmZgmVKUrOUDP7zgGuA38/MSeCfaA3rLEpm3pOZU5k5tWbNmlI1SlLjlAz+F4AXMnNPdXsHrV8ER6shHqrLYwVrkCTNUyz4M/OLwN9HxNz4/Y3A3wIPAZuqbZuAB0vVIEk6U+lZPf8N+FA1o+c54Cdo/bK5PyJuAw4C7yxcgySpTdHgz8wngTM+UaZ19i9JGgJX7kpSwxj8kkbSLXc/xi13PzbsMmrJ4JekhjH4JalhDH5JahiDX5IaxuCXpIYx+CWpYQx+SWoYg1+SGsbgl1RLvSzA2rX3EHsPHmfP519k/bbd7Np7qM/VLW8Gv6SRsmvvIbbu3MfJ2VMAHDp+gq079xn+bQx+SSPlrof3c+Ll2dO2nXh5lrse3j+kiurH4Jc0Ug4fP3FO25vI4Jc0Ui4dHzun7U1k8EsaKZs3rGNs5YrTto2tXMHmDeu6PKJ5Sn8DlyQN1MbJCQDu2PE0J2dPMTE+xuYN676+XQa/pBG0cXKC+x4/CMD2d10/5Grqx6EeSWoYg19S7bgAqyyDX1KtuACrPINfUq24AKs8g19SrbgAqzyDX1KtuACrPINfUq24AKs85/FLqhUXYJVn8EuqnX4swHLhVncO9UhSwxj8ktQwBr8kNYzBL0kNY/BLUsMY/JLUMEWnc0bE88A/ALPAK5k5FRGrge3AWuB54Icz86WSdUiS/s0gzvi/PzOvzsyp6vYW4NHMvBJ4tLotSRqQYSzgejvwxur6vcDHgPcOoQ5JNeYCrHJKn/En8JGIeCIibq+2XZyZRwCqy4s6PTAibo+I6YiYnpmZKVymJDVH6TP+9Zl5OCIuAh6JiM8u9oGZeQ9wD8DU1FSWKlCSmqboGX9mHq4ujwEPANcCRyPiEoDq8ljJGiRJpysW/BHx6oh4zdx14E3Ap4GHgE3VbpuAB0vVIGl4brn7MW65+7Fhl6EOSg71XAw8EBFzz/MnmflXEfFJ4P6IuA04CLyzYA1SY82Frh+Sar5iwZ+ZzwGv77D9y8CNpZ5XkrQwV+5KUsMY/JLUMAa/JDWMwS9JDWPwS1LDGPyS1DAGv6S+27X3EHsPHmfP519k/bbd7Np7aNglqY3BL6mvdu09xNad+zg5ewqAQ8dPsHXnPsO/Rgx+aQQN84z7rof3c+Ll2dO2nXh5lrse3j+wGrQwg18aMcM+4z58/MQ5bdfgGfzSiBn2Gfel42PntF2Dd9bgj4h3R8QFgyhGUu+Gfca9ecM6xlauOG3b2MoVbN6wbiDPr7NbzBn/twCfjIj7I+KmqNptSqqnYZ9xb5yc4M6br2LVila8TIyPcefNV7FxcmIgz6+zO2vwZ+YvAFcCHwR+HDgQEb8WEd9WuDZJS9CvM+5e+ulvnJxg8nXjvOHy1Xx8yw2Gfs0saow/MxP4YvXzCnABsCMifr1gbZKWwDNunc1Z+/FHxM/Q+qasLwEfADZn5ssR8Q3AAeCOsiVKOlcbJye47/GDgF/EojMt5otYLgRuzswvtG/MzFMR8dYyZUmSSjlr8Gfm+xa475n+liNJKs15/JLUMAa/VFO9zKqRFmLwS1LDGPyS1DCLmdUjqWHmunuenD3F+m272bxh3TmvA3AaaX0Z/NKIWmrwduvuCbgIbEQ41CPpNMPu7qnyDH5Jpxl2d0+VZ/BLOs2wu3uqPINf6qKp8+jtpz/6DH6phob5nbl29xx9zuqRaqYOs2rs7jnaPOOXClnqUJGzalSawS/VjLNqVJrBL9WMs2pUWvHgj4gVEbE3Ij5c3b48IvZExIGI2B4Rq0rXIC0nzqpRaYM44/9ZoP0LW94P/FZmXgm8BNw2gBqkZcNZNSqtaPBHxGXAf6b1Xb1ERAA3ADuqXe4FNpasQVqONk5OMPm6cd5w+Wo+vuUGQ199VfqM/7dpfRn7qer2a4HjmflKdfsFoOMrOiJuj4jpiJiemZkpXKYkNUex4K++iP1YZj7RvrnDrtnp8Zl5T2ZOZebUmjVritQodTPMBVRSaSUXcK0H3hYRbwHOB76J1juA8Yg4rzrrvww4XLAG6ZzVYQFVHbhwa3QVO+PPzK2ZeVlmrgVuBXZn5o8CHwXeUe22CXiwVA0aruXa66YfC6h8x6A6G8Y8/vcC/z0inqU15v/BIdQgddXrAqpu7xgMf9XFQII/Mz+WmW+trj+Xmddm5n/IzHdm5r8MogZpsXpdQGXLBdWdK3dVW8MaKup1AZUtF1R3Br80T68LqGy5oLqzLbPUQS9tiTdvWMfWnftOG+5ZSssFZ9WoFINf6rO5dwZ37Hiak7OnmBgfY/OGdY2aCqp6M/ilAvwiE9WZY/yS1DAGvyQ1jMGvIuqwcnW5rhyWSjP41XeuXJXqzeBX39nrRqo3Z/Wo70r1uoHBdsd0No5GlWf86jt73Uj1ZvCr7+x1I9WbQz3qu15Xrl46PsahDiG/3HrdOFSkuvKMX0X08mXhvb5jkLQwz/hrbG4OetPOHO11I5Vl8KuWeu11Mzcd9OTsKdZv2+0vDqmNQz0aOS4gkxZm8GvkOB1UWpjBr5HjdFBpYQa/Ro5ffSgtzODXyHE6qLQwZ/WomGFNQ3U6qLQwg18jya8+lLpzqEeSGsYz/ppyAZJn6lIpnvHXkAuQJJVk8NdQXRYg+Z210mgy+GvIBUiSSjL4a8gFSJJKMvhryAVIkkpyVk9BS+2n7wIkSSUZ/DXVjwVITf0ilzlN/XtLZ1NsqCcizo+IxyPiqYj4TET8UrX98ojYExEHImJ7RKwqVYMk6Uwlx/j/BbghM18PXA3cFBHXAe8HfiszrwReAm4rWIMkaZ5iwZ8t/1jdXFn9JHADsKPafi+wsVQNkqQzFZ3VExErIuJJ4BjwCPA54HhmvlLt8gLQ8RPLiLg9IqYjYnpmZqZkmepgrmXEns+/yPptu101LI2QosGfmbOZeTVwGXAt8O2dduvy2Hsycyozp9asWVOyTM1jywhptA1kHn9mHgc+BlwHjEfE3Gyiy4DDpZ7XlgNLU5eWEZLKKDmrZ01EjFfXx4AfAJ4BPgq8o9ptE/BgqRq0NLaMkEZbyXn8lwD3RsQKWr9g7s/MD0fE3wJ/GhG/AuwFPliwhqHpR1vlYc1Dv3R8jEMdQt6WEdJoKBb8mfk0MNlh+3O0xvtHVrcxcmBZrL7dvGEdW3fuO224x5YR0uiwV08BdRgj72VWzsbJCe68+SpWrWi9PCbGx7jz5quWxS8tSWdny4YChj1G3o93HH5nrTS6PONfwFJnBQ27rXId3nFIqi+Dv4Bht1Ue9jsOSfVm8Bcw7DHyYb/jkFRvBn8hGycnmHzdOG+4fDUf33LDQD8YHfY7Dkn1NrLB3+ReM8N+xyGp3kZyVs9yn0ffD87KkdTNSJ7xO6tFkrobyeB3VoskdTeSQz32mukPh4ik0TSSZ/z9mNXS5A+HJY22kTzjn/sA944dT3Ny9hQT42Pn1B3TD4cljbKRDH7obVbLQh8OG/ySlruRDf5e9OvDYcfIJdXRSI7x98qWB5JGmcHfwai0PNj+rut91yHpDA71dNDrh8OSVGcGfxe2PJA0qhzqkaSGMfglqWFGeqjHIRpJOpNn/JLUMAa/JDWMwS9JDWPwS1LDGPyS1DAjPaunV84KkjSKPOOXpIYx+CWpYQx+SWoYg1+SGsbgl6SGKRb8EfGtEfHRiHgmIj4TET9bbV8dEY9ExIHq8oJSNUiSzlTyjP8V4H9k5rcD1wH/NSK+A9gCPJqZVwKPVrclSQNSLPgz80hmfqq6/g/AM8AE8Hbg3mq3e4GNpWqQJJ1pIGP8EbEWmAT2ABdn5hFo/XIALurymNsjYjoipmdmZgZRpiQ1QmRm2SeI+Ebgr4FfzcydEXE8M8fb7n8pMxcc54+IGeALSyzhQuBLS3zsIFhfb6yvN9bXm7rX9+8zc838jUVbNkTESuDPgQ9l5s5q89GIuCQzj0TEJcCxs/05nQo/hxqmM3NqqY8vzfp6Y329sb7e1L2+bkrO6gngg8AzmfmbbXc9BGyqrm8CHixVgyTpTCXP+NcD/wXYFxFPVtt+DtgG3B8RtwEHgXcWrEGSNE+x4M/M/wdEl7tvLPW8HdwzwOdaCuvrjfX1xvp6U/f6Oir+4a4kqV5s2SBJDWPwS1LDjEzwR8RNEbE/Ip6NiDPaQETEv4uI7dX9e6pFZYOqrWPfonn7vDEivhIRT1Y/7xtUfdXzPx8R+6rnnu5wf0TE71bH7+mIuGaAta1rOy5PRsRXI+I98/YZ6PGLiD+KiGMR8em2bYvqQxURm6p9DkTEpk77FKrvroj4bPXv90BEjHd57IKvhYL1/WJEHGr7N3xLl8cu+H+9YH3b22p7vm3SyvzHFj9+PcvMZf8DrAA+B1wBrAKeAr5j3j4/DfxBdf1WYPsA67sEuKa6/hrg7zrU90bgw0M8hs8DFy5w/1uAv6T1gf11wJ4h/lt/kdbClKEdP+D7gGuAT7dt+3VgS3V9C/D+Do9bDTxXXV5QXb9gQPW9CTivuv7+TvUt5rVQsL5fBP7nIv79F/y/Xqq+eff/BvC+YR2/Xn9G5Yz/WuDZzHwuM08Cf0qrJ1C79h5BO4Abq7UGxWX3vkXLyduB/5MtnwDGqwV4g3Yj8LnMXOpK7r7IzL8BXpy3eTF9qDYAj2Tmi5n5EvAIcNMg6svMj2TmK9XNTwCX9ft5F6vL8VuMxfxf79lC9VW58cPAff1+3kEZleCfAP6+7fYLnBmsX9+nevF/BXjtQKprM69v0XzXR8RTEfGXEfGdAy0MEvhIRDwREbd3uH8xx3gQbqX7f7hhHj9YXB+quhzHn6T1Dq6Ts70WSnp3NRT1R12Gyupw/L4XOJqZB7rcP8zjtyijEvydztznz1NdzD5FRatv0Z8D78nMr867+1O0hi9eD/wvYNcgawPWZ+Y1wJtptdD+vnn31+H4rQLeBvxZh7uHffwWqw7H8edptU3/UJddzvZaKOX3gW8DrgaO0BpOmW/oxw/4ERY+2x/W8Vu0UQn+F4Bvbbt9GXC42z4RcR7wzSztreaSROe+RV+XmV/NzH+srv8FsDIiLhxUfZl5uLo8BjxA6y11u8Uc49LeDHwqM4/Ov2PYx69ydG74K7r3oRrqcaw+TH4r8KNZDUjPt4jXQhGZeTQzZzPzFPCHXZ532MfvPOBmYHu3fYZ1/M7FqAT/J4ErI+Ly6qzwVlo9gdq19wh6B7C72wu/36oxwU59i9r3+Za5zxwi4lpa/zZfHlB9r46I18xdp/Uh4Kfn7fYQ8GPV7J7rgK/MDWsMUNczrWEevzaL6UP1MPCmiLigGsp4U7WtuIi4CXgv8LbM/FqXfRbzWihVX/tnRj/U5XkX83+9pB8APpuZL3S6c5jH75wM+9Plfv3QmnXyd7Q+8f/5atsv03qRA5xPa4jgWeBx4IoB1vY9tN6OPg08Wf28Bfgp4Keqfd4NfIbWLIVPAN89wPquqJ73qaqGuePXXl8Av1cd333A1ID/fV9FK8i/uW3b0I4frV9AR4CXaZ2F3kbrM6NHgQPV5epq3yngA22P/cnqdfgs8BMDrO9ZWuPjc6/BuVlulwJ/sdBrYUD1/d/qtfU0rTC/ZH591e0z/q8Por5q+/+ee8217Tvw49frjy0bJKlhRmWoR5K0SAa/JDWMwS9JDWPwS1LDGPyS1DAGvyQ1jMEvSQ1j8EtLEBH/qWomdn61WvMzEfEfh12XtBgu4JKWKCJ+hdaK8DHghcy8c8glSYti8EtLVPWK+STwz7RaRMwOuSRpURzqkZZuNfCNtL5V7fwh1yItmmf80hJFxEO0vgHqcloNxd495JKkRTlv2AVIy1FE/BjwSmb+SUSsAP5/RNyQmbuHXZt0Np7xS1LDOMYvSQ1j8EtSwxj8ktQwBr8kNYzBL0kNY/BLUsMY/JLUMP8KnP3iiq2o2t0AAAAASUVORK5CYII=\n",
"text/plain": [
"