233 lines
18 KiB
Plaintext
233 lines
18 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 37,
|
|
"id": "cce80b24",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import numpy as np\n",
|
|
"\n",
|
|
"plt.rcParams[\"font.family\"] = \"D2Coding\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d2df6767",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Geometric Interpretation of LR\n",
|
|
"\n",
|
|
"## Definition\n",
|
|
"$$X, W\\in \\R^D$$\n",
|
|
"\n",
|
|
"$$W^TX = \\lVert W \\rVert \\lVert X \\rVert \\cos \\theta$$\n",
|
|
"\n",
|
|
"$$\\lVert X-W \\rVert^2 = \\lVert X \\rVert ^2 + \\lVert W \\rVert - 2 W^TX$$\n",
|
|
"\n",
|
|
"\n",
|
|
"$$\\lVert X \\rVert \\cos \\theta = \\frac{W^T X} {\\lVert W \\rVert} $$\n",
|
|
"it means length of projected vector."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "28627e12",
|
|
"metadata": {},
|
|
"source": [
|
|
"$\\begin{cases}\n",
|
|
"W^TX_i \\geq b &\\to y_i = 1\\\\\n",
|
|
"W_TX_i < b &\\to y_i = 2\n",
|
|
"\\end{cases}$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7d9d80da",
|
|
"metadata": {},
|
|
"source": [
|
|
"$\\sum \\frac{1}{2} \\left\\Vert f(x_i;w) - y_i\\right\\Vert$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d0086883",
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "powershell"
|
|
}
|
|
},
|
|
"source": [
|
|
"## Likelihood\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 38,
|
|
"id": "81be7a4c",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAi0AAAGhCAYAAACtc4RMAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjYsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvq6yFwwAAAAlwSFlzAAAPYQAAD2EBqD+naQAAJxpJREFUeJzt3X9MXfX9x/H3LUaYWmgH1baANKRrgxZ1Moc4ZW3M/lmy4CqprgYaky3WdAY0YYjR4LKY2qiVus5lMfk2dbXEQVpr5mbWJiX2H/6ZzjVlOof9QSmoUIUWC9X2fvP+XC/C7aVw7z33nvM55/lIrpd77rmXA4d6Xvfz4/0JhcPhsAAAAHjcPLcPAAAAYC4ILQAAwAqEFgAAYAVCCwAAsAKhBQAAWIHQAgAArEBoAQAAViC0AAAAKxBaAACAFQgtAADAn6Fl//79smTJEqmtrZ3T/r29vVJVVSU5OTlSXV0tJ06cSOY4AQBAwCUUWh566CHZsGGDrFq1as6vqaurk7KyMhkcHJSKigrzegAAgLSGlp/+9Kfy0UcfyY9+9KM57a+tKt3d3bJlyxZZsGCBLF++XLq6uqS/vz/hAwUAAMF2RSI719TUJPTmR44ckaKiIlm0aJEJKlu3bpWCggI5fPiwFBYWxn3NxMSEuUVdvHhRTp8+Lfn5+RIKhRL6/gAAwB3hcFjOnDkjS5culXnz5mU+tCQqGjbUpk2bpLW1VV566SWzfSabN2+W3/72t+k8LAAAkCF9fX2mAcPzoUVTVlZWlnR0dMjY2JjU19fLiy++eNnXtLS0yGOPPTb5eGRkRK6//nrzQ+fm5qbzcAEAgENGR0eluLhY5s+f79Rbpje05OXlyalTp6S5uVkOHDhgtmkry8KFC2d8TXZ2trnF0sBCaAEAwC5ODu1wtE6LBpSpVq5cKQMDA7Jx40YpLS2VoaEh02KSyOwjAACAhEPLuXPnzNTls2fPmsGy0a+jY1F0cG17e/vk/itWrJDy8nIZHx+X4eFhaWpqksrKStNcBAAAkLbQ8vrrr5vCci+88IL89a9/NV8///zz5rnFixeb7hudHTTVrl27ZO/evWb0cE9Pj+zcuTOhAwQAAFChsI6W9fhAHh0bowNyGdMCAIAd0nH9Zu0hAABgBUILAACwAqEFAABYgdACAACsQGgBAABWILQAAAArpLWMPwDY5sIFkUOHRAYGRJYsEbnrLpGsLLePCoAitADAN/bsEWloEDl58tttujjttm0ia9e6eWQAFN1DAPBNYKmtnR5YVH9/ZLs+D8BdhBYAgaddQtrCEq8+eHRbY2NkPwDuIbQACDwdwxLbwhIbXPr6IvsBcA+hBUDg6aBbJ/cDkB6EFgCBp7OEnNwPQHoQWgAEnk5r1llCoVD853V7cXFkPwDuIbQACDytw6LTmlVscIk+bmujXgvgNkILAEikDktnp0hh4fTt2gKj26nTAriP4nIA8A0NJjU1VMQFvIrQAgBTaEBZvdrtowAQD91DAADACoQWAABgBUILAACwAqEFAABYgYG4gI/pAn/MhAHgF4QWwKf27ImsXDx1IUCtOaJF1Kg5AsBGdA8BPg0stbWXrlzc3x/Zrs8DgG0ILYAPu4S0hSUcvvS56LbGxsh+AGATQgvgMzqGJbaFJTa49PVF9gMAmxBaAJ/RQbdO7gcAXkFoAXxGZwk5uR8AeAWhBfAZndass4RCofjP6/bi4sh+AGATQgvgM1qHRac1q9jgEn3c1ka9FgD2IbQAPqR1WDo7RQoLp2/XFhjdTp0WADaiuBzgUxpMamqoiAvAPwgtgI9pQFm92u2jAABn0D0EAACsQGgBAABWILQAAAArEFoAAIAVCC0AAMAKhBYAAGAFQgsAALACoQUAAFiB0AIAAKxAaAEAAFYgtAAAACsQWgAAgBUILQAAwAqs8gzAURcuiBw6JDIwILJkichdd0VWmwaAVBFaADhmzx6RhgaRkye/3VZUJLJtm8jatW4eGQA/oHsIgGOBpbZ2emBR/f2R7fo8AKSC0ALAkS4hbWEJhy99LrqtsTGyHwAki9ACIGU6hiW2hSU2uPT1RfYDgGQRWgCkTAfdOrkfAMRDaAGQMp0l5OR+ABAPoQVAynRas84SCoXiP6/bi4sj+wFAsggtAFKmdVh0WrOKDS7Rx21t1GsBkBpCCwBHaB2Wzk6RwsLp27UFRrdTpwVAqiguB8AxGkxqaqiICyA9CC0AHKUBZfVqt48CgB/RPQQAAKxAaAEAAFYgtAAAACsQWgAAgD9DS29vr1RVVUlOTo5UV1fLiRMnZn3NwYMH5aabbpKrrrpKbr75ZvMYAAAgraGlrq5OysrKZHBwUCoqKmTDhg2X3T8cDst9990n999/v3z22Wfy4IMPyr333ivnz59P9FsjYHRF4K4ukfb2yD0rBANAsCUUWrRVpbu7W7Zs2SILFiyQ5cuXS1dXl/T398/4Gg0qevvlL38pV199tfzqV7+Szz///LKvAfbsEVm2TGTNGpH16yP3+li3AwCCKaHQcuTIESkqKpJFixaZ0LF161YpKCiQw4cPz/ga3ffGG2+UHTt2yJdffimvvPKKrFy5Uop1IZI4JiYmZHR0dNoNwaLBpLZW5OTJ6ds15+p2gkvw0OoGIOHQcvr0acnPzzdfb9q0SVpbW6WkpMRsn0koFJLXXntNnnnmGdPS8uSTT8ru3bvliivi17XbvHmz5OXlTd5mCjfwJ70YNTRot+Klz0W3NTZy0QoSWt0AJBVadHxKVlaWdHR0yNjYmNTX18uFWa4eX331lRnT0tzcLF988YUJJevWrZNz587F3b+lpUVGRkYmb319fYkcIiyn5d9jW1hig4v+Seh+8D9a3QAkHVq05ePUqVMmgPzpT38y27SVZeHChTO+RmcKnTlzRp544gnz+kceecS0vrz99ttx98/Ozpbc3NxpNwSHrlfj5H6wF61uAFIKLToWZWBgQDZu3CilpaUyNDRkWkJWrVo1uY+GmqnOnj1rgogGlSid+qzbgVi6wJ6T+8FetLoBSCm0rFixQsrLy2V8fFyGh4elqalJKisrJ8edaNdPYWGhtOtouW/cdtttJsi8/PLLpktp165d8sEHH8jtt9+eyLdGQOiKwEVFOhYq/vO6Xf/cdD/4G61uAFKu06KhY+/evbJ06VLp6emRnTt3Tj63ePFi052jM4qiNND85S9/kT/+8Y9mJpEGGx2I+73vfS/Rb42ArBC8bVvk69jgEn3c1hbZD/5GqxuAWKGwjq71MJ3yrGNhdFAu41uCQwdY6niGqd0D2sKigWXtWjePDJmiY1V0lpAOuo33fykNsdoqd/QoIRYIyvU7/rxjwGUaTGpqIuMVtPlfP01rlxAXp+C1uuksIQ0oU4MLrW5AMBFa4Fl6MVq92u2jgNvhtbPz0lY3bWGh1Q0IHkILAE+j1Q1AFKEFgOfR6gYgqdlDAAAAbiC0AAAAKxBaAACAFQgtAADACoQWAABgBUILAACwAlOeAThefp+aKgDSgdACIK1rRmn1Wi3HT/VaAKmiewiAY4FF1wmaGliULnio2/V5AEgFoQWAI11C2sISbzXm6LbGxsh+AJAsQguAlOkYltgWltjg0tcX2Q8AkkVoAZAyHXTr5H4AEA+hBUDKdJaQk/sBQDyEFgAp02nNOksoFIr/vG4vLo7sBwDJIrQASJnWYdFpzSo2uEQft7VRrwVAaggtgMN0hkxXl0h7e+Q+KDNmtA5LZ6dIYeH07doCo9up0wIgVRSXAxwU9OJq+jPW1FARF0B6hMLheJUVvGN0dFTy8vJkZGREcnNz3T4cYNbiarH/oqLdI7Q2AAiS0TRcv+keAhxAcTUASD9CC+AAiqsBQPoRWgAHUFwNANKP0AI4gOJqAJB+hBbAARRXA4D0I7QADqC4GgCkH6EFcAjF1QAgvSguBziI4moAkD6EFsBhGlBWr3b7KADAf+geAgAAViC0AAAAKxBaAACAFQgtAADACoQWAABgBUILAACwAqEFAABYgdACAACsQGgBAABWILQAAAArEFoAAIAVCC0AAMAKhBYAAGAFVnkGEFgXLogcOiQyMCCyZInIXXdFVukG4E2EFgCBtGePSEODyMmT324rKhLZtk1k7Vo3jwzATOgeAhDIwFJbOz2wqP7+yHZ9HoD3EFoABK5LSFtYwuFLn4tua2yM7AfAWwgtAAJFx7DEtrDEBpe+vsh+ALyF0AIgUHTQrZP7AcgcQguAQNFZQk7uByBzCC0AAkWnNessoVAo/vO6vbg4sh8AbyG0AAgUrcOi05pVbHCJPm5ro14L4EWEFgCBo3VYOjtFCgunb9cWGN1OnRbAmyguByCQNJjU1FARdyoqBMPrCC0AAksvyKtXu30U3kCFYNiA7iEACDgqBMMWhBYACDAqBMMmhBYACDAqBMMmjGkJEAbZAYhFhWDYhNASEAyyAxAPFYJhE7qHAiCdg+y09aarS6S9PXJPvzdgFyoEwyaEFp9L5yA7DTvLlomsWSOyfn3kXh8z0wCwBxWC4evQ0tvbK1VVVZKTkyPV1dVy4sSJWV9z8eJFeeqpp2ThwoWyYMECeeihh5I9XnhkkB1TJAH/oEIwfBta6urqpKysTAYHB6WiokI2bNgw62u2b98ub7/9thw+fFiOHTtm3gP2DrJjiiTgPxpMjh0TOXhQZPfuyP3RowQWWDwQV1tVuru7Zd++fabFZPny5dLW1ib9/f1SGBvRp9i6davs3btXijS2i8idd96Z+pHDtUF2ibTeUG0UsAcVguGrlpYjR46Y4LFo0SITVDSMFBQUmBaUy3UnjY6OyquvvipXXXWVaaV57733Ztx/YmLC7D/1Bm8NsmOKJADA86Hl9OnTkp+fb77etGmTtLa2SklJidk+k48//lg+//xzOXv2rGmpueeee2TdunVmnEs8mzdvlry8vMlbsV5R4alBdkyRBAB4PrSEw2HJysqSjo4OGRsbk/r6erkwy8AFbTmJhhFtlXnyySfl+PHjptUmnpaWFhkZGZm89Wk/Azw1yI4pknAaU+cBOD6mRVs+Tp06Jc3NzXLgwAGzTVtZdFbQTK655hrTLaSBRV199dWmtebTTz+Nu392dra5wVkaTGpqnKmIG2290VlCGlCmDshliiQSReFDAGlpaVm5cqUMDAzIxo0bpbS0VIaGhkxLyKpVqyb30VAz1Q033CDnzp0z+6ozZ86Yr5cuXZrIt4aDg+x+8YvIfSqhgimScAJT5wGkLbSsWLFCysvLZXx8XIaHh6WpqUkqKysnx51oF5DOImrXNt5vXHvttWa20NNPP21aZfT+xhtvNANyYTemSCIVTJ0HkPY6Lbt27TLTl7WlpKenR3bu3Dn53OLFiyU3N3eyKyhqx44d8s9//tO85tChQ9KpH8XhC0623iBYWF0YQKJCYR1d62E65VnH0uigXA1EAPxBG2R1+YfZaCuehmIAdknH9Zu1hwC4gqnzABJFaAHgCqbOA0gUoQWAK1hdGECiCC0AXMPUeQBpKy4HAF4ufAjA3wgtAFzH6sIA5oLQAsDTtLgcrTAAFKEFgGexLhH8hACeOgbiAvAk1iWCn+jf67JlImvWRIoq6r0+5u84MYQWAJ7DukTwEwK4cwgtADyHdYngFwRwZxFaAHiO9vk7uR/gFgK4swgtADyHdYngFwRwZzF7CIBn1yXSPv94zepa5l+fZ10ieH1WDwHcWbS0APAc1iWCX2b1sDCoswgtADzJ5nWJ9JN7V5dIe3vknkGWwZ3VQwB3Vigcjtf46h2jo6OSl5cnIyMjkpub6/bhAMgw2wpyURDPnr8rbVGZaZBstAvy6FFn/t7i/V1oC4sGFr/+XYym4fpNaAEAhz+5x/5fNfqJ2ustREGiLWDaFTSbgwedWxfLtgDuxes3A3EBIAP1ODS4aD0OXdHazxcqW7gxq4eFQVPHmBYAcAD1OOzCrB47EVoAwAHU47ALs3rsRGgBAAfwyd0uzOqxE6EFABzAJ3f72DytPqgYiAsADn5y19lDGlCmDsjlk7t3aTDRwdFBmtVjM0ILkIKgTWHE3D65x6vT4ud6HLZjVo89CC1Akigihnj45A6kD8XlgCRQRAwAMn/9ZiAu4HARMaVFxFhvBgCcRWgBEkQRMQBwB6EFSBBFxADAHYQWIEEUEQMAdxBagARRRAwA3EFoARJE+W8AcAehBUgC5b8BIPMoLgckiSJiAJBZhBYgBZT/BoDMoXsIAABYgZYWwGUsuggAc0NoAVzEootwAsEXQUH3EODyoouxSwL090e26/PAbPTvZNkykTVrRNavj9zrY/5+4Ees8gy49MlYLywzrWGk9V60xeXoUT4xY2asNj4dLU7ewirPgE+w6CJSxWrj09HiFAyEFsAFLLqIVBF8v0VXa3AQWgAXsOgiUkXwjaDFKVgILYALWHTRP/Ri2NUl0t4euc/UxZHgG0GLU7AQWgAXsOiiP7g5joLgG0GLU7AQWgCXsOii3dweR0HwjaDFKViY8gy4zA/TNP3wM9g6ZT1egUJtYdHAEoTg29Ehsm7d5ffR3wflA/xx/Sa0AEhJEKv66tgV7QqazcGDmVlQM2ihca7hcWqw0dYv2H/9pow/AMeLm0W7SPzazeW1cRRBXW18tkG4UQUFzn7foIZEL2BMC4CkBHmqKeMoghseKWLnLkILgKQEeaopM3eCGR7dHnwNQgsAn3SRZBIzd4IXHoPcsuglhBYASQl6FwlT1oMVHoPcsuglhBYASaGLJBJMjh2LzBLavTtyr1NrCSz+C49Bbln0EmYPAUjpU6725WtAmdpsHqQuEr/O3LFphowGk5qa9B5v0FsWvYI6LQBSEvTiZn4UxNo7c60Jo4Nu4101M1lQ0BYUlyO0AJ5k06dyJFd7J9p6FuTxOtHfjYrXshjk3008hBZCCwAEYnkCr6Jlce6oiAsASJtEZsj4cRzPXFoOMzF+BjMjtAAADGbIzG08j18HX9uAKc8AACPoM2SoeOt9hBYAgAS99g4Vb30aWnp7e6WqqkpycnKkurpaTpw4MefXvvPOOxIKhWT79u2JflsASIpeZLq6RNrbI/dcdGYW5OUJqHjr09BSV1cnZWVlMjg4KBUVFbJhw4Y5ve7rr7+WTZs2yXXXXZfMcQJAwliRN3FBXZ6A8Tw+HIirrSrd3d2yb98+WbBggSxfvlza2tqkv79fCmP/wmPoft///vdnbZmZmJgwt6lTpgDAqXoj0fEJfr4ApyqIM2SCPp7Hly0tR44ckaKiIlm0aJEJKlu3bpWCggI5fPjwZV+n+77wwgvy7LPPzvo9Nm/ebOZ1R2/F2oEKAAlgfELqojNkfvGLyL2fA0vQx/P4NrScPn1a8vPzzdfa1dPa2iolJSVm++U8+uij0tDQIEuXLp31e7S0tJhCNNFbn3YiAkACGJ+ARAV5PI9vQ4sWz83KypKOjg4ZGxuT+vp6uTDLR5X9+/fLu+++a4LLXGRnZ5vKeVNvAJAIxicgGUEdz+PbMS3aXXPq1Clpbm6WAwcOmG3ayrJw4cIZX/PWW2/JwMCAaZGJ7q8h5v3335dXXnkl1eMHgEswPgHJCuJ4HpsktPbQf//7X1m5cqVs2bJFfvOb38jQ0JBce+21cvz48cmxJxpqpnYDnTlzxrTKRK1du1Z+9rOfycMPP2wG886GtYcAJIoVeQH3peP6nVD30IoVK6S8vFzGx8dleHhYmpqapLKycjKw6CBanUXUrgURvjF//nxZvHjx5O3KK6802+YSWAAgGYxPAPwp4Totu3btkr1795rWlJ6eHtm5c+fkcxpKNE3pjCIAcBPjE4CAdw+5wZbuocutCgrAPfzbBPxz/WaV5wytCgrAHazIC/gHCyamiFVBAQDIDEJLCqi6CQBA5hBaUkDVTQAAMofQkgKqbgIAkDmElhRQdRMAgMxh9pADq4LOVnWTVUGB4GGqNeA8WlpSQNVNAPHorEFdRmDNGpH16yP3+pjZhEBqCC0pouomgKkogwCkDxVxHUJTMIDoQo0zzSpkoUYEySgVcb2LqpsAEimDwP8vgMTRPQQADqEMApBehBYAcAhlEID0IrQAgMNlEGJnE0bp9uJiyiAAySK0AIBDKIMApBehBQAcRBkEIH2YPQT4HNPxM0+DSU0Nv3fAaYQWwMe0kFlDw/RpuPqJX7sw+MSfXpRBAJxH9xDgU1RmBeA3hBbAp11C2sISr951dFtjY2Q/ALAFoQUIeGVWALAFoQXwISqzAvAjQgvgQ1RmBeBHhBbAh6jMCsCPCC2AD1GZFYAfEVoAn6IyKwC/obgc4GNUZgXgJ4QWwOeozArALwgt8DTWzQEARBFa4FmsmwMAmIqBuPAk1s0BAMQitMBzWDcHABAPoQWew7o5AIB4CC3wHNbNAQDEQ2iB57BuDgAgHkILPId1cwAA8RBa4DmsmwMAiIfQAk9i3RwAQCyKy8GzWDcnM6g6DMAWhBZ4GuvmpBdVhwHYhO4hIKCoOgzANoQWIICoOgzARoQWIICoOgzARoQWIICoOgzARoQWIICoOgzARoQWIICoOgzARoQWIICoOgzARoQWIKCoOgzANhSXAwKMqsMAbEJoAQKOqsMAbEH3EAAAsAKhBQAAWIHQAgAArEBoAQAAViC0AAAAKxBaAACAFQgtAADACoQWAABgBYrLAYjrwgXnKuU6+V4AgovQAuASe/aINDSInDw5fU0iXWQx0TWJnHwvAMFG9xCAS0JGbe30kKH6+yPb9Xk33gsAQuFwOCweNjo6Knl5eTIyMiK5ubluHw7ga9qNs2zZpSEjKhSKtJIcPTp7946T7wXAPum4ftPSAmCSjjuZKWQo/YjT1xfZL5PvBQBJhZbe3l6pqqqSnJwcqa6ulhMnTlx2//Pnz8vjjz8uy5cvl/nz58vdd98tH374Ib99wIN0oKxT+zn5XqnSVp+uLpH29si9PgYQgNBSV1cnZWVlMjg4KBUVFbJhw4bL7n/69Glze/PNN81rbr31VnnggQdSOWYAaaIze5zaz8n3SoWOm9FuqjVrRNavj9zrY8bTAD4f06KtKsuWLZNPPvlEFi1aJH/4wx/k17/+tZw8eVIKCwvn9B7/+c9/ZNWqVaYFJitOR/bExIS5Te0TKy4uZkwLkAHRcSg6UDbe/xmSGdPixHslKzoQOPb76/dWnZ3MYAJ8O6blyJEjUlRUZAJLf3+/bN26VQoKCuTw4cNzfo/jx4/L4sWL4wYWtXnzZvNDRm8aWABkhv6z1KnIUy/sUdHHbW1zCxlOvlcyNDTpVOt4gSm6rbGRriLAJgmFFu3myc/PN19v2rRJWltbpaSkxGyfq+eff14efPDBGZ9vaWkxqSx669ORegAyRlsetAUitvFUW0USbZlw8r0SxUBgIODF5bQnSVtIOjo6ZGxsTOrr6+XFF1+c8+u3bdtmWlreeOONGffJzs42NwDu0TBRU+NMFVsn3ysRXhoIDMCF0KLdNadOnZLm5mY5cOCA2aatLAsXLpz1td3d3fLUU09JV1eXXHPNNckfMYCM0FCxerX33muuvDIQGIBL3UMrV66UgYEB2bhxo5SWlsrQ0JDpvtGBtVEaamLpfuvWrZMtW7aY2UMAkG7amqPdULHjaaJ0uw6Z0/0A+DC0rFixQsrLy2V8fFyGh4elqalJKisrJwfL6iBanUXUrsUQvnHx4kVZv3693HHHHfLwww87/xMAgAcHAgPwQJ2WXbt2yd69e2Xp0qXS09MjO3funHxOZwXptCadUTR1mvT+/fvl9ddfl1AoNHnTbiIASCc3BwIDcB5rDwHwPZ3WnOmBwEDQjabh+p3QQFwAsJEbA4EBOI8FEwEAgBUILQAAwAqEFgAAYAVCCwAAsAKhBQAAWIHZQwBcwTRkAIkitADIuD17RBoapq/CrAXftIItBd8AzITuIQAJtY5oMWtdqUPv9XEygaW2dnpgUf39ke36PADEQ2gBMCcaJpYtE1mzRmT9+si9Pk4kZGjI0RaWeHW4o9saG5MLQwD8j9DiQ058GgbS0TqiY1hi3yM2uPT1RfYDgFiEFp9x4tMwkK7WER10Oxdz3Q9AsBBafISxAkgHJ1tHdJbQXMx1PwDBQmjxCcYKIF2cbB3Rac06SygUiv+8bi8ujuwHALEILT7BWAGki5OtI1qHRac1q9jgEn3c1ka9FgDxEVp8grECSBenW0e0Dktnp0hh4fTt+j10O3VaAMyE4nI+wVgBpEu0dUTHRWlAmdoFmWzriAaTmhoq4gJITCgcjjcKwjtGR0clLy9PRkZGJDc31+3D8Swdq6KzhHTQbbwzqhcX/SR79CgXhiByomR+vCq22sKigYXWEQCZuH7T0uIT6fg0DH9wqmQ+rSMA3EZLi8/waRjxpsHH/iuPBlnGkACw6fpNaPEhVs/F1C7DmWaV0WUIIJ3oHsKc6AVo9Wq3jwI2TYPn7wWADZjyDPgU0+AB+A2hBfAppsED8BtCC+BTlMwH4DeEFsCnKJkPwG8ILYCPUTIfgJ8wewjwOYrCAfALQgsQAEyDB+AHdA8BAAArEFoAAIAVCC0AAMAKhBYAAGAFQgsAALACoQUAAFiB0AIAAKxAaAEAAFYgtAAAACsQWgAAgBUILQAAwAqEFgAAYAVCCwAAsAKhBQAAWIHQAgAArEBoAQAAViC0AAAAK1zh9gEASI8LF0QOHRIZGBBZskTkrrtEsrLcPioASB6hBfChPXtEGhpETp78dltRkci2bSJr17p5ZACQPLqHAB8Gltra6YFF9fdHtuvzAGAjQgvgI9olpC0s4fClz0W3NTZG9gMA2xBaAB/RMSyxLSyxwaWvL7IfANiG0AL4iA66dXI/APASQgvgIzpLyMn9AMBLCC2Aj+i0Zp0lFArFf163FxdH9gMA2xBaAB/ROiw6rVnFBpfo47Y26rUAsBOhBfAZrcPS2SlSWDh9u7bA6HbqtACwFcXlAB/SYFJTQ0VcAP5CaAF8SgPK6tVuHwUAOIfuIQAAYAVCCwAAsAKhBQAAWIHQAgAArEBoAQAA/gwtvb29UlVVJTk5OVJdXS0nTpxIy2sAAABSCi11dXVSVlYmg4ODUlFRIRs2bEjLawAAAJKu06ItJN3d3bJv3z5ZsGCBLF++XNra2qS/v18KY8tvJvmaiYkJc4saGRkx96Ojo4kcKgAAcFH0uh0Oh51703AC/va3v4WLi4vN1ydPngyXlpaGCwoKwn//+98de01ra6v+dNy4cePGjRs3H9x6e3vDTkmopeX06dOSn59vvt60aZO0trbKSy+9ZLY79ZqWlhZ57LHHJh9/8cUXUlJSYlps8vLyEjlcpCE1FxcXS19fn+Tm5rp9OIHGufAOzoW3cD68Q3tKrr/+evnud7/r2HsmFFq0iScrK0s6OjpkbGxM6uvr5cUXX3T0NdnZ2eYWSwMLf4DeoOeBc+ENnAvv4Fx4C+fDO+bNm+dOaNHgcOrUKWlubpYDBw6YbdpisnDhQkdfAwAAECuh+LNy5UoZGBiQjRs3SmlpqQwNDZkmuFWrVk3uowEl0dcAAAA4GlpWrFgh5eXlMj4+LsPDw9LU1CSVlZWm/1Bt3rzZzAhqb2+f82tmo11FOg4mXpcRMotz4R2cC+/gXHgL58Pf5yKko3ETecG///1vU2elp6dHbrnlFvnzn/9sgonasWOHNDY2Smdnp/zkJz+Z02sAAADSEloAAADcwNpDAADACoQWAABgBUILAACwAqEFAABYwfXQ0tvbK1VVVZKTkyPV1dWmXH86XgNx/Pd6/vx5efzxx80imPPnz5e7775bPvzww4wdr5+l8jf+zjvvSCgUku3bt6f1GIMimXNx8eJFeeqpp0wRTV0o9qGHHsrIsQZBMufj4MGDctNNN8lVV10lN998s3mM1O3fv1+WLFkitbW1c9rfiWu366Glrq5OysrKZHBwUCoqKszU6HS8BuL471UrG+vtzTffNK+59dZb5YEHHsjY8fpZsn/jX3/9tVnj67rrrkv7MQZFMudCA+Pbb78thw8flmPHjpn3gDvnQyfI3nfffXL//ffLZ599Jg8++KDce++95kMXkqdBXH/3iRSKdeTaHXbR8ePHw6FQKPzpp5+ax9u3bzcrQupq0E6+BrNz4vfa09MTnjdvXvjrr79O45H6Xyrn4rnnngvX1dWFf/zjH4d///vfZ+Bo/S3Zc1FSUhJ+9913M3SUwZHM+fjkk0/MPnqvzp49ax5//PHHGTtuP3rjjTfM77K1tTV87733zrq/U9duV1tajhw5IkVFRbJo0SLp7++XrVu3SkFBgfl04uRrMDsnfq/Hjx+XxYsXmwUykflzofu+8MIL8uyzz2bsWP0umXOhTeC60vCrr75quiP0k+V7772X0eP2q2TOh+574403muKnX375pbzyyitmeZm5VmVHfDU1NXL11VfLXDl17XY1tGjXQn5+vvlam7S13G9JSYnZ7uRrMDsnfq/PP/+8aXqFO+fi0UcflYaGBlm6dGmGjtT/kjkXH3/8sXz++edy9uxZ02d/zz33yLp168w4F2T+fOj4rtdee02eeeYZc5F98sknZffu3XLFFQmtF4wUOXXtdjW0aF+jfirv6OiQsbExqa+vlwsXLjj+Gswu1d/rtm3bTEuLDsxF5s+FDoh79913TXCBu+diYmJici02/SSpF0n9t6GfNJH58/HVV1+ZMS3Nzc3yxRdfmPOiIfLcuXMZO26IY9duV6NmXl6eWRVa/5gOHDhgtmnq0hH3Tr4Gs0vl99rd3W1mSnR1dck111yTgaP1t2TOxVtvvWVWU9dPLtH9NcS8//77pjkcmTsX+m9Au4U0sCj9dK+fMD/99NOMHbdfJXM+dKbQmTNn5IknnjCtLo888oi89NJLZqD0z3/+8wwefbDlOXTtdrWlRfsV9X+0GzdulNLSUhkaGpK+vr5po5H1h0z0NcjMuVC6n35q2bJli5k9BHfOxe9+9zszluJf//qXuf3gBz+QlpYWee6551z4CYJ9Lm644QbzKV73VXrB1K/ptnPnfGg3na4yrIElSkOlbkf6pOva7Wpo0ZWey8vLZXx8XIaHh6WpqUkqKysnB0hpM15hYaG0t7fP+TXI3LnQPvr169fLHXfcIQ8//LCLR+8vyZwLrZOjg6CjtyuvvNJs0xohyOy5uPbaa+XOO++Up59+2nyS1HsdCKoDcpH583HbbbeZC+jLL79suiV27dolH3zwgdx+++0u/iT2O3funJm6rOFPu0SjX6f92h122fvvvx++5ZZbwldeeWX4hz/8YfjDDz+cfO7//u//wrm5ueF//OMfc34NMncujh49aqasxd4OHjzo0k/gH8n8u5iKKc/unov//e9/4dtvvz2cnZ0dvu2228IfffSRC0fuT8mcj3379oVXrVoV/s53vhO+4YYbwp2dnS4cub/s2LHjkv/36/TndF+7Q/ofpxMYAACA01yviAsAADAXhBYAAGAFQgsAALACoQUAAFiB0AIAAKxAaAEAAFYgtAAAACsQWgAAgBUILQAAwAqEFgAAIDb4fwRNjdbIN8icAAAAAElFTkSuQmCC",
|
|
"text/plain": [
|
|
"<Figure size 640x480 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"np.random.seed(7)\n",
|
|
"\n",
|
|
"#y = np.random.randint(0, 2, size=20)\n",
|
|
"\n",
|
|
"#x = np.random.normal(0, 0.08, size=(2, 20)) + np.vstack([y / 2 + 0.25, y / 2 + 0.25])\n",
|
|
"\n",
|
|
"\n",
|
|
"x = np.random.normal(0.5, 0.2, size=(40, 2), )\n",
|
|
"plt.xlim(0, 1)\n",
|
|
"plt.ylim(0, 1)\n",
|
|
"\n",
|
|
"plt.scatter(x[:, 0], x[:, 1], color='blue')\n",
|
|
"#plt.scatter(x[0, :][y == 0], x[1, :][y == 0], color=\"red\", label=\"y = 0\")\n",
|
|
"#plt.scatter(x[0, :][y == 1], x[1, :][y == 1], color=\"blue\", label=\"y = 1\")\n",
|
|
"\n",
|
|
"#plt.legend()\n",
|
|
"\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "df8f9d0b",
|
|
"metadata": {},
|
|
"source": [
|
|
"Maximize the probability"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "417911ef",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 40,
|
|
"id": "620ef30c",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"MLE estimated mu: [0.50995502 0.46521124]\n",
|
|
"MLE estimated sigma: [0.19530993 0.22297348]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# X가 2차원 데이터이므로 각 차원별로 평균(mu)과 표준편차(sigma)를 추정합니다.\n",
|
|
"# MLE: mu = sample mean, sigma = sample std (biased=False)\n",
|
|
"mu_mle = np.mean(x, axis=0)\n",
|
|
"sigma_mle = np.std(x, axis=0, ddof=0) # ddof=0 for MLE\n",
|
|
"\n",
|
|
"print(\"MLE estimated mu:\", mu_mle)\n",
|
|
"print(\"MLE estimated sigma:\", sigma_mle)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c51e029e",
|
|
"metadata": {},
|
|
"source": [
|
|
"to assign high probability to the observed data"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "36b7c733",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Logistic Regression RV\n",
|
|
"\n",
|
|
"$y_i \\in \\set{0, 1}$\n",
|
|
"$x_i \\in \\R^D$\n",
|
|
"\n",
|
|
"logistic regression function\n",
|
|
"$$f(X;W) = \\frac{1}{1+\\exp(-W^TX)}$$\n",
|
|
"\n",
|
|
"$$\\text{max}\\; L(w) = \\prod_{i=1}^{N}f(x_i)^{y_i} (1-f(x_i))^{1-y_i}$$\n",
|
|
"where\n",
|
|
"* $f(x_i)$: prob of choosing $y_i = 1$\n",
|
|
"* $1-f(x_i)$: prob of choosing $y_i = 0$\n",
|
|
"\n",
|
|
"and then update like\n",
|
|
"$$w_{t+1} = w_t + \\eta \\left . \\frac{dL}{dw}\\right\\vert_{w=w_t}$$\n",
|
|
"\n",
|
|
"how to get:\n",
|
|
"\n",
|
|
"$$w^* = \\argmax L(w) = \\argmax \\ln L(w)$$\n",
|
|
"\n",
|
|
"because function $\\ln$ is monotonically increasing function\n",
|
|
"\n",
|
|
"so $\\ln L(w) = \\sum_{i=1}^{N}{(y_i\\ln f(x_i) + (1-y_i)\\ln(1-f(x_i)) )}$\n",
|
|
"\n",
|
|
"미분하면 된다"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6b19eb64",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "cba17457",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "0b189852",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "2025-02-AI (3.12.11)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.12.11"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|