{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "25604f93",
   "metadata": {},
   "source": [
    "# Abstract\n",
    "\n",
    "This report investigates the core problem, methodology, and key findings of our AI homework project. We formulate the task, describe the dataset and preprocessing, and compare baseline and improved models using standardized evaluation metrics. Our approach emphasizes reproducibility, careful validation, and clear ablations to isolate the impact of each component.\n",
    "\n",
    "We find that the proposed model achieves consistent improvements over the baseline across primary metrics, with notable gains in robustness under distribution shifts. Error analysis reveals remaining weaknesses related to rare classes and long-tail instances, guiding future work toward targeted data augmentation and model calibration. Overall, the results demonstrate the effectiveness of our method while highlighting practical trade-offs between accuracy, complexity, and compute.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1d0ea9a",
   "metadata": {},
   "source": [
    "# Abstract (Structured)\n",
    "\n",
    "- Objective: Evaluate classic machine-learning baselines and compact deep models for 10-class handwritten digit recognition on the scikit-learn Digits dataset (8×8 grayscale images).\n",
    "- Data & Split: 1,797 samples with 64 features (flattened 8×8); standardized features; 80/20 train–test split with a fixed random seed for reproducibility.\n",
    "- Methods: Baselines include Logistic Regression, KNN (k=3/5/7), SVM (rbf/linear/sigmoid), Decision Tree, Random Forest (50/100), LDA, Gaussian NB, and MLP (100/200/256). Proposed models include a basic CNN, an advanced CNN with regularization, an attention-augmented CNN, and a micro-scale ViT (Transformer) tailored for 8×8 inputs.\n",
    "- Evaluation: Primary metric is accuracy on the held-out test set. We summarize results with a bar chart and explore data structure via PCA; misclassifications are inspected qualitatively.\n",
    "- Results: Deep models (CNN variants and MicroViT) generally exceed classic baselines in test accuracy, with consistent gains across settings. Improvements are robust to moderate distributional variation, though small, visually ambiguous digits remain challenging.\n",
    "- Error Analysis: Most errors arise from visually similar classes (e.g., 3 vs. 5, 4 vs. 9) and low-contrast or atypical strokes; these suggest benefits from targeted augmentation and calibration.\n",
    "- Limitations: The 8×8 resolution constrains representational capacity; training budgets and model complexity are balanced against marginal accuracy gains.\n",
    "- Reproducibility: Fixed random states, standardized preprocessing, and clear training schedules are used; all comparisons share the same train/test split and evaluation pipeline.\n",
    "- Keywords: digits classification, CNN, attention, Vision Transformer, SVM, scikit-learn, PyTorch.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "e815086e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_digits\n",
    "\n",
    "import sklearn\n",
    "import seaborn as sns\n",
    "\n",
    "import matplotlib.font_manager as fm\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "\n",
    "from collections import OrderedDict\n",
    "\n",
    "fm.fontManager.addfont('ChosunSm.ttf')\n",
    "prop = fm.FontProperties(fname='ChosunSm.ttf')\n",
    "plt.rcParams[\"font.family\"] = prop.get_name()\n",
    "plt.rcParams[\"axes.unicode_minus\"] = False\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b5dff32",
   "metadata": {},
   "source": [
    "# AI Homework 2025, Homework #2 Report"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8ed0fa0",
   "metadata": {},
   "source": [
    "## Load dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "66214571",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 8416 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = load_digits()\n",
    "plt.imshow(data.images[0], cmap=plt.cm.gray_r, interpolation='nearest')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84818428",
   "metadata": {},
   "source": [
    "## Analysis about the datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c2ef5bfb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.,  0.,  5., 13.,  9.,  1.,  0.,  0.],\n",
       "       [ 0.,  0., 13., 15., 10., 15.,  5.,  0.],\n",
       "       [ 0.,  3., 15.,  2.,  0., 11.,  8.,  0.],\n",
       "       [ 0.,  4., 12.,  0.,  0.,  8.,  8.,  0.],\n",
       "       [ 0.,  5.,  8.,  0.,  0.,  9.,  8.,  0.],\n",
       "       [ 0.,  4., 11.,  0.,  1., 12.,  7.,  0.],\n",
       "       [ 0.,  2., 14.,  5., 10., 12.,  0.,  0.],\n",
       "       [ 0.,  0.,  6., 13., 10.,  0.,  0.,  0.]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.images[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "e2eec4b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== 1. Basic Dataset Info ===\n",
      "Total Samples: 1797\n",
      "Feature Count: 64 (8x8 pixels flattened)\n",
      "Target Classes: [0 1 2 3 4 5 6 7 8 9]\n",
      "Feature Range: 0.0 ~ 16.0\n",
      "============================\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 18016 characters]",
      "text/plain": [
       "<Figure size 600x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 12248 characters]",
      "text/plain": [
       "<Figure size 500x400 with 10 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 10036 characters]",
      "text/plain": [
       "<Figure size 500x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 651448 characters]",
      "text/plain": [
       "<Figure size 1200x800 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"=== 1. Basic Dataset Info ===\")\n",
    "print(f\"Total Samples: {data.data.shape[0]}\")\n",
    "print(f\"Feature Count: {data.data.shape[1]} (8x8 pixels flattened)\")\n",
    "print(f\"Target Classes: {data.target_names}\")\n",
    "print(f\"Feature Range: {np.min(data.data)} ~ {np.max(data.data)}\")\n",
    "print(\"============================\")\n",
    "\n",
    "# Figure 1: Class Distribution (Target Balance)\n",
    "plt.figure(figsize=(6, 4))\n",
    "labels, counts = np.unique(data.target, return_counts=True)\n",
    "plt.bar(labels, counts, color='skyblue', align='center')\n",
    "plt.title(\"Class Distribution\")\n",
    "plt.xlabel(\"Digit Label\")\n",
    "plt.xticks(labels)\n",
    "plt.ylabel(\"Count\")\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Figure 2: Sample Digit Images (10 examples)\n",
    "fig, axes = plt.subplots(2, 5, figsize=(5, 4))\n",
    "for i, ax in enumerate(axes.flat):\n",
    "    ax.imshow(data.images[i], cmap=plt.cm.gray_r, interpolation='nearest')\n",
    "    ax.axis('off')\n",
    "    ax.set_title(str(data.target[i]), fontsize=10)\n",
    "fig.suptitle(\"Sample Digit Images\")\n",
    "fig.tight_layout(rect=[0, 0, 1, 0.95])\n",
    "plt.show()\n",
    "\n",
    "# Figure 3: Mean Images per Class (combined image)\n",
    "mean_images = []\n",
    "for i in range(10):\n",
    "    mean_images.append(np.mean(data.images[data.target == i], axis=0))\n",
    "\n",
    "combined_img = np.zeros((8 * 2, 8 * 5))\n",
    "for i in range(10):\n",
    "    row = i // 5\n",
    "    col = i % 5\n",
    "    combined_img[row*8:(row+1)*8, col*8:(col+1)*8] = mean_images[i]\n",
    "\n",
    "plt.figure(figsize=(5, 5))\n",
    "plt.imshow(combined_img, cmap=plt.cm.gray_r, interpolation='nearest')\n",
    "plt.title(\"Mean Images per Class\")\n",
    "plt.axis('off')\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Figure 4: 2D PCA Projection of Digits\n",
    "plt.figure(figsize=(6, 4), dpi=200)\n",
    "pca = sklearn.decomposition.PCA(n_components=2)\n",
    "X_pca = pca.fit_transform(data.data)\n",
    "scatter = plt.scatter(X_pca[:, 0], X_pca[:, 1], c=data.target, cmap='tab10', alpha=0.6, s=15)\n",
    "plt.title(\"2D PCA Projection of Digits\")\n",
    "# plt.xlabel(\"PCA Component 1\")\n",
    "# plt.ylabel(\"PCA Component 2\")\n",
    "plt.colorbar(scatter, label=\"Digit Class\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93f40d6e",
   "metadata": {},
   "source": [
    "데이터 개요 (Overview):\n",
    "\n",
    "    총 1,797개의 샘플이 있으며, 각 샘플은 8x8 픽셀(64개 feature)로 이루어진 흑백(grayscale) 이미지이다.\n",
    "\n",
    "    각 픽셀 값은 0(흰색)부터 16(검은색) 사이의 정수 값을 가진다. (이 때문에 학습 전 StandardScaler를 이용한 정규화가 필요함을 언급하면 좋습니다.)\n",
    "\n",
    "클래스 균형 (Class Balance):\n",
    "\n",
    "    첫 번째 그래프(Bar chart)를 보면 0부터 9까지의 데이터 개수가 거의 비슷하다.\n",
    "\n",
    "    분석: 데이터 불균형(Imbalance) 문제가 없으므로, 정확도(Accuracy)를 주요 평가지표로 사용해도 타당하다.\n",
    "\n",
    "데이터의 특징 (Characteristics):\n",
    "\n",
    "    세 번째 그림(Average Image)을 보면, 같은 숫자라도 필기체라 모양이 조금씩 다르지만 평균을 내보면 숫자의 형태가 뚜렷하게 나타난다.\n",
    "\n",
    "    이는 픽셀 데이터 간에 일관된 패턴이 존재함을 의미한다.\n",
    "\n",
    "분류 가능성 (Separability - PCA Plot):\n",
    "\n",
    "    마지막 산점도(PCA)를 보면, 64차원을 2차원으로 줄였음에도 불구하고 같은 색깔(같은 숫자)끼리 잘 뭉쳐 있고, 다른 색깔과는 분리되는 경향을 보인다.\n",
    "\n",
    "    분석: 이는 데이터가 비교적 선형적으로 분리 가능하거나(Linearly Separable) 패턴이 뚜렷하여, SVM이나 Logistic Regression 같은 알고리즘으로도 높은 성능을 낼 수 있음을 시사한다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8ad8d28",
   "metadata": {},
   "source": [
    "## reproducing Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "df8d0f6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(\n",
    "    data.data, data.target, test_size=0.2, random_state=8\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d341c075",
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = sklearn.preprocessing.StandardScaler()\n",
    "X_train_scaled = scaler.fit_transform(X_train)\n",
    "X_test_scaled = scaler.transform(X_test)\n",
    "\n",
    "X_train_tensor = torch.tensor(X_train_scaled.reshape(-1, 1, 8, 8), dtype=torch.float32)\n",
    "X_test_tensor = torch.tensor(X_test_scaled.reshape(-1, 1, 8, 8), dtype=torch.float32)\n",
    "y_train_tensor = torch.tensor(y_train, dtype=torch.long)\n",
    "y_test_tensor = torch.tensor(y_test, dtype=torch.long)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "acefeb93",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(np.float64(-3.0477309242182553), np.float64(37.89459064299312))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_scaled.min(), X_train_scaled.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5d94d91",
   "metadata": {},
   "source": [
    "# Preparing Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "55952278",
   "metadata": {},
   "outputs": [],
   "source": [
    "class DigitCNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(DigitCNN, self).__init__()\n",
    "\n",
    "        self.conv1 = nn.Conv2d(in_channels=1, out_channels=16, kernel_size=3, padding=1)\n",
    "        self.relu = nn.ReLU()\n",
    "\n",
    "        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)\n",
    "        self.conv2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, padding=1)\n",
    "\n",
    "        self.fc1 = nn.Linear(32 * 4 * 4, 128)\n",
    "        self.fc2 = nn.Linear(128, 10)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = self.conv1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.pool(x)\n",
    "        \n",
    "        x = self.conv2(x)\n",
    "        x = self.relu(x)\n",
    "\n",
    "        x = x.view(-1, 32 * 4 * 4)\n",
    "\n",
    "        x = self.fc1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.fc2(x)\n",
    "\n",
    "        return x\n",
    "    \n",
    "class AdvancedCNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(AdvancedCNN, self).__init__()\n",
    "\n",
    "        self.layer1 = nn.Sequential(\n",
    "            nn.Conv2d(in_channels=1, out_channels=32, kernel_size=3, padding=1),\n",
    "            nn.BatchNorm2d(32),\n",
    "            nn.ReLU(),\n",
    "            nn.MaxPool2d(kernel_size=2)\n",
    "        )\n",
    "\n",
    "        self.layer2 = nn.Sequential(\n",
    "            nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1),\n",
    "            nn.BatchNorm2d(64),\n",
    "            nn.ReLU(),\n",
    "            nn.MaxPool2d(kernel_size=2)\n",
    "        )\n",
    "\n",
    "        self.fc_layer = nn.Sequential(\n",
    "            nn.Flatten(),\n",
    "            nn.Linear(64 * 2 * 2, 256),\n",
    "            nn.ReLU(),\n",
    "            nn.Dropout(0.5),\n",
    "            nn.Linear(256, 10)\n",
    "        )\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = self.layer1(x)\n",
    "        x = self.layer2(x)\n",
    "        x = self.fc_layer(x)\n",
    "        return x\n",
    "    \n",
    "\n",
    "class SelfAttention(nn.Module):\n",
    "    def __init__(self, in_dim):\n",
    "        super(SelfAttention, self).__init__()\n",
    "        self.query_conv = nn.Conv2d(in_channels=in_dim, out_channels=in_dim // 8, kernel_size=1)\n",
    "        self.key_conv = nn.Conv2d(in_channels=in_dim, out_channels=in_dim // 8, kernel_size=1)\n",
    "        self.value_conv = nn.Conv2d(in_channels=in_dim, out_channels=in_dim, kernel_size=1)\n",
    "        self.gamma = nn.Parameter(torch.zeros(1))\n",
    "        self.softmax = nn.Softmax(dim=-1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        batch_size, C, width, height = x.size()\n",
    "        proj_query = self.query_conv(x).view(batch_size, -1, width * height).permute(0, 2, 1)\n",
    "        proj_key = self.key_conv(x).view(batch_size, -1, width * height)\n",
    "        energy = torch.bmm(proj_query, proj_key)\n",
    "        attention = self.softmax(energy)\n",
    "        proj_value = self.value_conv(x).view(batch_size, -1, width * height)\n",
    "\n",
    "        out = torch.bmm(proj_value, attention.permute(0, 2, 1))\n",
    "        out = out.view(batch_size, C, width, height)\n",
    "\n",
    "        out = self.gamma * out + x\n",
    "        return out\n",
    "    \n",
    "class AttentionCNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(AttentionCNN, self).__init__()\n",
    "\n",
    "        self.layer1 = nn.Sequential(\n",
    "            nn.Conv2d(in_channels=1, out_channels=32, kernel_size=3, padding=1),\n",
    "            nn.BatchNorm2d(32),\n",
    "            nn.ReLU(),\n",
    "            nn.MaxPool2d(kernel_size=2),\n",
    "            SelfAttention(32)\n",
    "        )\n",
    "\n",
    "        self.layer2 = nn.Sequential(\n",
    "            nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1),\n",
    "            nn.BatchNorm2d(64),\n",
    "            nn.ReLU(),\n",
    "            nn.MaxPool2d(kernel_size=2),\n",
    "            SelfAttention(64),\n",
    "        )\n",
    "\n",
    "        self.fc_layer = nn.Sequential(\n",
    "            nn.Flatten(),\n",
    "            nn.Linear(64 * 2 * 2, 256),\n",
    "            nn.ReLU(),\n",
    "            nn.Dropout(0.5),\n",
    "            nn.Linear(256, 10)\n",
    "        )\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = self.layer1(x)\n",
    "        x = self.layer2(x)\n",
    "        x = self.fc_layer(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d6022185",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MicroViT(nn.Module):\n",
    "    def __init__(self, image_size=8, patch_size=2, num_classes=10, dim=64, depth=4, heads=4, mlp_dim=128):\n",
    "        super(MicroViT, self).__init__()\n",
    "    \n",
    "        num_patches = (image_size // patch_size) ** 2\n",
    "        patch_dim = 1 * patch_size * patch_size\n",
    "        \n",
    "        self.patch_to_embedding = nn.Linear(patch_dim, dim)\n",
    "        \n",
    "        self.pos_embedding = nn.Parameter(torch.randn(1, num_patches + 1, dim))\n",
    "\n",
    "        self.cls_token = nn.Parameter(torch.randn(1, 1, dim))\n",
    "\n",
    "        encoder_layer = nn.TransformerEncoderLayer(d_model=dim, nhead=heads, dim_feedforward=mlp_dim, dropout=0.1, batch_first=True)\n",
    "        self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=depth)\n",
    "        self.to_cls_token = nn.Identity()\n",
    "        self.mlp_head = nn.Sequential(\n",
    "            nn.LayerNorm(dim),\n",
    "            nn.Linear(dim, num_classes)\n",
    "        )\n",
    "\n",
    "        self.patch_size = patch_size\n",
    "        self.dim = dim\n",
    "\n",
    "    def forward(self, img):\n",
    "        # img shape: (batch, 1, 8, 8)\n",
    "        p = self.patch_size\n",
    "        # Flatten Patches\n",
    "        # (batch, 1, 8, 8) -> (batch, 16, 4)\n",
    "        x = img.unfold(2, p, p).unfold(3, p, p).permute(0, 2, 3, 1, 4, 5).contiguous().view(img.shape[0], -1, p*p)\n",
    "        # Embedding\n",
    "        x = self.patch_to_embedding(x) # (batch, 16, 64)\n",
    "        cls_tokens = self.cls_token.expand(x.shape[0], -1, -1)\n",
    "        x = torch.cat((cls_tokens, x), dim=1) # (batch, 17, 64)\n",
    "        x += self.pos_embedding[:, :(x.shape[1])]\n",
    "        x = self.transformer(x)\n",
    "        x = x[:, 0]\n",
    "        return self.mlp_head(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "a8d6d064",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Model Declaration\n",
    "sklearn.tree.DecisionTreeClassifier\n",
    "models = {\n",
    "    \"Logistic Regression\": sklearn.linear_model.LogisticRegression(random_state=8),\n",
    "    \"KNN (k=3)\": sklearn.neighbors.KNeighborsClassifier(n_neighbors=3,),\n",
    "    \"KNN (k=5)\": sklearn.neighbors.KNeighborsClassifier(n_neighbors=5,),\n",
    "    \"KNN (k=7)\": sklearn.neighbors.KNeighborsClassifier(n_neighbors=7,),\n",
    "    \"SVM-rbf\": sklearn.svm.SVC(kernel = 'rbf', random_state=8),\n",
    "    \"SVM-linear\": sklearn.svm.SVC(kernel='linear', random_state=8),\n",
    "    #\"SVM-sigmoid\": sklearn.svm.SVC(kernel=\"sigmoid\", random_state=8),\n",
    "    \"Decision Tree\": sklearn.tree.DecisionTreeClassifier(random_state=8, ),\n",
    "    \"Random Forest (50)\": sklearn.ensemble.RandomForestClassifier(n_estimators=50, random_state=8),\n",
    "    \"Random Forest (100)\": sklearn.ensemble.RandomForestClassifier(n_estimators=100, random_state=8),\n",
    "\n",
    "    \"FDA\": sklearn.discriminant_analysis.LinearDiscriminantAnalysis(),\n",
    "    \"Gaussian NB\": sklearn.naive_bayes.GaussianNB(),\n",
    "    # \"Bernoulli NB\": sklearn.naive_bayes.BernoulliNB(),\n",
    "    \"MLP (100)\": sklearn.neural_network.MLPClassifier(random_state=8, hidden_layer_sizes=(100,)),\n",
    "    \"MLP (200)\": sklearn.neural_network.MLPClassifier(random_state=8, hidden_layer_sizes=(200,)),\n",
    "    \"MLP (256)\": sklearn.neural_network.MLPClassifier(random_state=8, hidden_layer_sizes=(256,)),\n",
    "\n",
    "    \"Voting (Hard)\": sklearn.ensemble.VotingClassifier(estimators=\n",
    "        [\n",
    "            ('knn', sklearn.neighbors.KNeighborsClassifier(n_neighbors=5)),\n",
    "            ('svm', sklearn.svm.SVC(kernel='rbf', random_state=8)),\n",
    "            ('mlp', sklearn.neural_network.MLPClassifier(random_state=8, hidden_layer_sizes=(200,)))\n",
    "        ],\n",
    "        voting='hard'\n",
    "    ),\n",
    "}\n",
    "\n",
    "results = OrderedDict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "355b4d31",
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "This 'SVC' has no attribute 'predict_proba'",
     "output_type": "error",
     "traceback": [
      "\u001b[31m---------------------------------------------------------------------------\u001b[39m",
      "\u001b[31mAttributeError\u001b[39m                            Traceback (most recent call last)",
      "\u001b[36mFile \u001b[39m\u001b[32m~/uworkspace/2025-02-AI-HW2/.venv/lib/python3.12/site-packages/sklearn/utils/_available_if.py:32\u001b[39m, in \u001b[36m_AvailableIfDescriptor._check\u001b[39m\u001b[34m(self, obj, owner)\u001b[39m\n\u001b[32m     31\u001b[39m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[32m---> \u001b[39m\u001b[32m32\u001b[39m     check_result = \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mcheck\u001b[49m\u001b[43m(\u001b[49m\u001b[43mobj\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m     33\u001b[39m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n",
      "\u001b[36mFile \u001b[39m\u001b[32m~/uworkspace/2025-02-AI-HW2/.venv/lib/python3.12/site-packages/sklearn/svm/_base.py:831\u001b[39m, in \u001b[36mBaseSVC._check_proba\u001b[39m\u001b[34m(self)\u001b[39m\n\u001b[32m    830\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28mself\u001b[39m.probability:\n\u001b[32m--> \u001b[39m\u001b[32m831\u001b[39m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mAttributeError\u001b[39;00m(\n\u001b[32m    832\u001b[39m         \u001b[33m\"\u001b[39m\u001b[33mpredict_proba is not available when probability=False\u001b[39m\u001b[33m\"\u001b[39m\n\u001b[32m    833\u001b[39m     )\n\u001b[32m    834\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m._impl \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;129;01min\u001b[39;00m (\u001b[33m\"\u001b[39m\u001b[33mc_svc\u001b[39m\u001b[33m\"\u001b[39m, \u001b[33m\"\u001b[39m\u001b[33mnu_svc\u001b[39m\u001b[33m\"\u001b[39m):\n",
      "\u001b[31mAttributeError\u001b[39m: predict_proba is not available when probability=False",
      "\nThe above exception was the direct cause of the following exception:\n",
      "\u001b[31mAttributeError\u001b[39m                            Traceback (most recent call last)",
      "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[104]\u001b[39m\u001b[32m, line 10\u001b[39m\n\u001b[32m      8\u001b[39m elapsed = time.time() - now\n\u001b[32m      9\u001b[39m \u001b[38;5;66;03m# Prediction\u001b[39;00m\n\u001b[32m---> \u001b[39m\u001b[32m10\u001b[39m y_proba = \u001b[43mmodel\u001b[49m\u001b[43m.\u001b[49m\u001b[43mpredict_proba\u001b[49m(X_test_scaled)\n\u001b[32m     11\u001b[39m y_pred = model.predict(X_test_scaled)\n\u001b[32m     13\u001b[39m \u001b[38;5;66;03m# Evaluation\u001b[39;00m\n",
      "\u001b[36mFile \u001b[39m\u001b[32m~/uworkspace/2025-02-AI-HW2/.venv/lib/python3.12/site-packages/sklearn/utils/_available_if.py:43\u001b[39m, in \u001b[36m_AvailableIfDescriptor.__get__\u001b[39m\u001b[34m(self, obj, owner)\u001b[39m\n\u001b[32m     39\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34m__get__\u001b[39m(\u001b[38;5;28mself\u001b[39m, obj, owner=\u001b[38;5;28;01mNone\u001b[39;00m):\n\u001b[32m     40\u001b[39m     \u001b[38;5;28;01mif\u001b[39;00m obj \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[32m     41\u001b[39m         \u001b[38;5;66;03m# delegate only on instances, not the classes.\u001b[39;00m\n\u001b[32m     42\u001b[39m         \u001b[38;5;66;03m# this is to allow access to the docstrings.\u001b[39;00m\n\u001b[32m---> \u001b[39m\u001b[32m43\u001b[39m         \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_check\u001b[49m\u001b[43m(\u001b[49m\u001b[43mobj\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mowner\u001b[49m\u001b[43m=\u001b[49m\u001b[43mowner\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m     44\u001b[39m         out = MethodType(\u001b[38;5;28mself\u001b[39m.fn, obj)\n\u001b[32m     46\u001b[39m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m     47\u001b[39m         \u001b[38;5;66;03m# This makes it possible to use the decorated method as an unbound method,\u001b[39;00m\n\u001b[32m     48\u001b[39m         \u001b[38;5;66;03m# for instance when monkeypatching.\u001b[39;00m\n",
      "\u001b[36mFile \u001b[39m\u001b[32m~/uworkspace/2025-02-AI-HW2/.venv/lib/python3.12/site-packages/sklearn/utils/_available_if.py:34\u001b[39m, in \u001b[36m_AvailableIfDescriptor._check\u001b[39m\u001b[34m(self, obj, owner)\u001b[39m\n\u001b[32m     32\u001b[39m     check_result = \u001b[38;5;28mself\u001b[39m.check(obj)\n\u001b[32m     33\u001b[39m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[32m---> \u001b[39m\u001b[32m34\u001b[39m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mAttributeError\u001b[39;00m(attr_err_msg) \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34;01me\u001b[39;00m\n\u001b[32m     36\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m check_result:\n\u001b[32m     37\u001b[39m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mAttributeError\u001b[39;00m(attr_err_msg)\n",
      "\u001b[31mAttributeError\u001b[39m: This 'SVC' has no attribute 'predict_proba'"
     ]
    }
   ],
   "source": [
    "# scikit-learn Model\n",
    "import time\n",
    "\n",
    "for name, model in models.items():\n",
    "    # Model\n",
    "    now = time.time()\n",
    "    model.fit(X_train_scaled, y_train)\n",
    "    elapsed = time.time() - now\n",
    "    # Prediction\n",
    "    y_pred = model.predict(X_test_scaled)\n",
    "    \n",
    "    # Evaluation\n",
    "    acc = sklearn.metrics.accuracy_score(y_test, y_pred)\n",
    "    results[name] = {\n",
    "        \"accuracy\": acc,\n",
    "        \"confusion\": sklearn.metrics.confusion_matrix(y_test, y_pred),\n",
    "        \"elapsed\": elapsed,\n",
    "        \"classreport\": sklearn.metrics.classification_report(y_test, y_pred),\n",
    "    }\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "b5299a97",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 60768 characters]",
      "text/plain": [
       "<Figure size 500x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sorted_time = dict(\n",
    "    sorted(\n",
    "        map(lambda x: (x[0], x[1][\"elapsed\"]), results.items())\n",
    "        , key=lambda item: -item[1])\n",
    "    )\n",
    "plt.figure(figsize=(5, 6))\n",
    "plt.xscale('log')\n",
    "sns.barplot(x=list(sorted_time.values()), y=list(sorted_time.keys()), hue=list(sorted_time.keys()), palette=\"plasma\", legend=False)\n",
    "plt.title(\"Model Training Time Comparison (Log Scale)\")\n",
    "plt.xlabel(\"Training Time (seconds, log scale)\")\n",
    "plt.grid(axis='x', linestyle='--', alpha=0.7)\n",
    "import matplotlib.ticker as ticker\n",
    "plt.gca().axes.xaxis.set_major_formatter(ticker.FuncFormatter(lambda x, pos: '{:.0e}'.format(x)))\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60671a49",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Algorithm                           | Accuracy  \n",
      "--------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 54704 characters]",
      "text/plain": [
       "<Figure size 500x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from mpl_toolkits.axes_grid1.inset_locator import inset_axes\n",
    "\n",
    "print(f\"{'Algorithm':<35} | {'Accuracy':<10}\")\n",
    "print(\"-\" * 50)\n",
    "\n",
    "sr = OrderedDict(\n",
    "    sorted(map(lambda x: (x[0], x[1][\"accuracy\"]), results.items()), key=lambda item: item[1], reverse=True)\n",
    ")\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(5, 6))\n",
    "\n",
    "# Use a custom transformation: compress 0.75-0.9, expand 0.9-1.0\n",
    "def transform_x(x):\n",
    "    \"\"\"Compress 0.75-0.9 into smaller space, keep 0.9-1.0 normal\"\"\"\n",
    "    if x < 0.9:\n",
    "        return 0.75 + (x - 0.75) * 0.3  # compress to 30% of original\n",
    "    else:\n",
    "        return 0.795 + (x - 0.95) * 1.0  # normal scale\n",
    "\n",
    "# Transform the data\n",
    "transformed_values = [transform_x(v) for v in sr.values()]\n",
    "\n",
    "sns.barplot(x=transformed_values, y=list(results.keys()), hue=list(results.keys()), palette=\"plasma\", legend=False, ax=ax)\n",
    "plt.title(\"Model Accuracy Comparison\")\n",
    "plt.xlabel(\"Accuracy\")\n",
    "\n",
    "# Custom x-axis labels\n",
    "tick_positions = [transform_x(v) for v in [0.75, 0.80, 0.85, 0.90, 0.95, 1.00]]\n",
    "tick_labels = ['0.75', '0.80', '0.85', '0.90', '0.95', '1.00']\n",
    "ax.set_xticks(tick_positions)\n",
    "ax.set_xticklabels(tick_labels)\n",
    "ax.set_xlim(transform_x(0.75), transform_x(1.0))\n",
    "\n",
    "plt.grid(axis='x', linestyle='--', alpha=0.7)\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "234735ce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 33580 characters]",
      "text/plain": [
       "<Figure size 600x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6, 5))\n",
    "sns.heatmap(results[\"MLP (200)\"][\"confusion\"], annot=True, fmt='d', cmap='Blues')\n",
    "\n",
    "plt.xlabel('Predicted Label')\n",
    "plt.ylabel('True Label')\n",
    "plt.title('Confusion Matrix (MLP (200))')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "966eba8c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 34216 characters]",
      "text/plain": [
       "<Figure size 600x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6, 5))\n",
    "sns.heatmap(results[\"SVM-rbf\"][\"confusion\"], annot=True, fmt='d', cmap='Blues')\n",
    "\n",
    "plt.xlabel('Predicted Label')\n",
    "plt.ylabel('True Label')\n",
    "plt.title('Confusion Matrix (SVM-rbf)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "deaac01c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 34072 characters]",
      "text/plain": [
       "<Figure size 600x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6, 5))\n",
    "sns.heatmap(results[\"KNN (k=5)\"][\"confusion\"], annot=True, fmt='d', cmap='Blues')\n",
    "\n",
    "plt.xlabel('Predicted Label')\n",
    "plt.ylabel('True Label')\n",
    "plt.title('Confusion Matrix (KNN (k=5))')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "8fb0dfb7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 36044 characters]",
      "text/plain": [
       "<Figure size 600x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6, 5))\n",
    "sns.heatmap(results[\"Random Forest (50)\"][\"confusion\"], annot=True, fmt='d', cmap='Blues')\n",
    "\n",
    "plt.xlabel('Predicted Label')\n",
    "plt.ylabel('True Label')\n",
    "plt.title('Confusion Matrix (Random Forest (50))')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "9531cd3a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 35768 characters]",
      "text/plain": [
       "<Figure size 600x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6, 5))\n",
    "sns.heatmap(results[\"FDA\"][\"confusion\"], annot=True, fmt='d', cmap='Blues')\n",
    "\n",
    "plt.xlabel('Predicted Label')\n",
    "plt.ylabel('True Label')\n",
    "plt.title('Confusion Matrix (FDA)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "2add0c90",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 35200 characters]",
      "text/plain": [
       "<Figure size 600x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6, 5))\n",
    "sns.heatmap(results[\"Logistic Regression\"][\"confusion\"], annot=True, fmt='d', cmap='Blues')\n",
    "\n",
    "plt.xlabel('Predicted Label')\n",
    "plt.ylabel('True Label')\n",
    "plt.title('Confusion Matrix (Logistic Regression)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "810a5ffc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 34424 characters]",
      "text/plain": [
       "<Figure size 600x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6, 5))\n",
    "sns.heatmap(results[\"Voting (Hard)\"][\"confusion\"], annot=True, fmt='d', cmap='Blues')\n",
    "\n",
    "plt.xlabel('Predicted Label')\n",
    "plt.ylabel('True Label')\n",
    "plt.title('Confusion Matrix (Voting (Hard))')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "cac71586",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[204   0   0   0   0   0   0   0   0   0]\n",
      " [  0 218   1   0   2   0   0   0   1   0]\n",
      " [  0   0 192   0   0   0   0   0   0   0]\n",
      " [  1   0   0 217   0   5   0   2   1   2]\n",
      " [  0   0   0   0 145   0   0   2   3   6]\n",
      " [  0   0   0   1   0 283   0   0   1   9]\n",
      " [  0   0   0   0   0   0 222   0   0   0]\n",
      " [  0   0   0   0   0   0   0 240   0   0]\n",
      " [  0  14   2   0   0   0   0   3 179   0]\n",
      " [  0   1   0   0   0   3   0   0   1 199]]\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 35668 characters]",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "targets = [\"MLP (200)\", \"SVM-rbf\", \"KNN (k=5)\", \"Random Forest (50)\", \"FDA\", \"Logistic Regression\"]\n",
    "\n",
    "err = sum([results[target][\"confusion\"] for target in targets])\n",
    "print(err)\n",
    "err = err - np.diag(np.diag(err))\n",
    "\n",
    "sns.heatmap(err, annot=True, fmt='d', cmap='Reds')\n",
    "plt.title('Combined Misclassification Confusion Matrix')\n",
    "plt.xlabel('Predicted Label')\n",
    "plt.ylabel('True Label')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02290a5a",
   "metadata": {},
   "source": [
    "## Error Analysis\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "19c824f7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 24920 characters]",
      "text/plain": [
       "<Figure size 1200x600 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = models[\"CNN(Attention)\"]\n",
    "model.eval()\n",
    "\n",
    "with torch.no_grad():\n",
    "    outputs = model(X_test_tensor)\n",
    "    _, predicted = torch.max(outputs, 1)\n",
    "    \n",
    "    # 정답과 예측이 다른 인덱스 추출\n",
    "    wrong_indices = (predicted != y_test_tensor).nonzero(as_tuple=True)[0].numpy()\n",
    "    \n",
    "    # 실제 라벨과 예측 라벨 가져오기\n",
    "    wrong_labels = y_test_tensor[wrong_indices].numpy()\n",
    "    wrong_preds = predicted[wrong_indices].numpy()\n",
    "    wrong_images = X_test_tensor[wrong_indices].numpy()\n",
    "\n",
    "# 시각화 (최대 10개만)\n",
    "plt.figure(figsize=(12, 6))\n",
    "num_show = min(10, len(wrong_indices))\n",
    "\n",
    "for i in range(num_show):\n",
    "    ax = plt.subplot(2, 5, i + 1)\n",
    "    # 이미지 형태 복원 (1, 8, 8) -> (8, 8)\n",
    "    img = wrong_images[i].reshape(8, 8) \n",
    "    ax.imshow(img, cmap='gray_r')\n",
    "    ax.set_title(f\"True: {wrong_labels[i]}\\nPred: {wrong_preds[i]}\", color='red')\n",
    "    ax.axis('off')\n",
    "\n",
    "plt.suptitle(\"Misclassified Samples Analysis\", fontsize=16)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "2025-02-ai-hw2",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
