commit 2953a90fc365b6e6841ffa0faf43f0b2573cefa1 Author: MatsuneMikuroi Date: Mon Aug 31 19:49:16 2026 +0000 Upload files to "/" diff --git a/ML_Course_Full_Summary.ipynb b/ML_Course_Full_Summary.ipynb new file mode 100644 index 0000000..78ed7cc --- /dev/null +++ b/ML_Course_Full_Summary.ipynb @@ -0,0 +1,829 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n# Machine Learning β€” Complete Course Summary\n\n**Course:** Machine Learning (Bachelor, Spring 2026) Β· **Framework:** PyTorch + scikit-learn\n**Built from:** 12 assignment solutions, the 2026 midterm, the mock exam, the two 2025 final exams and the 7 reference scripts in the course folder.\n\nThis notebook is a *study guide*, not a lecture transcript. Every section follows the same shape:\n\n| Block | What it gives you |\n|---|---|\n| **Theory** | The formulas and the one-paragraph \"why\", in the wording the course uses |\n| **Code** | Ready-to-run snippets, grouped so you can copy one block and adapt it |\n| **Exam pattern** | What this topic actually looked like when it was assessed, and the trap it hides |\n\n**Conventions used throughout**\n\n- Cells marked `# runnable` execute on their own with no download (synthetic data or tiny tensors).\n- Cells marked `# needs download` pull a torchvision dataset the first time.\n- Cells marked `# reference` are canonical implementations to copy, not to run top-to-bottom.\n- Every section ends with a πŸ“Œ **Exam pattern** box.\n\n> Run the **Setup** cell (Section 3.3) once before running anything else β€” it defines `set_seed`, `device`, and the imports the rest of the notebook assumes." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## πŸ“‘ Table of Contents\n\n**Part I β€” Foundations**\n\n1. [Course map & exam mechanics](#s1)\n2. [Mathematical foundations](#s2)\n - 2.1 [Linear algebra](#s2-1) Β· 2.2 [Calculus & gradient descent](#s2-2) Β· 2.3 [Loss functions](#s2-3) Β· 2.4 [LaTeX survival kit](#s2-4)\n3. [Python & PyTorch toolkit](#s3)\n - 3.1 [Python essentials](#s3-1) Β· 3.2 [NumPy essentials](#s3-2) Β· 3.3 [Setup: seeds & device](#s3-3) Β· 3.4 [Tensors & shape surgery](#s3-4)\n\n**Part II β€” Classical ML & the data pipeline**\n\n4. [Classical ML & optimisation](#s4)\n - 4.1 [Logistic regression](#s4-1) Β· 4.2 [Perceptron](#s4-2) Β· 4.3 [Hill climbing & local minima](#s4-3) Β· 4.4 [K-Means & PCA](#s4-4) Β· 4.5 [Decision trees](#s4-5)\n5. [The data pipeline](#s5)\n - 5.1 [Tabular preprocessing](#s5-1) Β· 5.2 [Datasets & DataLoaders](#s5-2) Β· 5.3 [Splits & data leakage](#s5-3)\n\n**Part III β€” Neural networks**\n\n6. [MLPs & the training loop](#s6)\n - 6.1 [Defining models](#s6-1) Β· 6.2 [Activation functions](#s6-2) Β· 6.3 [Canonical train/eval/plot helpers](#s6-3) Β· 6.4 [Hyperparameters](#s6-4)\n7. [Regularisation & generalisation](#s7)\n - 7.1 [BatchNorm, LayerNorm, Dropout, weight decay](#s7-1) Β· 7.2 [Reading loss curves](#s7-2) Β· 7.3 [Early stopping, schedulers, checkpointing](#s7-3)\n8. [Convolutional Neural Networks](#s8)\n - 8.1 [Convolution & pooling maths](#s8-1) Β· 8.2 [What filters detect](#s8-2) Β· 8.3 [CNN architectures](#s8-3)\n9. [Data augmentation & evaluation metrics](#s9)\n - 9.1 [torchvision v2 transforms](#s9-1) Β· 9.2 [Confusion matrix & classification report](#s9-2) Β· 9.3 [Inspecting predictions](#s9-3)\n10. [Transfer learning & fine-tuning](#s10)\n\n**Part IV β€” Generative & sequence models**\n\n11. [Autoencoders & Variational Autoencoders](#s11)\n12. [RNNs & LSTMs](#s12)\n13. [Attention & Transformers](#s13)\n14. [Vision Transformers & DINOv2](#s14)\n15. [Sets & point clouds (DeepSets)](#s15)\n\n**Part V β€” Exam preparation**\n\n16. [Exam patterns & drills](#s16)\n - 16.1 [Debug drill A β€” 6 syntax errors](#s16-1) Β· 16.2 [Debug drill B β€” 3 conceptual errors](#s16-2) Β· 16.3 [True/false theory bank](#s16-3) Β· 16.4 [By-hand computation drills](#s16-4) Β· 16.5 [Exam-day checklist](#s16-5)\n17. [Quick reference sheets](#s17)\n18. [Source map](#s18)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 1. Course map & exam mechanics\n\n[↑ TOC](#toc)\n\n## 1.1 The arc of the course\n\nThe course walks a single line: **linear algebra β†’ a single neuron β†’ a network β†’ convolutions β†’ generative models β†’ sequences β†’ attention β†’ transformers.** Every assignment adds exactly one idea to the previous one.\n\n| # | Assignment | Topic added | Dataset |\n|---|---|---|---|\n| 01 | Linear Algebra | vectors, norms, cosine similarity, projection | β€” |\n| 02 | Calculus | gradients, chain rule, one GD step, MSE | β€” |\n| 03 | Perceptron | logistic regression, perceptron from scratch, learning rate, hill climbing | Iris |\n| 04 | MLP | first neural network, train/evaluate functions | FashionMNIST |\n| 05 | MLP & Regularization | MLP as a class, BatchNorm / LayerNorm / Dropout | Titanic (CSV) |\n| 06 | CNN | conv & pooling maths, kernels, `SimpleCNN` | MNIST |\n| 07 | Augmentation & Metrics | `v2` transforms, confusion matrix, precision/recall/F1, early stopping | MNIST subset |\n| 08 | VAE | latent space, reparameterization trick, generation | MNIST |\n| 08.2 | Transfer Learning | ResNet18, freezing, fine-tuning | dogs vs cats |\n| 09 | RNN | tokenizer, Elman RNN from scratch, `nn.RNN`, text generation | lyrics.txt |\n| 10 | Transformers | attention by hand, self-attention, attention maps | toy sentence + point clouds |\n| 11 | Vision Transformers | patches, CLS token, positional embeddings, DINOv2 | CIFAR10 |\n\n## 1.2 How the exam is built\n\nThree blocks, consistently, across the midterm, mock exam and both 2025 finals:\n\n| Block | Points (of 64) | Shape |\n|---|---|---|\n| **1. Theory** | 20 | 10 Γ— true/false quizzes, 4 statements each. **4 correct = 2 pts, 3 correct = 1 pt, ≀2 correct = 0 pts.** |\n| **2. Hands-On** | 20 | By-hand computation, answers written in $\\LaTeX$ (decision tree traversal, loss computation, perceptron activation, embeddings, gradient descent) |\n| **3. Coding** | 24 | Debug a broken network (6 syntax errors), debug a training script (3 conceptual errors), implement a model, train + plot + evaluate |\n\n> The mock exam is 48 points and skips the \"debug\" questions; the midterm is 64 with an easier theory block.\n\n## 1.3 Rules that cost points\n\n- **Answer in English**, markdown cells for text, code cells for code.\n- **0 points** for incomplete answers. Partial credit only if the core idea is right *and* the instructions were followed.\n- **Ignoring instructions = 0**, no matter the effort. Wrong dataset = 0 for that question.\n- **Code that does not run = 50 % deduction.** Always `Restart Session & Run All` before submitting.\n- Only packages seen in the course: `numpy`, `pandas`, `matplotlib`, `seaborn`, `scipy`, `sklearn`, `torch`, `torchvision`, `tqdm`, `PIL`, `plotly`.\n- Open-book with **local PDFs only**. No internet, no AI, no extra browser tabs. `help()` and `dir()` are allowed β€” learn them.\n- Extra detail earns nothing. Answer exactly what is asked.\n\nπŸ“Œ **Exam pattern.** The single most repeated instruction is *\"show all steps in $\\LaTeX$\"*. A correct final number with no derivation loses most of the points on Hands-On questions." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 2. Mathematical foundations\n\n[↑ TOC](#toc)\n\nEverything the course asks you to compute by hand lives in this section. All of it is doable without a calculator, and all of it must be written in $\\LaTeX$." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 2.1 Linear algebra\n\n### Notation you are expected to write\n\n| Concept | $\\LaTeX$ | Renders as |\n|---|---|---|\n| Scalar in the reals | `$x \\in \\mathbb{R}$` | $x \\in \\mathbb{R}$ |\n| Bold vector in $\\mathbb{R}^3$ | `$\\mathbf{v} \\in \\mathbb{R}^3$` | $\\mathbf{v} \\in \\mathbb{R}^3$ |\n| Euclidean norm | `$\\parallel \\mathbf{v}\\parallel_2$` | $\\parallel \\mathbf{v}\\parallel_2$ |\n| Norm as dot product | `$\\sqrt{\\mathbf{v}^\\top \\mathbf{v}}$` | $\\sqrt{\\mathbf{v}^\\top \\mathbf{v}}$ |\n| Dot product | `$\\mathbf{a} \\cdot \\mathbf{b}$` | $\\mathbf{a} \\cdot \\mathbf{b}$ |\n\n### The five operations\n\n**Matrix–vector product.** Row of $A$ dotted with $\\mathbf{x}$, one row at a time:\n\n$$\nA\\mathbf{x} =\n\\begin{pmatrix} 1 & 2 \\\\ 3 & 4 \\end{pmatrix}\n\\begin{pmatrix} 5 \\\\ 6 \\end{pmatrix}\n=\n\\begin{pmatrix} 1\\cdot 5 + 2\\cdot 6 \\\\ 3\\cdot 5 + 4\\cdot 6 \\end{pmatrix}\n=\n\\begin{pmatrix} 17 \\\\ 39 \\end{pmatrix}\n$$\n\n**Identity matrix.** $A \\times I = A$. Nothing changes. (Asked verbatim in Assignment 5.)\n\n**Euclidean norm.** $\\parallel \\mathbf{v} \\parallel = \\sqrt{\\sum_i v_i^2}$. Memorise the Pythagorean triples that keep appearing: $(3,4)\\to5$, $(5,12)\\to13$, $(8,15)\\to17$.\n\n**Cosine similarity.**\n\n$$\\cos(\\theta) = \\frac{\\mathbf{u} \\cdot \\mathbf{v}}{\\parallel\\mathbf{u}\\parallel \\parallel\\mathbf{v}\\parallel}$$\n\nWorked example with $\\mathbf{u}=(3,4)$, $\\mathbf{v}=(5,12)$:\n\n$$\n\\mathbf{u}\\cdot\\mathbf{v} = 15+48 = 63,\\quad\n\\parallel\\mathbf{u}\\parallel = 5,\\quad\n\\parallel\\mathbf{v}\\parallel = 13,\\quad\n\\cos\\theta = \\frac{63}{65}\n$$\n\nReading the value: $+1$ = same direction, $0$ = orthogonal, $-1$ = opposite. Parallel vectors like $(1,2)$ and $(2,4)$ give exactly $1$ β€” magnitude is irrelevant, only direction counts.\n\n**Vector projection.**\n\n$$\\mathrm{proj}_{\\mathbf{b}}(\\mathbf{a}) = \\frac{\\mathbf{a} \\cdot \\mathbf{b}}{\\mathbf{b} \\cdot \\mathbf{b}}\\, \\mathbf{b}$$\n\nWith $\\mathbf{a}=(2,2)$, $\\mathbf{b}=(1,0)$: $\\frac{2}{1}(1,0) = (2,0)$. Geometrically, $\\mathbf{b}$ lies on the $x$-axis, so the projection keeps the $x$-component of $\\mathbf{a}$ and drops the $y$-component.\n\n**Orthogonality.** Two vectors are perpendicular **iff their dot product is 0**. $(5,2)\\cdot(4,-10) = 20-20 = 0$." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” linear algebra verification toolkit\nimport numpy as np\n\ndef cosine_similarity(u, v):\n u, v = np.asarray(u, dtype=float), np.asarray(v, dtype=float)\n return np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v))\n\ndef project(a, b):\n a, b = np.asarray(a, dtype=float), np.asarray(b, dtype=float)\n return (np.dot(a, b) / np.dot(b, b)) * b\n\ndef is_orthogonal(u, v, tol=1e-12):\n return abs(float(np.dot(u, v))) < tol\n\nA = np.array([[1, 2], [3, 4]])\nx = np.array([5, 6])\n\nprint(\"A @ x =\", A @ x) # matrix-vector: use @ or np.dot, NEVER *\nprint(\"A * I =\\n\", A @ np.eye(2)) # identity leaves A unchanged\nprint(\"norm (3,4) =\", np.linalg.norm([3, 4]))\nprint(\"cos((3,4),(5,12))=\", cosine_similarity([3, 4], [5, 12]), \"= 63/65\")\nprint(\"cos((1,2),(2,4)) =\", cosine_similarity([1, 2], [2, 4]), \"(parallel -> 1)\")\nprint(\"proj (2,2)->(1,0)=\", project([2, 2], [1, 0]))\nprint(\"(5,2) βŸ‚ (4,-10)? =\", is_orthogonal([5, 2], [4, -10]))" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "πŸ“Œ **Exam pattern.**\n- *\"Which pair of vectors are orthogonal?\"* (Midterm 2.1, 2 pts) β€” compute dot products until one is 0.\n- *\"Give two 2D vectors whose cosine similarity is βˆ’1, using only 1 and βˆ’1.\"* (Midterm 2.2, 5 pts) β€” answer $\\mathbf{u}=(1,1)$, $\\mathbf{v}=(-1,-1)$; then show $\\mathbf{u}\\cdot\\mathbf{v}=-2$, $\\parallel\\mathbf{u}\\parallel=\\parallel\\mathbf{v}\\parallel=\\sqrt2$, ratio $=-1$.\n- ⚠️ In NumPy `*` is **element-wise**. Matrix multiplication is `@` or `np.dot`." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 2.2 Calculus & gradient descent\n\n### Derivatives you need\n\n| $f$ | $f'$ |\n|---|---|\n| $x$ | $1$ |\n| $x^2$ | $2x$ |\n| $a x^2$ | $2ax$ |\n| $x^n$ | $n x^{n-1}$ |\n\n### Gradient\n\n$$\\nabla f = \\begin{pmatrix} \\partial f/\\partial x \\\\ \\partial f/\\partial y \\end{pmatrix}$$\n\nFor $f(x,y)=x^2+2y^2$: $\\ \\partial f/\\partial x = 2x$, $\\ \\partial f/\\partial y = 4y$, so $\\nabla f(1,1) = (2,4)$.\n\n**Interpretation (write this sentence):** *the gradient points in the direction of steepest increase, and its larger $y$-component shows the function is steeper in the $y$-direction.*\nContour lines are ellipses stretched along the $x$-axis β€” stretched **away** from the steep direction.\n\n### One step of gradient descent\n\n$$\\mathbf{x}_{\\text{new}} = \\mathbf{x}_{\\text{old}} - \\eta\\, \\nabla f(\\mathbf{x}_{\\text{old}})$$\n\nWith $\\eta = 0.1$ from $(1,1)$: $(1,1) - 0.1\\cdot(2,4) = (0.8, 0.6)$.\nThen always report both function values: $f(1,1)=3$, $f(0.8,0.6)=1.36$ β†’ **the value decreases, so the step moved downhill.**\n\n### Multivariable chain rule\n\n$$\\frac{df}{dt} = \\frac{\\partial f}{\\partial x_1}\\frac{dx_1}{dt} + \\dots + \\frac{\\partial f}{\\partial x_n}\\frac{dx_n}{dt}$$\n\nWith $x(t)=y(t)=t$ and $f=x^2+2y^2$: $\\ \\frac{df}{dt} = 2x(1) + 4y(1) = 6t$ β€” the function grows linearly in $t$ along the line $x=y$, reflecting the bowl shape." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” gradient descent, one step at a time\nimport numpy as np\nimport matplotlib.pyplot as plt\n\ndef f(x, y):\n return x**2 + 2 * y**2\n\ndef grad_f(x, y):\n return np.array([2 * x, 4 * y])\n\n# ---- one step, by the book -------------------------------------------------\np0 = np.array([1.0, 1.0])\neta = 0.1\ng = grad_f(*p0)\np1 = p0 - eta * g\n\nprint(f\"grad at {tuple(p0)} = {tuple(g)}\")\nprint(f\"step: {tuple(p0)} - {eta}*{tuple(g)} = {tuple(np.round(p1, 4))}\")\nprint(f\"f before = {f(*p0):.4f} f after = {f(*p1):.4f} -> decreased (downhill)\")\n\n# ---- full descent + contour plot ------------------------------------------\npath = [p0.copy()]\np = p0.copy()\nfor _ in range(15):\n p = p - eta * grad_f(*p)\n path.append(p.copy())\npath = np.array(path)\n\nX, Y = np.meshgrid(np.linspace(-2, 2, 400), np.linspace(-2, 2, 400))\nplt.figure(figsize=(6, 5))\nplt.contour(X, Y, f(X, Y), levels=20)\nplt.plot(path[:, 0], path[:, 1], \"o-\", color=\"crimson\", markersize=4, label=\"GD path\")\nplt.xlabel(\"x\"); plt.ylabel(\"y\")\nplt.title(r\"Gradient descent on $f(x,y)=x^2+2y^2$\")\nplt.legend(); plt.show()" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "πŸ“Œ **Exam pattern.** Assignment 2 Β§1.2 and Midterm 2.3 (6 pts) are the *same question* with different numbers ($f=2x^2+y^2$ from $(2,1)$, $\\eta=0.1$ β†’ $(1.2, 0.8)$, $f: 9 \\to 3.52$). The five sub-steps are always: partial derivatives β†’ gradient at the point β†’ one step β†’ both function values β†’ one-sentence interpretation." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 2.3 Loss functions\n\n### Mean Squared Error β€” regression\n\n$$L(w) = \\frac{1}{n}\\sum_{i=1}^{n}\\bigl(y_i - \\hat y_i(w)\\bigr)^2$$\n\nWorked example, model $\\hat y = wx$ with $w=1.5$:\n\n| $x$ | $y$ | $\\hat y$ |\n|---|---|---|\n| 1 | 2 | 1.5 |\n| 2 | 4 | 3.0 |\n| 3 | 6 | 4.5 |\n\n$$L(1.5) = \\tfrac13\\bigl[(0.5)^2+(1)^2+(1.5)^2\\bigr] = \\tfrac{3.5}{3} = 1.17$$\n\n### Binary Cross-Entropy β€” binary classification\n\n$$L = -\\frac{1}{n}\\sum_{i=1}^{n}\\Bigl[y_i \\log(p_i) + (1-y_i)\\log(1-p_i)\\Bigr]$$\n\nOnly one of the two terms survives per sample: if $y=1$ use $-\\log p$, if $y=0$ use $-\\log(1-p)$.\n\n### Categorical Cross-Entropy β€” multi-class\n\n$$L = -\\frac{1}{n}\\sum_{i}\\sum_{c} y_{i,c}\\log(p_{i,c})$$\n\nIn PyTorch this is `nn.CrossEntropyLoss()`, and it **already applies LogSoftmax internally**.\n\n### Which loss for which task\n\n| Task | Loss | PyTorch |\n|---|---|---|\n| Continuous target (test scores, prices) | MSE | `nn.MSELoss()` |\n| Multi-class, one label per sample | Categorical CE | `nn.CrossEntropyLoss()` |\n| Binary (cancer / no cancer, price above threshold) | Binary CE | `nn.BCELoss()` / `nn.BCEWithLogitsLoss()` |\n| Sequence next-token, ignoring padding | CE with mask | `nn.CrossEntropyLoss(ignore_index=pad_id)` |\n| VAE reconstruction | BCE (sum) + KL | see [Β§11](#s11) |" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” losses computed the exam way (no torch needed)\nimport numpy as np\nfrom math import log\n\n# --- MSE --------------------------------------------------------------------\nx = np.array([1, 2, 3]); y = np.array([2, 4, 6]); w = 1.5\ny_hat = w * x\nmse = np.mean((y - y_hat) ** 2)\nprint(f\"MSE = {mse:.2f}\")\n\n# --- MSE on 0/1 predictions (Exam SS25 2.2) ---------------------------------\nlabels = np.array([1, 1, 1, 0, 0, 0, 1, 1])\npreds = np.array([1, 0, 1, 1, 0, 0, 0, 0])\nprint(f\"MSE (0/1) = {np.mean((labels - preds) ** 2)} = {int(((labels-preds)**2).sum())}/{len(labels)}\")\n\n# --- Binary Cross-Entropy (Exam AS25 2.2) -----------------------------------\nlabel_prob = [(1, 0.9), (1, 0.7), (1, 0.4), (0, 0.3),\n (0, 0.1), (0, 0.2), (1, 0.8), (0, 0.6)]\nterms = [-(log(p) if t == 1 else log(1 - p)) for t, p in label_prob]\nfor (t, p), term in zip(label_prob, terms):\n print(f\" y={t} p={p} -> -log({p if t==1 else round(1-p,2)}) = {term:.4f}\")\nprint(f\"BCE = {np.mean(terms):.2f}\")" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "πŸ“Œ **Exam pattern.** AS25 2.2 gives 8 (label, probability) pairs and 4 points: 2 for writing the formula, 1 for the calculation in a code cell, 1 for the number rounded to 2 decimals. `from math import log, exp, sqrt` is provided as your \"calculator\" β€” you are expected to use it, not to estimate." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 2.4 $\\LaTeX$ survival kit\n\nYou will be graded on notation you type into markdown cells, with no internet. Keep this table.\n\n| Need | Type | Renders |\n|---|---|---|\n| inline / display | `$...$` / `$$...$$` | β€” |\n| reals | `\\mathbb{R}` | $\\mathbb{R}$ |\n| bold vector | `\\mathbf{v}` | $\\mathbf{v}$ |\n| norm | `\\parallel \\mathbf{v} \\parallel` | $\\parallel \\mathbf{v} \\parallel$ |\n| fraction | `\\frac{a}{b}` | $\\frac{a}{b}$ |\n| square root | `\\sqrt{x}` | $\\sqrt{x}$ |\n| sum | `\\sum_{i=1}^{n}` | $\\sum_{i=1}^{n}$ |\n| partial | `\\frac{\\partial f}{\\partial x}` | $\\frac{\\partial f}{\\partial x}$ |\n| gradient | `\\nabla f` | $\\nabla f$ |\n| transpose | `\\mathbf{v}^\\top` | $\\mathbf{v}^\\top$ |\n| hat | `\\hat{y}` | $\\hat{y}$ |\n| approx / in / cdot | `\\approx \\in \\cdot` | $\\approx\\ \\in\\ \\cdot$ |\n| floor | `\\lfloor x \\rfloor` | $\\lfloor x \\rfloor$ |\n| eta / theta / sigma / mu | `\\eta \\theta \\sigma \\mu` | $\\eta\\ \\theta\\ \\sigma\\ \\mu$ |\n\n**Column vector**\n```latex\n$\\begin{pmatrix} 1 \\\\ 2 \\end{pmatrix}$\n```\n\n**Matrix**\n```latex\n$\\begin{bmatrix} 1 & 2 \\\\ 3 & 4 \\end{bmatrix}$\n```\n\n**Aligned multi-step derivation** β€” this is the one that earns Hands-On points:\n```latex\n$\\begin{aligned}\nh[0,0,0] &= 3(-1 \\times 1) + 3(2 \\times 4) + 3(-1 \\times 1) \\\\\n &= 3(-1) + 3(8) + 3(-1) \\\\\n &= 18\n\\end{aligned}$\n```\n\n$$\\begin{aligned}\nh[0,0,0] &= 3(-1 \\times 1) + 3(2 \\times 4) + 3(-1 \\times 1) \\\\\n &= 3(-1) + 3(8) + 3(-1) \\\\\n &= 18\n\\end{aligned}$$\n\n> πŸ’‘ In Colab, `Ctrl+M+M` converts a cell to markdown, `Ctrl+M+Y` to code. Double-click any rendered markdown cell to see the source that produced it." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 3. Python & PyTorch toolkit\n\n[↑ TOC](#toc)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 3.1 Python essentials\n\nThe readiness check in Assignment 1 defines the floor: if any of this is unfamiliar, fix it before anything else.\n\n**Conventions the course enforces:** variables `snake_case`, classes `CamelCase`, and *cell execution order matters* β€” a variable only exists after its defining cell has run." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” the Assignment-1 readiness check, condensed\nimport math\nimport numpy as np\n\n# f-strings\nname, age = \"Olivia\", 29\nprint(f\"My name is {name}, I am {age} years old.\")\n\n# def vs lambda\ndef square(x):\n return x ** 2\n\nadd_one = lambda x: x + 1\nprint(square(5), add_one(10))\n\n# types and length\nfor var in [3, 3.14, \"hello\", [1, 2, 3]]:\n print(var, type(var))\n\nmy_list = [1, 2, 3, 4, 5]\nprint(my_list, \"len =\", len(my_list))\n\n# classes and inheritance (the pattern every nn.Module follows)\nclass Animal:\n def __init__(self):\n self.legs = 4\n\n def speak(self):\n print(\"The animal makes a sound\")\n\nclass Dog(Animal): # inherits __init__, so Dog also has .legs\n def speak(self): # overrides the parent method\n print(\"The dog barks\")\n\nAnimal().speak()\nd = Dog(); print(d.legs); d.speak()" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 3.2 NumPy essentials\n\n`len(array)` gives the **first** dimension; `array.shape` gives all of them. For a $(3,4)$ array, `len` is 3." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable\nimport numpy as np\n\nA = np.array([[1, 2, 3, 4],\n [5, 6, 7, 8],\n [9, 10, 11, 12]])\n\nprint(A)\nprint(\"len :\", len(A)) # 3 -> first dimension only\nprint(\"shape :\", A.shape) # (3, 4)\nprint(\"mean :\", np.mean(A), \"| axis=0:\", np.mean(A, axis=0))\nprint(\"linspace:\", np.linspace(-2, 2, 5))\nprint(\"meshgrid shapes:\", [g.shape for g in np.meshgrid(np.arange(3), np.arange(4))])\nprint(\"argmax:\", np.argmax([0.1, 0.7, 0.2]), \"| where:\", np.where(np.array([0, 1, 0]) == 0, -1, 1))\nprint(\"permutation:\", np.random.default_rng(0).permutation(5))" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 3.3 Setup: seeds, device, imports\n\n**Run this cell once.** Everything below assumes `set_seed`, `device` and these imports exist. This is also the exact boilerplate every assignment and exam starts with β€” memorise it." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” THE SETUP CELL. Run this before anything else.\nimport os, copy, math, random\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nimport torchvision\nimport torchvision.datasets as datasets\nimport torchvision.transforms as transforms\nfrom torchvision.transforms import v2\nfrom torch.utils.data import Dataset, DataLoader, TensorDataset, Subset, random_split\n\nfrom tqdm.auto import tqdm\n\nsns.set_theme(rc={\"figure.figsize\": (8, 6)}, style=\"whitegrid\")\n\n\ndef set_seed(seed):\n \"\"\"Full reproducibility across random / numpy / torch / cuda.\"\"\"\n random.seed(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.cuda.manual_seed_all(seed)\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = False\n\n\nset_seed(42)\n\n# Exams add this line to force bit-identical results:\n# torch.use_deterministic_algorithms(True)\n\ng = torch.Generator().manual_seed(42) # for DataLoader(generator=g)\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(\"device:\", device, \"| torch\", torch.__version__)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "πŸ“Œ **Exam pattern.** The `set_seed` + `g` + `device` block is given to you as a *DO NOT EDIT* cell. One of the six planted syntax bugs is usually a bad `.to(...)` call β€” `model.to('gpu1')`, `model.to('gpu-cluster')`. The fix is always `model.to(device)`." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 3.4 Tensors & shape surgery\n\nImages are tensors of shape $(C, H, W)$; batches are $(B, C, H, W)$. Nearly every runtime error in this course is a shape error, so learn these six operations.\n\n| Operation | What it does | Typical use |\n|---|---|---|\n| `.view(B, -1)` / `.reshape(...)` | flatten keeping the batch | feed images to an MLP |\n| `.squeeze()` / `.unsqueeze(0)` | drop / add a size-1 axis | show a grayscale image, add a batch dim |\n| `.permute(1, 2, 0)` | reorder axes | $(C,H,W)\\to(H,W,C)$ for `imshow` |\n| `.unfold(dim, size, step)` | sliding window | cut an image into ViT patches |\n| `torch.stack` / `torch.cat` | new axis / existing axis | batch of samples / prepend CLS token |\n| `.argmax(dim=1)` | index of max | logits β†’ predicted class |\n| `.item()` | tensor β†’ Python number | accumulate a loss |\n| `.detach().cpu().numpy()` | leave the graph, go to NumPy | plotting |" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” shape surgery cheat-run\nimport torch\n\nx = torch.randn(8, 3, 32, 32) # batch of 8 RGB 32x32 images\nprint(\"batch :\", tuple(x.shape))\nprint(\"flatten for MLP :\", tuple(x.view(x.shape[0], -1).shape)) # (8, 3072)\nprint(\"one image :\", tuple(x[0].shape)) # (3, 32, 32)\nprint(\"channels-last (plot) :\", tuple(x[0].permute(1, 2, 0).shape)) # (32, 32, 3)\nprint(\"add batch dim :\", tuple(x[0].unsqueeze(0).shape)) # (1, 3, 32, 32)\n\ngray = torch.randn(1, 28, 28)\nprint(\"squeeze grayscale :\", tuple(gray.squeeze().shape)) # (28, 28)\n\n# unfold = the ViT patch trick\nimg = torch.randn(3, 112, 112)\npatches = img.unfold(1, 14, 14).unfold(2, 14, 14) # (3, 8, 8, 14, 14)\npatches = patches.permute(1, 2, 0, 3, 4).reshape(-1, 3, 14, 14)\nprint(\"patches :\", tuple(patches.shape), \"-> 8*8 = 64 tokens\")\n\nlogits = torch.randn(4, 10)\nprint(\"predicted classes :\", logits.argmax(dim=1).tolist())\nprint(\"torch.max variant :\", torch.max(logits, 1).indices.tolist())" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "πŸ“Œ **Exam pattern.** Two shape facts are asked verbatim: *\"why do we reshape with `data.view(-1, 784)`?\"* (because `nn.Linear` expects 1-D vectors, and $28\\times28=784$) and *\"what shape does this image have?\"* (`[3, 112, 112]` β€” 3 channels because RGB, 112 because that is the resize target)." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 4. Classical ML & optimisation\n\n[↑ TOC](#toc)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 4.1 Logistic regression\n\nA linear model squashed through a sigmoid: it outputs a probability, and its **decision boundary is a straight line**.\n\nFor a 2-feature model, the boundary is where $f(x) = w_1 x_1 + w_2 x_2 + b = 0$. Solving for $x_2$:\n\n$$x_2 = \\frac{-w_1}{w_2}x_1 + \\frac{-b}{w_2}\n\\qquad\\Longrightarrow\\qquad\nm = \\frac{-w_1}{w_2},\\quad q = \\frac{-b}{w_2}$$\n\n- $f(x) > 0$ β†’ one class (above the line)\n- $f(x) < 0$ β†’ the other class\n- $f(x) = 0$ β†’ the boundary itself\n\nIn scikit-learn the parameters live in `model.coef_` (weights) and `model.intercept_` (bias). Both are **lists**, because scikit-learn models are built for multi-class; for binary problems take element `[0]`." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” logistic regression on Iris + decision boundary\nimport numpy as np, seaborn as sns, matplotlib.pyplot as plt\nfrom sklearn.linear_model import LogisticRegression\n\niris = sns.load_dataset(\"iris\")\nX = iris[[\"sepal_length\", \"petal_length\"]].values\ny = iris[\"species\"].astype(\"category\").cat.codes.values\ny = (y != 0).astype(int) # setosa (0) vs the rest (1)\nprint(\"shapes:\", X.shape, y.shape)\n\nlog_reg = LogisticRegression().fit(X, y)\n\n\ndef wb2mq(w, b):\n \"\"\"Weights+bias -> slope, intercept of the decision boundary (2D only).\"\"\"\n assert len(w) == 2, \"Only works in 2D\"\n m = -w[0] / w[1] if w[1] != 0 else float(\"inf\")\n q = -b / w[1] if w[1] != 0 else (float(\"inf\") if w[0] != 0 else 0)\n return m, q\n\n\ndef params2boundary(w, b, verbose=True):\n m, q = wb2mq(w, b)\n if verbose:\n print(f\"m: {m}, q: {q}\")\n return lambda x: m * x + q\n\n\ndef plot_decision_boundary(w, b, X, y, x1_name, x2_name, title, label=None, ax=None):\n boundary = params2boundary(w, b, verbose=False)\n x_vals = np.linspace(X[:, 0].min() - 1, X[:, 0].max() + 1, 100)\n if ax is None:\n plt.figure(figsize=(8, 6)); ax = plt.gca()\n sns.scatterplot(x=X[:, 0], y=X[:, 1], hue=y, palette=\"RdYlBu\",\n edgecolor=\"k\", s=50, ax=ax)\n ax.plot(x_vals, boundary(x_vals), \"k--\", label=label or \"Decision Boundary\")\n ax.set_xlabel(x1_name); ax.set_ylabel(x2_name); ax.set_title(title)\n ax.legend()\n return ax\n\n\nplot_decision_boundary(log_reg.coef_[0], log_reg.intercept_[0], X, y,\n \"Sepal Length\", \"Petal Length\",\n \"Logistic Regression Decision Boundary\")\nplt.show()" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 4.2 The Perceptron\n\n### Activation of a single neuron β€” the most-repeated exam question\n\n$$z = \\sum_{i=1}^{n} w_i x_i + b, \\qquad \\text{output} = \\sigma(z)$$\n\n**ReLU:** $\\sigma(z) = \\max(0, z)$\n**Leaky ReLU:** $\\sigma(z) = z$ if $z>0$, else $\\alpha z$ (typically $\\alpha = 0.01$)\n\nWorked example (Midterm 2.4): $x=[2.0,-2.0,1.0]$, $w=[1.0,-1.0,1.0]$, $b=-2$, LeakyReLU with $\\alpha=0.01$:\n\n$$\\begin{aligned}\nz &= (1\\cdot 2) + (-1\\cdot -2) + (1\\cdot 1) - 2 \\\\\n &= 2 + 2 + 1 - 2 = 3.0\n\\end{aligned}$$\n\n$$\\sigma(3.0) = 3.0$$\n\n### The learning rule\n\nFor each sample, predict, then nudge the weights towards the truth:\n\n$$\\Delta = \\eta\\,(y - \\hat y), \\qquad \\mathbf{w} \\mathrel{+}= \\Delta\\,\\mathbf{x}, \\qquad b \\mathrel{+}= \\Delta$$\n\nIf the prediction is right, $\\Delta = 0$ and nothing moves.\n\n### Limitations (theory-quiz material)\n\n- βœ… It can only reliably classify **linearly separable** data.\n- βœ… It **converges** if the data is linearly separable β€” and stops updating once no example is misclassified.\n- ❌ It does **not** always find the *best* hyperplane; it stops at the first one that works.\n- ❌ It does not \"approximate any function given enough data\" β€” that is the Universal Approximation Theorem, and it needs a hidden layer." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” perceptron activation calculator (exam style)\ndef weighted_sum(x, w, b):\n return sum(wi * xi for wi, xi in zip(w, x)) + b\n\ndef relu(z):\n return max(0.0, z)\n\ndef leaky_relu(z, alpha=0.01):\n return z if z > 0 else alpha * z\n\ncases = [\n (\"Midterm 2026 (LeakyReLU)\", [2.0, -2.0, 1.0], [1.0, -1.0, 1.0], -2, leaky_relu),\n (\"Exam AS25 2.3 (ReLU)\", [1.5, -2.0, 3.0], [0.7, 1.2, -0.8], -0.3, relu),\n (\"Exam SS25 2.3 (ReLU)\", [2.0, -1.5, 0.5], [0.8, -0.5, 1.0], 0.2, relu),\n]\nfor name, x, w, b, act in cases:\n z = weighted_sum(x, w, b)\n terms = \" + \".join(f\"({wi}*{xi})\" for wi, xi in zip(w, x))\n print(f\"{name}\\n z = {terms} + ({b}) = {z:.2f}\\n sigma({z:.2f}) = {act(z):.2f}\\n\")" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” MyPerceptron from scratch, compared to scikit-learn\nimport numpy as np, seaborn as sns, matplotlib.pyplot as plt\nfrom sklearn.datasets import load_iris\nfrom sklearn.linear_model import Perceptron\n\n\nclass MyPerceptron:\n \"\"\"Bias is stored LAST in the weight array (exam requirement).\"\"\"\n\n def __init__(self, dim, learning_rate=0.01, epochs=100):\n self.weights = np.zeros(dim + 1) # dim weights + 1 bias\n self.learning_rate = learning_rate\n self.epochs = epochs\n\n def predict(self, x):\n z = np.dot(x, self.weights[:-1]) + self.weights[-1]\n return 1 if z >= 0 else 0\n\n def train(self, X, y):\n for _ in range(self.epochs):\n for i in range(X.shape[0]):\n prediction = self.predict(X[i])\n delta = self.learning_rate * (y[i] - prediction)\n self.weights[:-1] += delta * X[i]\n self.weights[-1] += delta\n\n def score(self, X, y):\n predictions = np.array([self.predict(x) for x in X])\n return np.mean(predictions == y)\n\n\niris = load_iris()\nX = iris.data[:, [0, 2]] # sepal length, petal length\ny = (iris.target != 0).astype(int) # setosa vs rest\n\nmine = MyPerceptron(dim=2, learning_rate=0.1, epochs=100)\nmine.train(X, y)\nprint(\"Trained Weights:\", mine.weights)\nprint(f\"Accuracy: {mine.score(X, y) * 100:.2f}%\")\n\nskl = Perceptron().fit(X, y)\n*my_w, my_b = mine.weights\nw_skl, b_skl = skl.coef_[0], skl.intercept_[0]\n\nx_vals = np.array([X[:, 0].min(), X[:, 0].max()])\nsns.scatterplot(x=X[:, 0], y=X[:, 1], hue=y, palette=\"coolwarm\", edgecolor=\"k\")\nplt.plot(x_vals, (-my_w[0] / my_w[1]) * x_vals - my_b / my_w[1],\n \"k--\", label=\"Implemented Perceptron\")\nplt.plot(x_vals, (-w_skl[0] / w_skl[1]) * x_vals - b_skl / w_skl[1],\n \"--\", color=\"darkgreen\", label=\"SKL Perceptron\")\nplt.xlabel(\"Sepal Length\"); plt.ylabel(\"Petal Length\")\nplt.title(\"Perceptron Decision Boundary (Setosa vs Rest)\")\nplt.legend(); plt.show()" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” learning rate & initialisation effects\nimport numpy as np, seaborn as sns, matplotlib.pyplot as plt\nfrom sklearn.linear_model import Perceptron\nfrom sklearn.datasets import make_circles\n\n# Non-linearly separable data: the perceptron cannot solve it\nX, y = make_circles(n_samples=100, noise=0.1, factor=0.4, random_state=42)\n\nsns.scatterplot(x=X[:, 0], y=X[:, 1], hue=y, palette=\"coolwarm\", edgecolor=\"k\")\nfor state in range(3):\n p = Perceptron(random_state=state).fit(X, y)\n w, b = p.coef_[0], p.intercept_[0]\n xs = np.linspace(X[:, 0].min() - 1, X[:, 0].max() + 1, 100)\n plt.plot(xs, (-w[0] / w[1]) * xs - b / w[1], \"--\", label=f\"Random State {state}\")\nplt.title(\"Perceptron Decision Boundaries for Different Random States\")\nplt.legend(); plt.show()" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "**What to write about these two experiments**\n\n*Learning rate.* A very small rate (0.001) makes tiny updates β€” after 10 iterations the boundary has barely moved. A moderate rate (0.01) converges efficiently and stops at the first boundary that separates the data. A large rate (1.0) makes dramatic jumps before settling. **Small = slow, large = unstable, moderate = balanced.**\n\n*Initialisation on non-separable data.* The perceptron never converges on the circles dataset because a straight line cannot separate concentric rings. Each random state therefore ends somewhere completely different β€” when no good solution exists, small changes in the starting weights cause large changes in the result.\n\nπŸ“Œ **Exam pattern.** *\"What are the limitations of a single-layer perceptron?\"* appears in both 2025 finals. The always-true answers: **A** (only linearly separable data) and **D** (stops updating when no examples are misclassified) / **A** + convergence guarantee." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 4.3 Hill climbing, local minima & `scipy.optimize`\n\nHill climbing takes a random step; if the objective improves it keeps it, otherwise it discards it. It is **greedy and local** β€” it stops at the first optimum it reaches, which is almost never the global one.\n\nThe lesson the course draws: **where you start decides what you find.** The standard fix is many random restarts.\n\n`scipy.optimize.fmin` (downhill simplex / Nelder–Mead) behaves the same way on a multi-modal function:\n\n| Start | Result |\n|---|---|\n| $-225$ | a **local** minimum |\n| $-550$ | the **global** minimum πŸ₯³ |\n| $-775$, `maxiter=2` | still descending β€” stopped early, not converged |" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” hill climbing and random restarts\nimport numpy as np, matplotlib.pyplot as plt\nfrom scipy import optimize\n\n# --- naive hill climbing (maximisation) -------------------------------------\ndef f_hill(x):\n return np.sin(6 * x) + x**2 * np.cos(x**2)\n\nrng = np.random.default_rng(0)\nx = rng.uniform(-1.75, 1.75)\nsteps = [x]\nfor _ in range(15):\n new_x = x + rng.uniform(-0.3, 0.3)\n if f_hill(new_x) > f_hill(x): # accept only improvements\n x = new_x\n steps.append(x)\n\nxs = np.linspace(-2, 2, 400)\nplt.plot(xs, f_hill(xs), label=\"f(x)\")\nplt.plot(steps, [f_hill(s) for s in steps], \"ro-\", label=\"hill-climbing steps\")\nplt.title(\"Hill climbing accepts only uphill moves\"); plt.legend(); plt.show()\n\n# --- scipy downhill simplex on a multi-modal function -----------------------\ndef f(x):\n return (x % 521) * np.sin(x / 47) * np.exp(-0.002 * x)\n\nx_vals = np.linspace(-1000, 0, 1001)\ny_vals = f(x_vals)\n\nfor start, kwargs in [(-225, {}), (-550, {}), (-775, {\"maxiter\": 2})]:\n m = optimize.fmin(f, x0=start, disp=False, full_output=True, **kwargs)\n print(f\"start {start:>5} -> x={m[0][0]:8.2f} f={m[1]:8.2f}\")\n\n# --- how often do random restarts find the global minimum? ------------------\nglobal_min = int(min(y_vals))\nn_trials, found = 100, 0\nfor _ in range(n_trials):\n m = optimize.fmin(f, x0=np.random.randint(-1000, 0), disp=False, full_output=True)\n if int(m[1]) == global_min:\n found += 1\nprint(f\"After {n_trials} random initializations, the algorithm found the global minimum {found} times\")" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "πŸ“Œ **Exam pattern.** The mock exam shows a plotted curve and asks where hill climbing stops from $x=1$, $x=2.5$, $x=4$ β€” and whether each stop is a local or global optimum. Read the curve, follow the slope uphill from each start, name the nearest peak. Answer format: *\"stops at approximately $x=0.5$, a local maximum\"*." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 4.4 Unsupervised learning: K-Means & PCA\n\n**Unsupervised** = no labels. Grouping customers by purchase behaviour is unsupervised; spam classification, animal-image classification and house-price prediction are all **supervised**.\n\n### Choosing $k$ for K-Means\n\n| Method | What you plot | What you look for |\n|---|---|---|\n| **Elbow** | inertia (sum of squared distances) vs $k$, for $k=1\\ldots10$ | the bend where extra clusters stop helping |\n| **Silhouette** | silhouette score vs $k$, for $k=2\\ldots10$ | the maximum |\n\nSilhouette needs at least 2 clusters, so its range starts at 2 while the elbow starts at 1.\n\n### PCA\n\nPCA finds the **directions of maximum variance** and projects onto them. It is used here for two things: reducing to 2D so clusters can be plotted, and (in [Β§14](#s14)) turning DINOv2 features into a segmentation mask.\n\nTheory statements that are **true**: *PCA finds directions of maximum variance*; *both PCA and autoencoders can reduce dimensionality*.\n**False**: *autoencoders require labeled data* (they don't β€” they reconstruct their own input); *the latent dimension must be a multiple of two* (it can be anything, e.g. 2, 3 or 32)." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” K-Means: elbow, silhouette, PCA visualisation\nimport numpy as np, matplotlib.pyplot as plt\nfrom sklearn.cluster import KMeans\nfrom sklearn.decomposition import PCA\nfrom sklearn.metrics import silhouette_score\nfrom sklearn.datasets import make_blobs\n\nnp.random.seed(42)\n# In the exam you load it: data = pd.read_csv('blobs_data.csv'); X = data.values\nX, _ = make_blobs(n_samples=200, n_features=4, centers=4, random_state=42)\n\n# 1) Elbow method (k = 1..10)\ninertia = []\nfor k in range(1, 11):\n inertia.append(KMeans(n_clusters=k, random_state=42, n_init=10).fit(X).inertia_)\n\nplt.figure(figsize=(8, 5))\nplt.plot(range(1, 11), inertia, marker=\"o\")\nplt.title(\"Elbow Method: Inertia vs. Number of Clusters\")\nplt.xlabel(\"Number of Clusters\"); plt.ylabel(\"Inertia (Sum of Squared Distances)\")\nplt.show()\n\n# 2) Silhouette score (k = 2..10)\nsil_scores = []\nfor k in range(2, 11):\n km = KMeans(n_clusters=k, random_state=42, n_init=10).fit(X)\n sil_scores.append(silhouette_score(X, km.labels_))\n\nplt.figure(figsize=(8, 5))\nplt.plot(range(2, 11), sil_scores, marker=\"o\", color=\"orange\")\nplt.title(\"Silhouette Score vs. Number of Clusters\")\nplt.xlabel(\"Number of Clusters\"); plt.ylabel(\"Silhouette Score\")\nplt.show()\n\n# 3) Final clustering, visualised in 2D via PCA\noptimal_k = int(np.argmax(sil_scores)) + 2\nprint(\"optimal k =\", optimal_k)\nkmeans = KMeans(n_clusters=optimal_k, random_state=42, n_init=10).fit(X)\n\npca = PCA(n_components=2)\nX_pca = pca.fit_transform(X)\ncentroids_pca = pca.transform(kmeans.cluster_centers_)\n\nplt.scatter(X_pca[:, 0], X_pca[:, 1], c=kmeans.labels_, cmap=\"viridis\", s=50, alpha=0.5)\nplt.scatter(centroids_pca[:, 0], centroids_pca[:, 1], c=\"darkred\", s=200,\n marker=\"X\", label=\"Centroids\")\nplt.title(\"K-Means Clustering with PCA (Centroids in Red)\")\nplt.xlabel(\"PCA Component 1\"); plt.ylabel(\"PCA Component 2\")\nplt.legend(); plt.show()" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 4.5 Decision trees (by hand)\n\nBoth 2025 finals open the Hands-On block with a decision-tree traversal worth 4 points: a tree is given as a picture, a table of items follows, and you fill in *correctly classified yes/no* and *misclassified as*.\n\n**Method β€” mechanical, no cleverness required:**\n\n1. Start at the root. Read the condition and the item's feature value.\n2. Follow the matching branch. If no branch matches, take the *Else* branch.\n3. Repeat until a leaf. That leaf is the assigned class.\n4. Compare with the true class. If they differ, write the leaf's class in the \"misclassified as\" column.\n\nWorked example on the mock-exam tree:\n\n- **Root:** `X1 = 1` β†’ Node 2 Β· `X1 = 2` β†’ Node 4 Β· else β†’ Class B\n- **Node 2:** `X2 = 1` β†’ Class A Β· else β†’ Node 3\n- **Node 3:** `X3 = 1` β†’ Class A Β· else β†’ Class B\n- **Node 4:** `X3 = 1` β†’ Class B Β· `X3 = 2` β†’ Class A Β· else β†’ Class C\n\n| Input | $X_1$ | $X_2$ | $X_3$ | Path | Class |\n|---|---|---|---|---|---|\n| 1 | 1 | 2 | 1 | Root β†’ Node 2 (X2β‰ 1) β†’ Node 3 (X3=1) | **A** |\n| 2 | 2 | 1 | 0 | Root β†’ Node 4 (X3βˆ‰{1,2}) | **C** |\n| 3 | 3 | 1 | 2 | Root (X1βˆ‰{1,2}) | **B** |\n| 4 | 1 | 1 | 3 | Root β†’ Node 2 (X2=1) | **A** |\n| 5 | 2 | 2 | 2 | Root β†’ Node 4 (X3=2) | **A** |" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” traverse the mock-exam decision tree programmatically\ndef classify(x1, x2, x3):\n if x1 == 1: # Node 2\n if x2 == 1:\n return \"A\"\n return \"A\" if x3 == 1 else \"B\" # Node 3\n if x1 == 2: # Node 4\n if x3 == 1:\n return \"B\"\n if x3 == 2:\n return \"A\"\n return \"C\"\n return \"B\" # Else at the root\n\nfor i, inp in enumerate([(1, 2, 1), (2, 1, 0), (3, 1, 2), (1, 1, 3), (2, 2, 2)], start=1):\n print(f\"Input {i}: X1={inp[0]}, X2={inp[1]}, X3={inp[2]} -> Class {classify(*inp)}\")" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "πŸ“Œ **Exam pattern.** The 2025 finals used a **planets** tree (distance / diameter / moons) and a **fruits** tree (weight / sugar / water). The traps are always the same two: a value that matches *no* branch (take *Else*), and a threshold that is `>` rather than `>=`. Write the path, not just the class β€” it protects your partial credit." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 5. The data pipeline\n\n[↑ TOC](#toc)\n\nEvery coding question in this course starts here. Get the pipeline right and the model is the easy part." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 5.1 Tabular preprocessing\n\n**One-hot encoding, not `LabelEncoder`.** Categorical variables like `Sex` have no natural order. `LabelEncoder` would map maleβ†’0, femaleβ†’1, and the model would read that as *female > male*. One-hot gives each category its own binary column, so they stay distinct groups rather than ranked values.\n\nThe exception: use ordinal encoding when the categories *do* have an order (T-shirt sizes, marathon placement). Never one-hot a regression target.\n\n**Standardisation.** `Age` and `Fare` live on wildly different scales. Without scaling, the model over-weights the feature with the bigger numbers purely because the numbers are bigger. `StandardScaler` maps each feature to mean 0, std 1.\n\n⚠️ `input_dim` must equal the number of columns **after** one-hot encoding β€” that is why the Titanic MLP takes 14, not 8." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# reference β€” the Titanic preprocessing block (needs 05_titanic_clean.csv)\nimport pandas as pd, numpy as np\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import train_test_split\n\nt_df = pd.read_csv(\"05_titanic_clean.csv\")\nt_df.drop([\"Cabin\", \"Name\", \"PassengerId\", \"Embarked\", \"Ticket\"], axis=1, inplace=True)\ndisplay(t_df.describe())\n\n# categorical -> one-hot (each category gets its own binary column)\nt_df = pd.get_dummies(t_df, columns=[\"Sex\"], prefix=[\"Sex\"])\nt_df = pd.get_dummies(t_df, columns=[\"Title\"])\n\n# numerical -> mean 0, std 1\nscaler = StandardScaler()\nt_df[[\"Age\", \"Fare\"]] = scaler.fit_transform(t_df[[\"Age\", \"Fare\"]])\n\n# features / target; float32 is the standard dtype for PyTorch inputs\nX = t_df.drop([\"Survived\"], axis=1).astype(np.float32).values\ny = t_df[\"Survived\"].values\nprint(\"input_dim must be\", X.shape[1])" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 5.2 Datasets & DataLoaders\n\nFour ways to get data into a `DataLoader`, all used in the course:\n\n| Source | Class | Used in |\n|---|---|---|\n| Tensors already in memory | `TensorDataset(X, y)` | Titanic, midterm |\n| Built-in benchmark | `datasets.MNIST / FashionMNIST / CIFAR10 / USPS` | most assignments |\n| Folder of images, one subfolder per class | `datasets.ImageFolder(root)` | dogs vs cats |\n| Your own format | subclass `Dataset` with `__len__` + `__getitem__` | lyrics, point clouds |\n\n**`DataLoader` rules:** `shuffle=True` for training (so the model doesn't learn the order of the data and gradients stay less correlated), `shuffle=False` for validation and test (so evaluation is consistent and reproducible).\n\n**Transforms are applied on access**, not once at load time. That is why an augmented dataset shows a slightly different version of the same image every epoch." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” the four dataset flavours (only the first two run offline)\nimport torch\nfrom torch.utils.data import TensorDataset, DataLoader, Dataset, Subset\n\n# 1) TensorDataset --------------------------------------------------------\nX_train = torch.randn(100, 14); y_train = torch.randint(0, 2, (100,))\nX_val = torch.randn(40, 14); y_val = torch.randint(0, 2, (40,))\n\ntrain_dataset = TensorDataset(X_train, y_train)\nval_dataset = TensorDataset(X_val, y_val) # ⚠ NOT (X_train, y_train)!\n\nbatch_size = 32\ntrain_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\nval_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False)\nprint(\"batches:\", len(train_loader), \"| samples:\", len(train_loader.dataset))\n\n# 2) Custom Dataset -------------------------------------------------------\nclass TextDataset(Dataset):\n def __init__(self, sequences):\n self.sequences = sequences\n\n def __len__(self):\n return len(self.sequences)\n\n def __getitem__(self, idx):\n return self.sequences[idx]\n\n# 3) Built-in benchmark # needs download\n# transform = transforms.Compose([transforms.ToTensor(),\n# transforms.Normalize((0.5,), (0.5,))]) # -> [-1, 1]\n# train_data = datasets.FashionMNIST(root='./data', train=True, transform=transform, download=True)\n# test_data = datasets.FashionMNIST(root='./data', train=False, transform=transform, download=True)\n# print(len(train_data), train_data.classes)\n\n# 4) ImageFolder # needs an unzipped folder DATA_PATH/train/{cats,dogs}\n# train_dataset = datasets.ImageFolder(os.path.join(DATA_PATH, 'train'))" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# reference β€” RAMDatasetWrapper: cache a whole dataset in memory\nfrom tqdm.auto import tqdm\nimport torch, PIL\n\n\nclass RAMDatasetWrapper(torch.utils.data.Dataset):\n \"\"\"Cache an entire dataset in RAM, then apply transforms on access.\n\n Reading many small files from disk becomes the bottleneck when you iterate\n over the same images for many epochs. Loading once at the start removes it.\n\n ⚠️ Only practical when the dataset fits in memory β€” otherwise it will\n exhaust RAM and crash the session.\n \"\"\"\n\n def __init__(self, dataset, transform=None):\n self.data = [sample for sample in tqdm(dataset)]\n self.n = len(self.data)\n self.transform = transform\n\n def __getitem__(self, ind):\n if self.transform is not None and isinstance(self.data[ind][0], PIL.Image.Image):\n return self.transform(self.data[ind][0]), self.data[ind][1]\n return self.data[ind]\n\n def set_transform(self, transform):\n self.transform = transform\n\n def __len__(self):\n return self.n" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 5.3 Splits & data leakage\n\n| Set | Purpose |\n|---|---|\n| **Train** | update the model parameters |\n| **Validation** | evaluate performance *during* training, tune hyperparameters, trigger early stopping |\n| **Test** | final, once, on data never seen β€” estimates generalisation |\n\n**You must never use the test set to choose the best model.** The test set is not for tuning hyperparameters.\n\nWhen no hyperparameter tuning happens, the course drops the validation set and uses train/test only β€” but says so explicitly.\n\n### k-fold cross-validation\n\nUse it when you want a **more reliable estimate** of performance, especially on small datasets, and when you want every data point to serve as both training and validation to reduce evaluation instability. It **does not** save computation, and it **does not** remove the need for a separate test set.\n\n### Data leakage β€” the planted midterm bug\n\n```python\ntrain_dataset = TensorDataset(X_train, y_train)\nval_dataset = TensorDataset(X_train, y_train) # ❌ validation IS the training set\n```\n\nSymptom: validation accuracy far above what chance allows. For 10 classes of pure random data, chance is **~10 %** β€” anything near 100 % means the model memorised data it is being tested on.\n\nFix: `val_dataset = TensorDataset(X_val, y_val)`. After the fix the accuracy drops to roughly chance, which is the *correct* result for random labels." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” reproducing the leakage bug and its fix\nimport torch\nfrom torch.utils.data import TensorDataset\n\nn_train = n_val = 100\nX_train = torch.randn(n_train, 1, 28, 28); y_train = torch.randint(0, 10, (n_train,))\nX_val = torch.randn(n_val, 1, 28, 28); y_val = torch.randint(0, 10, (n_val,))\n\nleaky = TensorDataset(X_train, y_train) # ❌ what the exam gives you\nfixed = TensorDataset(X_val, y_val) # βœ… data leakage fixed!\n\nprint(\"chance accuracy for 10 classes: ~10%\")\nprint(\"leaky val set is the train set:\", torch.equal(leaky.tensors[0], X_train))\nprint(\"fixed val set is independent :\", not torch.equal(fixed.tensors[0], X_train))" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "πŸ“Œ **Exam pattern.** Midterm 3.3 (6 pts) walks you through the diagnosis in five one-word answers: chance level (~10 %), overfitting or underfitting (**overfitting**), what is suspicious (**data leakage β€” the validation set was built from the training data**), the fix, the retrain, and *\"are the new results reasonable?\"* (**yes, as expected**). The mock exam asks the same thing about a colleague's suspiciously perfect history dict: *100 % validation accuracy from epoch 1 indicates overfitting or data leakage.*" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 6. MLPs & the training loop\n\n[↑ TOC](#toc)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 6.1 Three ways to define a model\n\nThe **Universal Function Approximation theorem** says a sufficiently large single hidden layer can approximate any continuous function. It does **not** guarantee that learning will be efficient or practical, and it does **not** guarantee generalisation. Non-linear activations are required for it to hold β€” without them, stacked linear layers collapse into one linear layer.\n\nAn MLP is an input layer, one or more hidden layers, and an output layer, **fully connected**: every neuron connects to every neuron in the next layer." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” the three model-definition styles, all equivalent\nimport torch, torch.nn as nn, torch.nn.functional as F\n\n# (a) bare nn.Sequential β€” quickest, used in Assignment 4\nmodel_a = nn.Sequential(\n nn.Linear(28 * 28, 128),\n nn.ReLU(),\n nn.Linear(128, 10),\n)\n\n# (b) nn.Module wrapping one Sequential β€” the exam's preferred style\nclass MLP(nn.Module):\n def __init__(self):\n super().__init__()\n self.model = nn.Sequential(\n nn.Flatten(), # (B,1,28,28) -> (B,784), no manual .view needed\n nn.Linear(28 * 28, 128), nn.ReLU(),\n nn.Linear(128, 64), nn.ReLU(),\n nn.Linear(64, 10),\n )\n\n def forward(self, x):\n return self.model(x)\n\n# (c) explicit layers + functional forward β€” when you need branching\nclass MLPNoSeq(nn.Module):\n def __init__(self, input_dim, hidden_dim, output_dim):\n super().__init__()\n self.fc1 = nn.Linear(input_dim, hidden_dim)\n self.fc2 = nn.Linear(hidden_dim, hidden_dim)\n self.out = nn.Linear(hidden_dim, output_dim)\n\n def forward(self, x):\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n return self.out(x)\n\nmodel = MLP()\nprint(model)\nprint(\"\\nforward on a batch:\", model(torch.randn(4, 1, 28, 28)).shape)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "**Every `nn.Module` needs exactly three things:** `super().__init__()`, layers defined in `__init__`, and a `forward(self, x)` that returns the output. Forgetting `super().__init__()` or returning the wrong variable from `forward` are two of the six planted exam bugs.\n\n**Sizing rules**\n\n- Input layer = number of features. Flattened $28\\times28$ image β†’ **784**. Titanic after one-hot β†’ **14**. USPS $16\\times16$ β†’ **256**.\n- Output layer = number of classes. Binary classification β†’ **2** outputs with `CrossEntropyLoss` (or 1 with `BCEWithLogitsLoss`). MNIST β†’ **10**.\n- `nn.LazyLinear(out)` infers `in_features` on the first forward pass β€” very useful after `nn.Flatten()` when you don't want to compute the flattened size by hand." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 6.2 Activation functions\n\n| Name | Definition | Notes |\n|---|---|---|\n| **ReLU** | $\\max(0, z)$ | default; outputs 0 for **all** negative inputs; range $[0,\\infty)$ β€” *not* $[0,1]$ |\n| **Leaky ReLU** | $z$ if $z>0$ else $\\alpha z$ | small non-zero gradient for negatives; **not** \"ReLU shifted upward\" |\n| **GELU** | $z\\,\\Phi(z)$ | used inside transformer blocks |\n| **Sigmoid** | $1/(1+e^{-z})$ | squashes to $(0,1)$; VAE decoder output |\n| **Tanh** | $\\tanh(z)$ | squashes to $(-1,1)$; RNN hidden state |\n| **Softmax** | $e^{z_i}/\\sum_j e^{z_j}$ | turns scores into a probability distribution |\n\n⚠️ **MSE is a loss, not an activation.** That distractor appears in the mock exam." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” the activation zoo, plotted\nimport torch, torch.nn as nn, matplotlib.pyplot as plt\n\nz = torch.linspace(-4, 4, 400)\nacts = {\n \"ReLU\": nn.ReLU(), \"LeakyReLU(0.1)\": nn.LeakyReLU(0.1), \"GELU\": nn.GELU(),\n \"Sigmoid\": nn.Sigmoid(), \"Tanh\": nn.Tanh(),\n}\nplt.figure(figsize=(9, 5))\nfor name, fn in acts.items():\n plt.plot(z, fn(z), label=name)\nplt.axhline(0, color=\"k\", lw=0.5); plt.axvline(0, color=\"k\", lw=0.5)\nplt.legend(); plt.title(\"Activation functions\"); plt.show()\n\nprint(\"softmax([2,1,0.1]) =\", torch.softmax(torch.tensor([2.0, 1.0, 0.1]), dim=0))" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 6.3 The canonical train / evaluate / plot helpers\n\nThese four functions (adapted in the course from *Deep Learning*, Prof. Paolo Favaro, University of Bern) appear **unchanged** in Assignments 5, 6, the midterm, the mock exam and both finals. Learn their signatures β€” the exam hands them to you as *DO NOT EDIT* and asks you to call them correctly." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# reference β€” memorise these four. They are given in every exam.\nimport numpy as np, torch, matplotlib.pyplot as plt\nfrom tqdm.auto import tqdm\n\n\ndef train_epoch(model, train_dataloader, optimizer, loss_fn):\n losses, correct_predictions = [], 0\n for features, labels in tqdm(train_dataloader):\n features, labels = features.to(device), labels.to(device)\n output = model(features) # 1. forward\n optimizer.zero_grad() # 2. clear old gradients\n loss = loss_fn(output, labels) # 3. loss\n loss.backward() # 4. backprop\n optimizer.step() # 5. update weights\n losses.append(loss.item())\n correct_predictions += (output.argmax(dim=1) == labels).sum().item()\n accuracy = 100.0 * correct_predictions / len(train_dataloader.dataset)\n return np.array(losses).mean(), accuracy\n\n\ndef evaluate(model, dataloader, loss_fn):\n losses, correct_predictions = [], 0\n with torch.no_grad(): # no gradients -> less memory, faster\n for features, labels in dataloader:\n features, labels = features.to(device), labels.to(device)\n output = model(features)\n loss = loss_fn(output, labels)\n correct_predictions += (output.argmax(dim=1) == labels).sum().item()\n losses.append(loss.item())\n accuracy = 100.0 * correct_predictions / len(dataloader.dataset)\n return np.array(losses).mean(), accuracy\n\n\ndef train(model, train_dataloader, val_dataloader, optimizer, n_epochs, loss_fn):\n train_losses, val_losses, train_accuracies, val_accuracies = [], [], [], []\n for epoch in range(n_epochs):\n model.train() # enables Dropout / BatchNorm training mode\n train_loss, train_accuracy = train_epoch(model, train_dataloader, optimizer, loss_fn)\n model.eval() # disables them for a deterministic pass\n val_loss, val_accuracy = evaluate(model, val_dataloader, loss_fn)\n train_losses.append(train_loss); val_losses.append(val_loss)\n train_accuracies.append(train_accuracy); val_accuracies.append(val_accuracy)\n print('Epoch {}/{}: train_loss: {:.4f}, train_accuracy: {:.4f}, '\n 'val_loss: {:.4f}, val_accuracy: {:.4f}'.format(\n epoch + 1, n_epochs, train_losses[-1], train_accuracies[-1],\n val_losses[-1], val_accuracies[-1]))\n return train_losses, val_losses, train_accuracies, val_accuracies\n\n\ndef plot(train_losses, val_losses, train_accuracies, val_accuracies, title):\n plt.figure()\n plt.plot(np.arange(len(train_losses)), train_losses)\n plt.plot(np.arange(len(val_losses)), val_losses)\n plt.legend(['train_loss', 'val_loss'])\n plt.xlabel('epoch'); plt.ylabel('loss value')\n plt.xticks(np.arange(len(train_losses)), np.arange(1, len(train_losses) + 1))\n plt.title('{}: Train/val loss'.format(title))\n\n plt.figure()\n plt.plot(np.arange(len(train_accuracies)), train_accuracies)\n plt.plot(np.arange(len(val_accuracies)), val_accuracies)\n plt.legend(['train_acc', 'val_acc'])\n plt.xlabel('epoch'); plt.ylabel('accuracy')\n plt.xticks(np.arange(len(train_losses)), np.arange(1, len(train_losses) + 1))\n plt.title('{}: Train/val accuracy'.format(title))" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "### The training loop, ordered\n\nThe order of these operations matters. Incorrect ordering prevents the model from learning:\n\n1. Flatten / move the batch to `device`\n2. `optimizer.zero_grad()` β€” gradients accumulate by default\n3. **Forward pass:** `output = model(x)`\n4. **Loss:** `loss = criterion(output, labels)`\n5. `loss.backward()` β€” compute gradients\n6. `optimizer.step()` β€” update weights\n7. `total_loss += loss.item()` β€” `.item()` detaches, so you don't keep the graph alive\n\nThe three lines people mix up: `zero_grad` must come **before** `backward`, and `step` must come **after** it." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” a complete miniature training run on synthetic data\nimport torch, torch.nn as nn, torch.optim as optim, numpy as np\nfrom torch.utils.data import TensorDataset, DataLoader\n\nset_seed(42)\n\nX_train, y_train = torch.randn(512, 20), torch.randint(0, 3, (512,))\nX_val, y_val = torch.randn(128, 20), torch.randint(0, 3, (128,))\ntrain_loader = DataLoader(TensorDataset(X_train, y_train), batch_size=32, shuffle=True)\nval_loader = DataLoader(TensorDataset(X_val, y_val), batch_size=32, shuffle=False)\n\nmodel = nn.Sequential(nn.Linear(20, 32), nn.ReLU(), nn.Linear(32, 3)).to(device)\n\n# --- the four lines that define a training setup ---------------------------\ncriterion = nn.CrossEntropyLoss()\noptimizer = optim.Adam(model.parameters(), lr=0.001)\nn_epochs = 5\n\nhist = train(model, train_loader, val_loader, optimizer, n_epochs, criterion)\nplot(*hist, title=\"MLP\")\n\nval_loss, val_accuracy = evaluate(model, val_loader, criterion)\nprint('MLP. Validation loss: {:.2f}, validation accuracy: {:.2f}'.format(val_loss, val_accuracy))" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 6.4 Hyperparameters & what they do\n\n| Hyperparameter | Typical value | Effect |\n|---|---|---|\n| **Learning rate** | `0.001` (Adam), `0.01` (SGD) | how far each weight moves per step. Too small = slow; too large = overshoots the minimum. Purpose: balance fast convergence against stable learning. |\n| **Batch size** | 32, 64, 128 | how many samples before one weight update |\n| **Epochs** | 5–100 | full passes over the training set |\n| **Hidden dim** | 32, 64, 128, 256 | model capacity |\n| **Optimizer** | `Adam` (default), `SGD`, `AdamW` | Adam adapts the step size per parameter |\n| **weight_decay** | `1e-4` | L2 regularisation, passed to the optimizer |\n\n⚠️ Reinitialise **both** the model *and* the optimizer whenever you change the architecture β€” the optimizer holds references to the old parameters.\n\nπŸ“Œ **Exam pattern.** *\"Set the training parameters\"* is worth 2 points and is always the same three lines:\n\n```python\ncriterion = nn.CrossEntropyLoss()\noptimizer = optim.Adam(model.parameters(), lr=0.001)\nn_epochs = 10\n```" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 7. Regularisation & generalisation\n\n[↑ TOC](#toc)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 7.1 BatchNorm, LayerNorm, Dropout, weight decay\n\n| Technique | What it normalises / does | Train vs inference |\n|---|---|---|\n| **BatchNorm** (`nn.BatchNorm1d/2d`) | across the **batch** dimension, per feature | **different** behaviour: uses batch statistics while training, running averages at inference |\n| **LayerNorm** (`nn.LayerNorm`) | across the **features**, per sample | **same** behaviour in both modes |\n| **Dropout** (`nn.Dropout(p)`) | randomly zeroes activations with probability $p$ | active in training, disabled by `model.eval()` |\n| **Weight decay** (`weight_decay=1e-4`) | L2 penalty on the weights, applied by the optimizer | same in both |\n\nThose four statements about BatchNorm/LayerNorm are a verbatim theory question in the SS25 exam β€” **all four are true**.\n\n### What each one buys you\n\n- **No regularisation:** the model memorises the training data. Training loss falls while validation loss **rises** β€” classic overfitting, poor generalisation.\n- **BatchNorm:** normalises activations within mini-batches, stabilising training and making it robust to small variations. Especially effective on small datasets like Titanic.\n- **LayerNorm:** reduces overfitting slightly but struggles here β€” with fixed batch sizes it cannot normalise activations across varying batches as effectively, limiting its stabilising effect on small datasets.\n- **Dropout:** randomly deactivates neurons, forcing the model to rely on a broad set of features instead of memorising specific patterns. Usually the strongest generaliser of the three in this course's experiments.\n\n`nn.Identity()` is the trick that lets one class switch any of them on or off: it passes its input through unchanged." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” one MLP class with switchable regularisation\nimport torch, torch.nn as nn, torch.optim as optim\n\n\nclass MLP(nn.Module):\n \"\"\"3 hidden layers, each optionally BatchNorm / LayerNorm / Dropout.\"\"\"\n\n def __init__(self, input_dim, hidden_dim,\n use_batchnorm=False, use_layernorm=False, use_dropout=False):\n super().__init__()\n\n def block(in_dim, out_dim):\n return [\n nn.Linear(in_dim, out_dim),\n nn.BatchNorm1d(out_dim) if use_batchnorm else nn.Identity(),\n nn.LayerNorm(out_dim) if use_layernorm else nn.Identity(),\n nn.ReLU(),\n nn.Dropout(0.5) if use_dropout else nn.Identity(),\n ]\n\n self.model = nn.Sequential(\n *block(input_dim, hidden_dim),\n *block(hidden_dim, hidden_dim),\n *block(hidden_dim, hidden_dim),\n nn.Linear(hidden_dim, 2), # binary classification -> 2 logits\n )\n\n def forward(self, x):\n return self.model(x)\n\n\nfor flags in [{}, {\"use_batchnorm\": True}, {\"use_layernorm\": True}, {\"use_dropout\": True}]:\n m = MLP(14, 32, **flags)\n name = list(flags)[0].replace(\"use_\", \"\") if flags else \"none\"\n print(f\"{name:>10}: output {tuple(m(torch.randn(8, 14)).shape)}\")\n\n# weight decay = L2 regularisation, set on the optimizer, not the model\nmodel = MLP(14, 32)\noptimizer_l2 = optim.Adam(model.parameters(), lr=0.001, weight_decay=0.0001)\nprint(\"\\nAdam with L2:\", optimizer_l2.param_groups[0][\"weight_decay\"])" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "### Comparing runs β€” the table + curve pattern\n\nThe assignment asks you to collect the final numbers into a table and overlay the validation-loss curves." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” the comparison table and overlaid loss curves (synthetic numbers)\nimport numpy as np, matplotlib.pyplot as plt\n\nval_losses = list(np.linspace(0.55, 0.72, 30)) # no regularisation: goes UP\nval_losses_bn = list(np.linspace(0.58, 0.47, 30))\nval_losses_ln = list(np.linspace(0.57, 0.55, 30))\nval_losses_drop = list(np.linspace(0.60, 0.45, 30))\nval_acc, val_acc_bn, val_acc_ln, val_acc_drop = [78.2], [81.5], [79.3], [82.1]\n\ntable_data = [\n [\"Regularization type\", \"Val Loss\", \"Val accuracy\"],\n [\"No regularization\", val_losses[-1], val_acc[-1]],\n [\"Batch Norm\", val_losses_bn[-1], val_acc_bn[-1]],\n [\"Layer Norm\", val_losses_ln[-1], val_acc_ln[-1]],\n [\"Dropout\", val_losses_drop[-1], val_acc_drop[-1]],\n]\nprint(\"{: >20}| {: >20}| {: >20}\".format(*table_data[0]))\nprint(\"-\".join(\"\" for _ in range(65)))\nfor row in table_data[1:]:\n print(\"{: >20}| {:20.4f}| {: >20}\".format(*row))\n\nloss_curves = [val_losses, val_losses_bn, val_losses_ln, val_losses_drop]\nreg_types = [\"no regularization\", \"batch norm\", \"layer norm\", \"dropout\"]\nplt.figure()\nfor curve in loss_curves:\n plt.plot(np.arange(len(curve)), curve)\nplt.title(\"Validation loss curves for different regularization\")\nplt.legend(reg_types); plt.show()" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 7.2 Reading loss curves\n\n| What you see | Diagnosis | What to write |\n|---|---|---|\n| Train loss ↓, **val loss ↑** | **Overfitting** | \"the model memorises the training data and fails to generalise\" β€” regularisation (Dropout, weight decay) could help |\n| Both losses high and flat | **Underfitting** | the model is too simple to capture the patterns |\n| Both ↓ together, converge to similar values | **Generalising well** | no clear sign of overfitting |\n| Val loss spikes early, then falls | **Instability**, not overfitting | parameters are still random and updates are aggressive; generalisation temporarily worsens before improving |\n| Both drop to ~0 and accuracy jumps to 100 % | The model found a **strongly discriminative feature** β€” the task became trivial |\n| Val accuracy = 100 % from epoch 1 | **Data leakage** β€” see [Β§5.3](#s5-3) |\n\n**Two statements that are always false:** *\"high training loss and low validation loss indicates overfitting\"* (that's backwards), and *\"zero training loss means the model generalised well\"* (it usually means the opposite)." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 7.3 Early stopping, LR scheduling, best-model checkpointing\n\nAssignment 7 introduces a production-grade `train_model` that adds four things to the basic loop:\n\n1. **Validation monitoring** β€” the loop already had it.\n2. **`ReduceLROnPlateau`** β€” halves the learning rate when validation loss stops improving for 4 epochs.\n3. **Early stopping** β€” stops after `patience` epochs with no improvement of at least `min_delta`.\n4. **Best-model saving** β€” keeps a `deepcopy` of the weights with the lowest validation loss and **restores them at the end**, so the returned model is the best one seen, not the last one." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# reference β€” EarlyStopping + the full train_model / test_model / plot_training_history suite\nimport copy, numpy as np, torch, matplotlib.pyplot as plt, seaborn as sns\nfrom tqdm.auto import tqdm\nfrom sklearn.metrics import confusion_matrix, classification_report\n\n\nclass EarlyStopping:\n \"\"\"Stops training when validation loss has not improved for `patience` epochs.\"\"\"\n\n def __init__(self, patience=10, min_delta=0.0):\n self.patience = patience\n self.min_delta = min_delta\n self.best_score = None\n self.counter = 0\n self.should_stop = False\n\n def step(self, current_score):\n if self.best_score is None: # first epoch always \"improves\"\n self.best_score, self.counter, self.should_stop = current_score, 0, False\n return True\n if current_score < self.best_score - self.min_delta: # for loss, LOWER is better\n self.best_score, self.counter, self.should_stop = current_score, 0, False\n return True\n self.counter += 1\n if self.counter >= self.patience:\n self.should_stop = True\n return False\n\n\ndef train_model(model, train_loader, val_loader, criterion, optimizer,\n num_epochs=50, name=\"model\", patience=10,\n use_early_stopping=False, verbose=True, min_delta=0.001):\n \"\"\"Train with validation monitoring, LR scheduling, early stopping and\n automatic restoration of the best weights.\n\n Returns\n -------\n model : the model with the BEST validation weights restored\n history : dict with train_loss / train_acc / val_loss / val_acc / lr\n \"\"\"\n if len(train_loader) == 0 or len(val_loader) == 0:\n raise ValueError(\"Empty DataLoader.\")\n\n best_val_loss = float(\"inf\")\n best_model_wts = copy.deepcopy(model.state_dict())\n early_stopper = EarlyStopping(patience=patience, min_delta=min_delta)\n scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(\n optimizer, mode=\"min\", factor=0.5, patience=4)\n\n history = {\"train_loss\": [], \"train_acc\": [], \"val_loss\": [], \"val_acc\": [], \"lr\": []}\n\n for epoch in tqdm(range(num_epochs), desc=\"Training Progress\", unit=\"epoch\",\n disable=not verbose):\n # ---- train ---------------------------------------------------------\n model.train()\n running_loss = correct_train = total_train = 0\n for images, labels in train_loader:\n images, labels = images.to(device), labels.to(device)\n optimizer.zero_grad()\n outputs = model(images)\n loss = criterion(outputs, labels)\n loss.backward()\n optimizer.step()\n bs = labels.size(0)\n running_loss += loss.item() * bs # weighted: batch sizes may vary\n total_train += bs\n correct_train += (outputs.argmax(1) == labels).sum().item()\n train_loss = running_loss / total_train\n train_acc = 100.0 * correct_train / total_train\n\n # ---- validate ------------------------------------------------------\n model.eval()\n val_running_loss = correct_val = total_val = 0\n with torch.no_grad():\n for images, labels in val_loader:\n images, labels = images.to(device), labels.to(device)\n outputs = model(images)\n loss = criterion(outputs, labels)\n bs = labels.size(0)\n val_running_loss += loss.item() * bs\n total_val += bs\n correct_val += (outputs.argmax(1) == labels).sum().item()\n val_loss = val_running_loss / total_val\n val_acc = 100.0 * correct_val / total_val\n\n scheduler.step(val_loss) # LR halves on plateau\n current_lr = optimizer.param_groups[0][\"lr\"]\n\n for k, v in zip(history, [train_loss, train_acc, val_loss, val_acc, current_lr]):\n history[k].append(v)\n\n if verbose:\n print(f\"\\nEpoch [{epoch+1}/{num_epochs}] | LR: {current_lr:.6f}\")\n print(f\"Train Loss: {train_loss:.4f}, Train Accuracy: {train_acc:.2f}%\")\n print(f\"Val Loss: {val_loss:.4f}, Val Accuracy: {val_acc:.2f}%\")\n\n if val_loss < best_val_loss: # ---- checkpoint -------\n best_val_loss = val_loss\n best_model_wts = copy.deepcopy(model.state_dict())\n torch.save(model.state_dict(), f\"best_{name}.pth\")\n\n if use_early_stopping: # ---- early stopping ---\n early_stopper.step(val_loss)\n if early_stopper.should_stop:\n if verbose:\n print(f\" Early stopping triggered after epoch {epoch+1}.\")\n break\n\n model.load_state_dict(best_model_wts) # restore the BEST weights\n return model, history\n\n\ndef plot_training_history(history, plot_lr=False):\n epochs = range(1, len(history[\"train_loss\"]) + 1)\n for keys, ylabel, title in [\n ((\"train_loss\", \"val_loss\"), \"Loss\", \"Training vs Validation Loss\"),\n ((\"train_acc\", \"val_acc\"), \"Accuracy (%)\", \"Training vs Validation Accuracy\"),\n ]:\n plt.figure(figsize=(8, 5))\n for k in keys:\n plt.plot(epochs, history[k], label=k)\n plt.title(title); plt.xlabel(\"Epoch\"); plt.ylabel(ylabel)\n plt.legend(); plt.grid(True); plt.tight_layout(); plt.show()\n\n if plot_lr:\n plt.figure(figsize=(8, 5))\n plt.plot(epochs, history[\"lr\"], label=\"Learning Rate\")\n plt.title(\"Learning Rate Over Epochs\"); plt.xlabel(\"Epoch\")\n plt.ylabel(\"Learning Rate\"); plt.legend(); plt.grid(True); plt.show()" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 8. Convolutional Neural Networks\n\n[↑ TOC](#toc)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 8.1 Convolution & pooling maths\n\n**Output size of a convolutional layer:**\n\n$$\\text{output size} = \\left\\lfloor \\frac{W + 2P - K}{S} \\right\\rfloor + 1$$\n\n**Output size of a max-pooling layer:**\n\n$$\\text{output size} = \\left\\lfloor \\frac{W - K}{S} \\right\\rfloor + 1$$\n\nwhere $W$ = input height/width, $K$ = kernel size, $S$ = stride, $P$ = padding.\n\nRules to remember:\n\n- Images are tensors of shape $(C, H, W)$; the channel count of the output is `out_channels`, not something you compute.\n- `ReLU()` **does not change the shape** β€” it is element-wise.\n- If no stride is given to `MaxPool2d`, PyTorch uses `stride = kernel_size`.\n- `Flatten()` turns $(C,H,W)$ into a vector of length $C \\times H \\times W$.\n- `padding=1` with a $3\\times3$ kernel and stride 1 **preserves** the spatial size; without padding the filter cannot reach the edges so the output shrinks.\n\n### Worked shape trace (Assignment 6 Β§1.3)\n\nInput $(1, 28, 28)$:\n\n| Layer | Output shape | Calculation |\n|---|---|---|\n| Input | (1, 28, 28) | |\n| Conv1 `k=3, s=1, p=0`, 8 ch | (8, 26, 26) | $\\lfloor(28+0-3)/1\\rfloor+1 = 26$ |\n| ReLU1 | (8, 26, 26) | unchanged |\n| MaxPool1 `k=2` | (8, 13, 13) | $\\lfloor(26-2)/2\\rfloor+1 = 13$ |\n| Conv2 `k=3, s=1, p=1`, 16 ch | (16, 13, 13) | $\\lfloor(13+2-3)/1\\rfloor+1 = 13$ |\n| ReLU2 | (16, 13, 13) | unchanged |\n| MaxPool2 `k=3, s=2` | (16, 6, 6) | $\\lfloor(13-3)/2\\rfloor+1 = 6$ |\n| Flatten | (576) | $16\\times6\\times6 = 576$ |\n| Linear | (10) | `out_features` |\n\nSo `nn.Linear(in_features=576, out_features=10)`." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” shape calculators + automatic verification\nimport math, torch, torch.nn as nn\n\n\ndef conv_out(width, kernel_size, stride, padding, verbose=False):\n out = math.floor((width + 2 * padding - kernel_size) / stride) + 1\n if verbose:\n print(out)\n return out\n\n\ndef maxpool_out(width, kernel_size, stride=None, verbose=False):\n if stride is None:\n stride = kernel_size # PyTorch default\n out = math.floor((width - kernel_size) / stride) + 1\n if verbose:\n print(out)\n return out\n\n\nx = conv_out(28, kernel_size=3, stride=1, padding=0, verbose=True) # 26\nx = maxpool_out(x, kernel_size=2, verbose=True) # 13\nx = conv_out(x, kernel_size=3, stride=1, padding=1, verbose=True) # 13\nx = maxpool_out(x, kernel_size=3, stride=2, verbose=True) # 6\nprint(\"flatten:\", 16 * 6 * 6) # 576\n\n# --- always double-check by running a dummy tensor through the layers -------\nnet = nn.Sequential(\n nn.Conv2d(1, 8, kernel_size=3, stride=1), nn.ReLU(), nn.MaxPool2d(kernel_size=2),\n nn.Conv2d(8, 16, kernel_size=3, stride=1, padding=1), nn.ReLU(),\n nn.MaxPool2d(kernel_size=3, stride=2),\n)\nt = torch.randn(1, 1, 28, 28)\nfor layer in net:\n t = layer(t)\n print(f\"{layer.__class__.__name__:<12} -> {tuple(t.shape)}\")\nprint(\"flattened in_features =\", t.numel())" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "### Manual convolution by hand (Assignment 6 Β§1.1)\n\nGiven filters and an input, apply the filter at each valid position and sum the element-wise products. Use the notation $h[\\text{row},\\text{column},\\text{matrix}]$.\n\n$$\nW_1 = \\begin{pmatrix} -1 & -1 & -1 \\\\ 2 & 2 & 2 \\\\ -1 & -1 & -1 \\end{pmatrix}\n\\quad\nW_2 = \\begin{pmatrix} -1 & 2 & -1 \\\\ -1 & 2 & -1 \\\\ -1 & 2 & -1 \\end{pmatrix}\n\\quad\nX = \\begin{pmatrix} 1&1&1&1 \\\\ 4&4&4&4 \\\\ 1&1&1&1 \\\\ 1&1&1&1 \\end{pmatrix}\n$$\n\nOutput volume: $\\text{H}_{out} = 4-3+2(0)+1 = 2$, likewise for width, and 2 filters β†’ **$2\\times2\\times2$**.\n\n$$\\begin{aligned}\nh[0,0,0] &= 3(-1 \\times 1) + 3(2 \\times 4) + 3(-1 \\times 1) \\\\\n &= 3(-1) + 3(8) + 3(-1) = 18\n\\end{aligned}$$\n\n$$\\begin{aligned}\nh[1,0,0] &= 3(-1 \\times 4) + 3(2 \\times 1) + 3(-1 \\times 1) \\\\\n &= 3(-4) + 3(2) + 3(-1) = -9\n\\end{aligned}$$\n\n$$\\begin{aligned}\nh[0,0,1] &= -1(1+4+1) + 2(1+4+1) - 1(1+4+1) = 0\n\\end{aligned}$$\n\n$$f_1 = \\begin{pmatrix} 18 & 18 \\\\ -9 & -9 \\end{pmatrix},\n\\qquad f_2 = \\begin{pmatrix} 0 & 0 \\\\ 0 & 0 \\end{pmatrix}$$\n\n**Why they differ:** $f_1$ prefers a **high centre row** with low rows above and below β€” a horizontal-pattern detector, and $X$ has exactly that. $f_2$ is the same filter rotated 90Β°, so it detects **vertical** patterns; the data has none, and the filter's symmetry makes the response exactly zero.\n\n**$2\\times2$ max pooling on that output** gives $p_1 = (18)$, $p_2 = (0)$. These are **not scalars**: the two filters belong to one conv layer, so the output is a single $2\\times2\\times2$ volume; pooling each slice independently yields a $1\\times1\\times2$ volume." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” verify the by-hand convolution with F.conv2d\nimport torch, torch.nn.functional as F\n\nX = torch.tensor([[1., 1., 1., 1.],\n [4., 4., 4., 4.],\n [1., 1., 1., 1.],\n [1., 1., 1., 1.]]).view(1, 1, 4, 4)\n\nW = torch.empty(2, 1, 3, 3)\nW[0, 0] = torch.tensor([[-1., -1., -1.], [2., 2., 2.], [-1., -1., -1.]]) # horizontal\nW[1, 0] = torch.tensor([[-1., 2., -1.], [-1., 2., -1.], [-1., 2., -1.]]) # vertical\n\nout = F.conv2d(X, W) # no padding, stride 1\nprint(\"conv output shape:\", tuple(out.shape), \"-> 2 x 2 x 2 volume\")\nprint(\"f1 =\\n\", out[0, 0])\nprint(\"f2 =\\n\", out[0, 1])\n\npooled = F.max_pool2d(out, kernel_size=2)\nprint(\"after 2x2 max pool:\", tuple(pooled.shape), \"->\", pooled.flatten().tolist())" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 8.2 What filters detect\n\nFive classic $3\\times3$ kernels, applied to a real image, show what a conv layer *learns* to do." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” the five classic kernels on a synthetic digit-like image\nimport torch, torch.nn.functional as F, matplotlib.pyplot as plt\n\n# A synthetic \"digit\": a bright cross on a dark field (replace with mnist_train.data[12])\nimg = torch.zeros(28, 28)\nimg[10:18, 4:24] = 200.0\nimg[4:24, 12:16] = 255.0\nx = img.view(1, 1, 28, 28)\n\nweight = torch.empty(5, 1, 3, 3)\nweight[0, 0] = torch.tensor([[0., 0., 0.], [0., 1., 0.], [0., 0., 0.]]) # identity\nweight[1, 0] = torch.tensor([[1., 1., 1.], [1., 1., 1.], [1., 1., 1.]]) # box blur\nweight[2, 0] = torch.tensor([[-1., 0., 1.], [-1., 0., 1.], [-1., 0., 1.]]) # vertical edge\nweight[3, 0] = torch.tensor([[-1., -1., -1.], [0., 0., 0.], [1., 1., 1.]]) # horizontal edge\nweight[4, 0] = torch.tensor([[0., -1., 0.], [-1., 4., -1.], [0., -1., 0.]]) # laplacian\n\ny = F.conv2d(x, weight)\n\nfig, axes = plt.subplots(2, 3, figsize=(10, 6))\naxes = axes.flatten()\naxes[0].imshow(x[0, 0].numpy(), cmap=\"gray\"); axes[0].set_title(\"Original Image\"); axes[0].axis(\"off\")\nfor i, name in enumerate([\"Identity\", \"Box Blur\", \"Vertical Edge\", \"Horizontal Edge\", \"Sharpening\"]):\n axes[i + 1].imshow(y[0, i].detach().numpy(), cmap=\"gray\")\n axes[i + 1].set_title(name); axes[i + 1].axis(\"off\")\nplt.tight_layout(); plt.show()" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "**Effect of the three hyperparameters** (the interactive-slider takeaway):\n\n- **Kernel ↑** β€” the filter sees a larger region at once, so the output looks smoother / more influenced by surrounding pixels.\n- **Stride ↑** β€” the filter moves further each step, so the output gets **smaller** and loses fine detail (fewer positions sampled).\n- **Padding ↑** β€” the border is preserved, so the output stays closer to the original size and edge information is not cut off.\n\n> A conv output is a **feature map**, not an image: values can be negative, and Matplotlib rescales them for display. That is why filtered digits can look \"inverted\"." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 8.3 CNN architectures\n\n### `SimpleCNN` β€” the lecture baseline\n\n⚠️ **Remove `nn.Softmax`** when you use `nn.CrossEntropyLoss`. The loss expects **raw logits**: internally it applies `LogSoftmax` followed by `NLLLoss`. Adding a Softmax first is redundant and makes training unstable or plain wrong." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” the three CNNs from Assignment 6, smallest to largest\nimport torch, torch.nn as nn\n\n\nclass SimpleCNN(nn.Module):\n \"\"\"Lecture baseline (Softmax REMOVED - CrossEntropyLoss wants logits).\"\"\"\n\n def __init__(self, num_channels=1, num_classes=10):\n super().__init__()\n self.model = nn.Sequential(\n nn.Conv2d(num_channels, 32, kernel_size=3, stride=1), nn.ReLU(),\n nn.MaxPool2d(kernel_size=2),\n nn.Conv2d(32, 32, kernel_size=3, stride=1), nn.ReLU(),\n nn.MaxPool2d(kernel_size=2),\n nn.Flatten(1),\n nn.Dropout(),\n nn.LazyLinear(num_classes), # infers in_features on first forward\n )\n\n def forward(self, x):\n return self.model(x)\n\n\nclass ExpCNN(nn.Module):\n \"\"\"Simplified: one conv block. Nearly the same accuracy, far fewer params.\"\"\"\n\n def __init__(self, num_channels=1, num_classes=10):\n super().__init__()\n self.model = nn.Sequential(\n nn.Conv2d(num_channels, 32, kernel_size=3, stride=1), nn.ReLU(),\n nn.MaxPool2d(kernel_size=2),\n nn.Flatten(1), nn.Dropout(0.3), nn.LazyLinear(num_classes),\n )\n\n def forward(self, x):\n return self.model(x)\n\n\nclass ExpCNN2(nn.Module):\n \"\"\"Deeper + BatchNorm + graduated Dropout: >99% on MNIST after 5 epochs.\"\"\"\n\n def __init__(self, num_channels=1, num_classes=10):\n super().__init__()\n self.model = nn.Sequential(\n nn.Conv2d(num_channels, 32, kernel_size=3, padding=1), nn.BatchNorm2d(32), nn.ReLU(),\n nn.Conv2d(32, 32, kernel_size=3, padding=1), nn.BatchNorm2d(32), nn.ReLU(),\n nn.MaxPool2d(2), nn.Dropout(0.1),\n\n nn.Conv2d(32, 64, kernel_size=3, padding=1), nn.BatchNorm2d(64), nn.ReLU(),\n nn.Conv2d(64, 64, kernel_size=3, padding=1), nn.BatchNorm2d(64), nn.ReLU(),\n nn.MaxPool2d(2), nn.Dropout(0.15),\n\n nn.Conv2d(64, 128, kernel_size=3, padding=1), nn.BatchNorm2d(128), nn.ReLU(),\n\n nn.Flatten(), nn.LazyLinear(256), nn.ReLU(), nn.Dropout(0.3),\n nn.Linear(256, num_classes),\n )\n\n def forward(self, x):\n return self.model(x)\n\n\ndummy = torch.randn(2, 1, 28, 28)\nfor cls in (SimpleCNN, ExpCNN, ExpCNN2):\n m = cls()\n out = m(dummy) # first pass materialises LazyLinear\n n_params = sum(p.numel() for p in m.parameters())\n print(f\"{cls.__name__:<10} out {tuple(out.shape)} params {n_params:,}\")" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” the exam's features/classifier two-block CNN (Exams AS25 & SS25 3.3)\nimport torch, torch.nn as nn\n\n\nclass CNNModel(nn.Module):\n \"\"\"SS25 variant: Conv(1->32,k5,p2) -> BN -> ReLU -> MaxPool(2,2)\n Conv(32->64,k3,s2) -> BN -> ReLU -> Flatten\n then LazyLinear(128) -> ReLU -> Dropout(0.5) -> Linear(128,10).\"\"\"\n\n def __init__(self):\n super().__init__()\n self.features = nn.Sequential(\n nn.Conv2d(1, 32, kernel_size=5, padding=2),\n nn.BatchNorm2d(32),\n nn.ReLU(),\n nn.MaxPool2d(kernel_size=2, stride=2),\n nn.Conv2d(32, 64, kernel_size=3, stride=2),\n nn.BatchNorm2d(64),\n nn.ReLU(),\n nn.Flatten(),\n )\n self.classifier = nn.Sequential(\n nn.LazyLinear(128),\n nn.ReLU(),\n nn.Dropout(p=0.5),\n nn.Linear(128, 10),\n )\n\n def forward(self, x):\n x = self.features(x)\n x = self.classifier(x)\n return x\n\n\nmodel = CNNModel()\nprint(model(torch.randn(4, 1, 16, 16)).shape, \" # USPS is 16x16 -> (batch, 64, 3, 3) -> 576\")\nprint(model)\n\n\n# ⚠️ Backup model, only if yours does not work β€” it still earns the training points\nclass BackupCNN(nn.Module):\n def __init__(self):\n super().__init__()\n self.conv = nn.Conv2d(1, 8, kernel_size=3)\n self.relu = nn.ReLU()\n self.fc = nn.Linear(8 * 14 * 14, 10)\n\n def forward(self, x):\n x = self.conv(x)\n x = self.relu(x)\n x = x.view(x.size(0), -1)\n return self.fc(x)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "πŸ“Œ **Exam pattern.** The CNN implementation question is worth 8–10 points and is *dictated layer by layer* β€” read the list literally, in order. The AS25 variant uses `Conv(1β†’16, k3, p1)` and `Dropout(0.4)`; SS25 uses `Conv(1β†’32, k5, p2)` and `Dropout(0.5)`. Then 3.4 asks you to instantiate, print the model, set `CrossEntropyLoss` + `Adam(lr=0.001)` + 10 epochs, call `train(...)`, `plot(...)`, `evaluate(...)` and print `CNN USPS. Validation loss: x.xx, validation accuracy: x.xx`." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 9. Data augmentation & evaluation metrics\n\n[↑ TOC](#toc)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 9.1 torchvision `v2` transforms\n\nAugmentation creates **plausible variations** of the training data so the model generalises instead of memorising. The key word is *plausible*: augmentation that destroys the class identity makes things worse.\n\n### The catalogue\n\n| Transform | Typical call |\n|---|---|\n| Horizontal flip | `v2.RandomHorizontalFlip(p=0.5)` |\n| Vertical flip | `v2.RandomVerticalFlip(p=0.5)` |\n| Random crop | `v2.RandomCrop(160)` |\n| Random resized crop | `v2.RandomResizedCrop(size=160, scale=(0.1, 1))` |\n| Photometric distortion | `v2.RandomPhotometricDistort(p=0.5)` |\n| Affine (rotate/translate/scale/shear) | `v2.RandomAffine(degrees=45, translate=(0.1,0.1), scale=(0.75,1.5), shear=(0,10), fill=0)` |\n| Rotation | `v2.RandomRotation(degrees=10, fill=0)` |\n| Gaussian blur | `v2.GaussianBlur(kernel_size=(5,9), sigma=(0.1,5))` |\n| Random erasing | `v2.RandomErasing(p=1)` |\n| Random resize | `v2.RandomResize(min_size=5, max_size=28)` |\n\n### The four pipeline steps that are always there\n\n```python\nv2.ToImage() # into the Image class\nv2.Resize((28, 28)) # fixed spatial size\nv2.ToDtype(torch.float32, scale=True) # -> float32 in [0, 1]\nv2.Normalize(mean=(0.5,), std=(0.5,)) # -> [-1, 1]\n```\n\n`mean=0.5, std=0.5` maps $[0,1]\\to[-1,1]$. For 3-channel images pass three values. To undo it for display: `image * 0.5 + 0.5`.\n\nCIFAR10 has its own statistics: `mean=(0.4914, 0.4822, 0.4465), std=(0.2023, 0.1994, 0.2010)`. MNIST: `mean=(0.1307,), std=(0.3081,)`." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” basic vs bad vs good augmentation pipelines\nimport torch\nfrom torchvision.transforms import v2\n\ntransforms_basic = v2.Compose([\n v2.ToImage(),\n v2.Resize((28, 28)),\n v2.ToDtype(torch.float32, scale=True),\n v2.Normalize(mean=(0.5,), std=(0.5,)),\n])\n\n# ❌ too aggressive for MNIST: flips turn 6 into 9, blur destroys thin strokes\ntransforms_aug = v2.Compose([\n v2.ToImage(),\n v2.Resize((28, 28)),\n v2.RandomAffine(degrees=20, translate=(0.15, 0.15), scale=(0.9, 1.1), shear=15, fill=0),\n v2.RandomVerticalFlip(0.75),\n v2.RandomHorizontalFlip(0.75),\n v2.GaussianBlur(kernel_size=(5, 9), sigma=(0.5, 1.2)),\n v2.ToDtype(torch.float32, scale=True),\n v2.Normalize(mean=(0.5,), std=(0.5,)),\n])\n\n# βœ… mild and label-preserving: models handwriting variation, nothing more\ntransforms_aug2 = v2.Compose([\n v2.ToImage(),\n v2.RandomRotation(degrees=10, fill=0), # writing-style variation\n v2.GaussianBlur(kernel_size=(3, 5), sigma=(0.1, 1)), # mild, prevents overfitting\n v2.ToDtype(torch.float32, scale=True),\n v2.Normalize(mean=(0.5,), std=(0.5,)),\n])\n\nimg = torch.rand(1, 28, 28)\nfor name, t in [(\"basic\", transforms_basic), (\"aggressive\", transforms_aug), (\"mild\", transforms_aug2)]:\n print(f\"{name:>11}: {tuple(t(img).shape)} range [{t(img).min():.2f}, {t(img).max():.2f}]\")" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# reference β€” plot_transform: show 9 augmented versions next to the original\nimport torch, numpy as np, matplotlib.pyplot as plt\n\n\ndef plot_transform(image, transformation_fn, cmap=None):\n if isinstance(image, torch.Tensor):\n image = image.squeeze(0) # (C,H,W) -> (H,W) for grayscale\n image = image * 0.5 + 0.5 # denormalise [-1,1] -> [0,1]\n image = image.numpy()\n\n plt.figure(figsize=(12, 6))\n plt.subplot(2, 5, 1); plt.imshow(image, cmap=cmap)\n plt.axis(\"off\"); plt.title(\"Original Image\")\n\n for i in range(9):\n t = transformation_fn(image)\n if isinstance(t, torch.Tensor):\n t = (t.squeeze(0) * 0.5 + 0.5).numpy()\n plt.subplot(2, 5, i + 2); plt.imshow(t, cmap=cmap)\n plt.axis(\"off\"); plt.title(f\"Augmentation {i+1}\")\n\n plt.tight_layout(); plt.show()" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "### What the augmentation experiment shows\n\nThree models, identical architecture, identical hyperparameters, identical 5 000-sample subset β€” only the transforms differ:\n\n| Model | Test accuracy | Why |\n|---|---|---|\n| CNN Basic | 97.2 % | solid baseline |\n| CNN Augmented (aggressive) | **79.9 %** | vertical + horizontal flips made MNIST too diverse; unnatural variations confused visually similar digits |\n| CNN Augmented 2 (mild) | **98.8 %** | well-designed augmentations genuinely improve generalisation |\n\n> **The conclusion to write:** augmentation only helps when the transformations preserve the label. Combining all eight transforms produces samples that deviate so far from the original that important features are lost and the samples no longer represent the true data distribution.\n\nBoth augmented and basic subsets must use the **same indices** so the comparison is fair." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 9.2 Confusion matrix & classification report\n\n### Confusion matrix, by hand\n\nGiven $y = (+1, -1, +1, -1, +1, -1)$ and $\\hat y = (+1, -1, +1, +1, +1, -1)$, compare pairwise:\n\n| Position | $y$ | $\\hat y$ | Outcome |\n|---|---|---|---|\n| 1 | +1 | +1 | TP |\n| 2 | βˆ’1 | βˆ’1 | TN |\n| 3 | +1 | +1 | TP |\n| 4 | βˆ’1 | **+1** | **FP** |\n| 5 | +1 | +1 | TP |\n| 6 | βˆ’1 | βˆ’1 | TN |\n\n**TP = 3, FP = 1, TN = 2, FN = 0.**\n\n### The metrics\n\n$$\n\\text{Accuracy} = \\frac{TP+TN}{TP+TN+FP+FN}\n\\qquad\n\\text{Precision} = \\frac{TP}{TP+FP}\n\\qquad\n\\text{Recall} = \\frac{TP}{TP+FN}\n$$\n\n$$\nF_1 = 2\\cdot\\frac{\\text{Precision}\\cdot\\text{Recall}}{\\text{Precision}+\\text{Recall}}\n$$\n\n- **Precision** β€” of everything I flagged positive, how much really was?\n- **Recall** β€” of everything that really was positive, how much did I catch?\n- **F1** β€” their harmonic mean; punishes a model that is good at only one.\n- **Support** β€” the number of true samples of that class in the test set.\n\n### Imbalanced data\n\nWith imbalanced classes the model becomes biased towards the majority class and **plain accuracy becomes misleading** (99 % accuracy by always predicting the majority class). Augment the minority class, or judge by precision/recall/F1. Simply collecting more data does not automatically fix the imbalance." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” confusion matrix by hand and with sklearn\nimport numpy as np\nfrom sklearn.metrics import confusion_matrix, classification_report\n\ny = np.array([+1, -1, +1, -1, +1, -1])\ny_hat = np.array([+1, -1, +1, +1, +1, -1])\n\nTP = int(np.sum((y == 1) & (y_hat == 1)))\nTN = int(np.sum((y == -1) & (y_hat == -1)))\nFP = int(np.sum((y == -1) & (y_hat == 1)))\nFN = int(np.sum((y == 1) & (y_hat == -1)))\nprint(f\"TP: {TP}, FP: {FP}, TN: {TN}, FN: {FN}\")\n\nprecision = TP / (TP + FP)\nrecall = TP / (TP + FN)\nf1 = 2 * precision * recall / (precision + recall)\nprint(f\"accuracy={(TP+TN)/len(y):.3f} precision={precision:.3f} recall={recall:.3f} f1={f1:.3f}\")\n\nprint(\"\\nsklearn confusion matrix (rows = true, cols = predicted):\")\nprint(confusion_matrix(y, y_hat, labels=[-1, 1]))\nprint(classification_report(y, y_hat, labels=[-1, 1],\n target_names=[\"negative\", \"positive\"], zero_division=0))" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# reference β€” test_model: accuracy + classification report + confusion-matrix heatmap\nimport numpy as np, torch, seaborn as sns, matplotlib.pyplot as plt\nfrom tqdm.auto import tqdm\nfrom sklearn.metrics import confusion_matrix, classification_report\n\n\ndef test_model(model, test_loader, categories=None):\n \"\"\"Evaluate on the test set with a classification report and a confusion matrix.\n\n Returns (test_acc, cm, all_labels, all_preds).\n `categories` is the list of class names, e.g. [str(i) for i in range(10)],\n so the report and heatmap show meaningful labels instead of indices.\n \"\"\"\n if len(test_loader) == 0:\n raise ValueError(\"test_loader is empty.\")\n\n model.eval()\n correct = total = 0\n all_preds, all_labels = [], []\n\n with torch.no_grad():\n for images, labels in tqdm(test_loader, desc=\"Testing\", unit=\"batch\"):\n images, labels = images.to(device), labels.to(device)\n outputs = model(images)\n _, predicted = torch.max(outputs, 1)\n total += labels.size(0)\n correct += (predicted == labels).sum().item()\n all_preds.extend(predicted.cpu().numpy())\n all_labels.extend(labels.cpu().numpy())\n\n test_acc = 100.0 * correct / total\n unique_labels = sorted(set(all_labels) | set(all_preds))\n cm = confusion_matrix(all_labels, all_preds, labels=unique_labels)\n display_names = ([f\"Class {i}\" for i in unique_labels] if categories is None\n else [categories[i] for i in unique_labels])\n\n print(f\"\\nTest Accuracy: {test_acc:.2f}%\\n\")\n print(\"Classification Report:\\n\")\n print(classification_report(all_labels, all_preds, labels=unique_labels,\n target_names=display_names, zero_division=0))\n\n plt.figure(figsize=(8, 6))\n sns.heatmap(cm, annot=True, fmt=\"d\", cmap=\"Blues\",\n xticklabels=display_names, yticklabels=display_names)\n plt.title(\"Confusion Matrix\"); plt.xlabel(\"Predicted Label\"); plt.ylabel(\"True Label\")\n plt.tight_layout(); plt.show()\n\n return test_acc, cm, all_labels, all_preds" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "**How to read a report in one paragraph** (the model answer style):\n\n> *The model achieved an overall accuracy of 79.9 %, which indicates fairly weak performance. It performs well on classes 0, 1, 4 and 8, where recall is high, meaning it correctly identifies most of those digits. Performance is poor for classes 6 and 9, where both precision and recall fall below 0.50. This suggests the model learned some digit patterns but struggles to generalise across all classes, likely due to the small training set.*\n\nName the strong classes, name the weak ones, quote precision/recall for the weak ones, and give one plausible cause." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 9.3 Inspecting predictions" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# reference β€” visualize_predictions: correct on the top row, mistakes on the bottom\nimport numpy as np, torch, matplotlib.pyplot as plt\n\n\ndef visualize_predictions(model, test_loader, categories, num_images=10):\n \"\"\"Show correctly (green) and incorrectly (red) classified test images.\"\"\"\n device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n model.to(device); model.eval()\n\n correct_samples, incorrect_samples = [], []\n with torch.no_grad():\n for images, labels in test_loader:\n images, labels = images.to(device), labels.to(device)\n predicted = torch.argmax(model(images), dim=1)\n for i in range(len(predicted)):\n sample = (images[i].detach().cpu(), labels[i].item(), predicted[i].item())\n (correct_samples if predicted[i] == labels[i] else incorrect_samples).append(sample)\n\n num_correct = min(num_images // 2, len(correct_samples))\n num_incorrect = min(num_images - num_correct, len(incorrect_samples))\n num_cols = max(num_correct, num_incorrect)\n if num_cols == 0:\n print(\"No images available to display.\"); return\n\n mean = torch.tensor([0.5, 0.5, 0.5]).view(3, 1, 1)\n std = torch.tensor([0.5, 0.5, 0.5]).view(3, 1, 1)\n plt.figure(figsize=(4 * num_cols, 8))\n\n for row, (samples, n, colour) in enumerate([(correct_samples, num_correct, \"green\"),\n (incorrect_samples, num_incorrect, \"red\")]):\n for i in range(num_cols):\n plt.subplot(2, num_cols, row * num_cols + i + 1)\n if i < n:\n img_tensor, true_label, pred_label = samples[i]\n img_tensor = img_tensor * std + mean # reverse normalisation\n img = np.clip(img_tensor.permute(1, 2, 0).numpy(), 0, 1)\n plt.imshow(img)\n plt.title(f\"True: {categories[true_label]}\\nPred: {categories[pred_label]}\",\n color=colour, fontsize=12)\n plt.axis(\"off\")\n\n plt.tight_layout(); plt.show()" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 10. Transfer learning & fine-tuning\n\n[↑ TOC](#toc)\n\n**Transfer learning** = take a network pretrained on a huge dataset, replace its classifier head, and train **only the head**. The backbone's features already encode edges, textures and shapes, so you get good results with little data and little compute.\n\n**Fine-tuning** = additionally unfreeze the *last* convolutional block so it can adapt to your domain, usually with a smaller learning rate.\n\nThe four-step recipe:\n\n1. Load the pretrained model and its default weights.\n2. `print(model)` to find the name of the classifier layer (`fc` for ResNet, `classifier` for VGG/DenseNet).\n3. Read `in_features` from it, then replace it with a `nn.Linear(in_features, num_classes)` β€” a **newly created layer always has `requires_grad=True`**.\n4. Freeze everything, then unfreeze exactly what you want to train." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# reference β€” ResNet18 transfer learning, then fine-tuning # needs download\nimport copy, torch, torch.nn as nn, torch.optim as optim\nfrom torchvision import models\n\n# 1-2) load pretrained weights and inspect\nresnet_weights = models.ResNet18_Weights.DEFAULT\nresnet_model = models.resnet18(weights=resnet_weights)\n# print(resnet_model) # -> the last layer is called `fc`\n\n# 3) replace the classifier head for 2 classes (cats vs dogs)\nin_features = resnet_model.fc.in_features\nresnet_model.fc = nn.Linear(in_features, 2)\nresnet_model = resnet_model.to(device)\n\n# 4) freeze everything, then unfreeze ONLY the new head\nfor param in resnet_model.parameters():\n param.requires_grad = False\nfor param in resnet_model.fc.parameters():\n param.requires_grad = True\n\nfor name, param in resnet_model.named_parameters(): # always double-check\n if param.requires_grad:\n print(\"trainable:\", name)\n\ncriterion = nn.CrossEntropyLoss()\noptimizer = optim.Adam(resnet_model.parameters(), lr=0.001)\nn_epochs = 50\n\n# best_model_resnet, history_resnet = train_model(\n# model=resnet_model, train_loader=train_dataloader, val_loader=val_dataloader,\n# criterion=criterion, optimizer=optimizer, num_epochs=n_epochs,\n# name='resnet', use_early_stopping=True, verbose=True, min_delta=0.001)\n# plot_training_history(history_resnet, plot_lr=True)\n\n# ---- FINE-TUNING: copy the model, then also unfreeze layer4 ----------------\nfinetune_model = copy.deepcopy(resnet_model)\n\nfor param in finetune_model.parameters(): # safe reset\n param.requires_grad = False\nfor name, layer in finetune_model.named_children(): # named_children -> top-level blocks\n if name in [\"layer4\", \"fc\"]:\n for param in layer.parameters():\n param.requires_grad = True\n\noptimizer_finetune = optim.Adam(finetune_model.parameters(), lr=0.001)\n# best_model_finetune, history_finetune = train_model(\n# model=finetune_model, train_loader=train_dataloader, val_loader=val_dataloader,\n# criterion=criterion, optimizer=optimizer_finetune, num_epochs=n_epochs,\n# name='resnet finetune', use_early_stopping=True, verbose=True)" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# reference β€” the dogs/cats data pipeline that feeds it\nimport os, torch\nfrom torchvision import datasets\nfrom torchvision.transforms import v2\nfrom torch.utils.data import DataLoader\n\n# DATA_PATH/\n# β”œβ”€β”€ train/{cats,dogs}\n# └── test/{cats,dogs}\nDATA_PATH = \"/content\"\ntrain, test = \"train\", \"test\"\ncategories = os.listdir(os.path.join(DATA_PATH, train))\n\ntrain_dataset = datasets.ImageFolder(os.path.join(DATA_PATH, train))\nval_dataset = datasets.ImageFolder(os.path.join(DATA_PATH, test))\n\ntransforms_rgb = v2.Compose([\n v2.Resize((100, 100)),\n v2.ToImage(),\n v2.ToDtype(torch.float32, scale=True),\n v2.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),\n])\n\ntrain_RAM = RAMDatasetWrapper(train_dataset, transforms_rgb)\nval_RAM = RAMDatasetWrapper(val_dataset, transforms_rgb)\n\ntrain_dataloader = DataLoader(train_RAM, batch_size=32, shuffle=True, num_workers=0)\nval_dataloader = DataLoader(val_RAM, batch_size=32, shuffle=False, num_workers=0)\n\nfor images, labels in train_dataloader:\n print(f\"image shape: {images.shape}, labels shape: {labels.shape}\")\n break\n\n\ndef class_distribution(dataset):\n \"\"\"Print total sample count and per-class counts for a split.\"\"\"\n path = os.path.join(DATA_PATH, dataset)\n counts = [len(os.listdir(os.path.join(path, cat))) for cat in categories]\n print(f\"Number of samples: {sum(counts)}\")\n for cat, count in zip(categories, counts):\n print(f\"{cat}: {count} samples\")" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "πŸ“Œ **Exam pattern.** *\"Why does transfer learning often outperform training from scratch?\"* β€” because it starts from features learned on large datasets, so the model learns faster and performs better with less training data. The mechanical part that is graded: get `in_features` from the existing layer, freeze all, unfreeze the head, and **verify with `named_parameters()`**.\n\n\n\n---\n\n# 11. Autoencoders & Variational Autoencoders\n\n[↑ TOC](#toc)\n\nAn **autoencoder** compresses input into a latent code and reconstructs it. It needs **no labels** β€” the input is its own target. A **VAE** makes the latent space *probabilistic*: the encoder predicts a mean $\\mu$ and a log-variance $\\log\\sigma^2$, and you sample from that distribution. That is what makes the space continuous enough to generate new samples by decoding random points.\n\n### The reparameterization trick\n\nYou cannot backpropagate through random sampling. So instead of drawing $z \\sim \\mathcal{N}(\\mu, \\sigma^2)$ directly, draw the noise separately and shift it:\n\n$$z = \\mu + \\varepsilon \\odot \\sigma, \\qquad \\varepsilon \\sim \\mathcal{N}(0, 1), \\qquad \\sigma = e^{\\tfrac12 \\log \\sigma^2}$$\n\nNow the randomness sits in $\\varepsilon$, which has no parameters, and gradients flow cleanly through $\\mu$ and $\\sigma$.\n\nPredicting $\\log\\sigma^2$ rather than $\\sigma^2$ keeps the value unconstrained (any real number) while $e^{\\cdot}$ guarantees the variance stays positive.\n\n### The loss\n\n$$\\mathcal{L} = \\underbrace{\\text{BCE}(\\tilde x, x)}_{\\text{reconstruction}} + \\underbrace{-\\tfrac{1}{2}\\sum\\bigl(1 + \\log\\sigma^2 - \\mu^2 - \\sigma^2\\bigr)}_{\\text{KL divergence}}$$\n\nThe reconstruction term pushes outputs to look like the inputs; the KL term pulls the latent distribution towards $\\mathcal{N}(0,1)$ so the space stays smooth and samplable. `reduction=\"sum\"` is used, and the mean loss is divided by `len(train_loader.dataset)`." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” the full VAE (architecture, loss, training, sampling)\nimport numpy as np, torch, torch.nn as nn, torch.nn.functional as F\nimport matplotlib.pyplot as plt\nfrom torchvision.utils import make_grid\n\n\nclass VAE(nn.Module):\n def __init__(self, input_dim=784, hidden_dim=64, latent_dim=32):\n super().__init__()\n self.encoder = nn.Sequential(\n nn.Flatten(),\n nn.Linear(input_dim, hidden_dim),\n nn.ReLU(),\n )\n self.fc_mu = nn.Linear(hidden_dim, latent_dim)\n self.fc_logvar = nn.Linear(hidden_dim, latent_dim)\n self.decoder = nn.Sequential(\n nn.Linear(latent_dim, hidden_dim),\n nn.ReLU(),\n nn.Linear(hidden_dim, input_dim),\n nn.Sigmoid(), # outputs in [0,1] -> matches BCE\n )\n self.latent_dim = latent_dim\n\n def encode(self, x):\n z = self.encoder(x)\n return self.fc_mu(z), self.fc_logvar(z)\n\n def decode(self, z):\n return self.decoder(z)\n\n def sample(self, mu, logvar):\n std = torch.exp(0.5 * logvar) # e^(1/2 * log(std^2)) = std\n eps = torch.randn_like(std) # eps ~ N(0, 1)\n return mu + eps * std # <- the reparameterization trick\n\n def forward(self, x):\n mu, logvar = self.encode(x)\n z = self.sample(mu, logvar)\n return self.decode(z), mu, logvar\n\n\ndef vae_loss(x_tilde, x, mu, logvar):\n x = x.view(x_tilde.size()) # a view, no extra memory\n BCE = F.binary_cross_entropy(x_tilde, x, reduction=\"sum\")\n KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())\n return BCE + KLD\n\n\ndef train_vae(model, optimizer, train_loader, device):\n model.train()\n train_loss = 0\n for data, _ in train_loader:\n x = data.to(device)\n x_tilde, mu, logvar = model(x)\n loss = vae_loss(x_tilde, x, mu, logvar)\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n train_loss += loss.item()\n return train_loss / len(train_loader.dataset)\n\n\n# smoke test on random \"images\" so the cell runs offline\nvae = VAE(latent_dim=2).to(device)\nx = torch.rand(8, 1, 28, 28).to(device)\nx_tilde, mu, logvar = vae(x)\nprint(\"reconstruction:\", tuple(x_tilde.shape), \"| mu:\", tuple(mu.shape))\nprint(\"loss:\", vae_loss(x_tilde, x, mu, logvar).item())" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# reference β€” the three VAE visualisations\nimport numpy as np, torch, matplotlib.pyplot as plt\nfrom torchvision.utils import make_grid\n\n\ndef visualize_samples(model, device, num_samples=16):\n \"\"\"Sample random z ~ N(0,1) and decode -> a grid of *new* digits.\"\"\"\n model.eval()\n with torch.no_grad():\n g = torch.Generator(device=device); g.manual_seed(42)\n z = torch.randn(num_samples, model.latent_dim, generator=g, device=device)\n samples = model.decode(z).cpu().view(-1, 1, 28, 28)\n grid = make_grid(samples, nrow=int(num_samples ** 0.5), padding=0)\n plt.figure(figsize=(5, 5))\n plt.imshow(np.transpose(grid.numpy(), (1, 2, 0)), cmap=\"gray\")\n plt.axis(\"off\"); plt.tight_layout(pad=0); plt.show()\n\n\ndef plot_latent_clusters(model, train_loader, device, num_batches=100):\n \"\"\"Encode the training set and scatter z, coloured by digit label.\"\"\"\n model.eval()\n zs, labels = [], []\n with torch.no_grad():\n for i, (data, label) in enumerate(train_loader):\n if i >= num_batches:\n break\n data = data.view(-1, 784).to(device) # nn.Linear needs 1-D input\n mu, logvar = model.encode(data)\n zs.append(model.sample(mu, logvar).cpu())\n labels.append(label)\n zs, labels = torch.cat(zs), torch.cat(labels)\n\n plt.figure(figsize=(8, 6))\n for digit in range(10):\n mask = labels == digit\n plt.scatter(zs[mask, 0], zs[mask, 1], label=str(digit), alpha=0.4, s=15)\n plt.legend(title=\"Digit\", loc=\"upper right\")\n plt.title(\"Latent Space Clusters (z ~ N(mu, σ²))\")\n plt.xlabel(\"z1\"); plt.ylabel(\"z2\"); plt.grid(True); plt.tight_layout(); plt.show()\n\n\ndef generate_image_from_input_z(model, z, device, digit_size=28):\n \"\"\"Decode ONE hand-picked latent coordinate.\"\"\"\n model.eval()\n z_tensor = torch.tensor([z], dtype=torch.float32).to(device)\n with torch.no_grad():\n return model.decode(z_tensor).view(digit_size, digit_size).cpu().numpy()\n\n\ndef plot_latent_space(model, device, scale=2.0, n=25, digit_size=28, figsize=8):\n \"\"\"Decode a full grid of latent coordinates -> the classic VAE manifold.\"\"\"\n assert model.latent_dim == 2, \"Latent space must be 2D to plot this grid.\"\n grid_x = np.linspace(-scale, scale, n)\n grid_y = np.linspace(scale, -scale, n)\n figure = np.zeros((digit_size * n, digit_size * n))\n\n model.eval()\n with torch.no_grad():\n for i, yi in enumerate(grid_y):\n for j, xi in enumerate(grid_x):\n z_sample = torch.tensor([[xi, yi]], dtype=torch.float32).to(device)\n digit = model.decode(z_sample).view(digit_size, digit_size).cpu().numpy()\n figure[i * digit_size:(i + 1) * digit_size,\n j * digit_size:(j + 1) * digit_size] = digit\n\n plt.figure(figsize=(figsize, figsize))\n ticks = np.arange(digit_size // 2, n * digit_size + digit_size // 2, digit_size)\n plt.xticks(ticks, np.round(grid_x, 1)); plt.yticks(ticks, np.round(grid_y, 1))\n plt.xlabel(\"z1\"); plt.ylabel(\"z2\")\n plt.imshow(figure, cmap=\"gray\")\n plt.title(\"Decoded Digits Across 2D Latent Space\")\n plt.tight_layout(); plt.show()" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "### Reading the latent space\n\n**Why `latent_dim = 2`?** So the space can be *seen*: plotted as a 2-D scatter, interpreted (similar digits cluster together β€” 4 and 9 overlap, 5 and 8 overlap), and sampled at chosen coordinates to see how latent values affect outputs. Higher dimensions reconstruct better but are much harder to interpret visually.\n\n**Hand-picked coordinates.** Read a cluster's location off the scatter plot, put it in a dict, decode each one:\n\n```python\nlatent_coords_by_digit = {0: [0, -3], 1: [-3, 0], 2: [-1, -1], 3: [-1, 0], 4: [1, 0.5],\n 5: [2.5, -0.5], 6: [0.5, -1], 7: [2, 2.5], 8: [0.5, 0], 9: [0, 1]}\n```\n\nIf the \"4\" comes out looking like a 9, that is the answer: **the latent clusters for 4 and 9 overlap**, meaning the model learned similar features for digits that share visual traits (curves, loops).\n\n**Hallucinations.** Decoding coordinates far outside the trained region (e.g. $z=[10,-10]$) still produces digit-like images β€” the decoder extrapolates, usually towards whichever cluster is nearest. Points between clusters produce blends of two digits.\n\nπŸ“Œ **Exam pattern.** Expect: *what does `visualize_samples` do?* (samples random $z$ from a normal distribution and decodes them into a grid of images that **resemble** MNIST digits), *why `data.view(-1, 784)`?* (linear layers need 1-D vectors; $28\\times28=784$), and *are autoencoders supervised?* (**no** β€” they need no labels)." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 12. RNNs & LSTMs\n\n[↑ TOC](#toc)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## 12.1 Tokenization & one-hot encoding\n\n### One-hot by hand\n\nVocabulary `['a', 'c', 'd', 'e', 'o', 'r']` mapped to indices $0\\ldots5$. The one-hot vector for a character is a column of zeros with a single 1 at its index:\n\n$$\\text{one-hot}(\\texttt{e}) = \\begin{bmatrix} 0 \\\\ 0 \\\\ 0 \\\\ 1 \\\\ 0 \\\\ 0 \\end{bmatrix}$$\n\nDecoding `[2, 3, 1, 4, 2, 3, 5]` β†’ `d e c o d e r` β†’ **\"decoder\"**.\n\nThe one-hot matrix for a sequence of 7 characters over a 6-symbol vocabulary is $X \\in \\mathbb{R}^{6\\times7}$ β€” **one column per time step**:\n\n$$X = \\begin{bmatrix}\n0&0&0&0&0&0&0 \\\\\n0&0&1&0&0&0&0 \\\\\n1&0&0&0&1&0&0 \\\\\n0&1&0&0&0&1&0 \\\\\n0&0&0&1&0&0&0 \\\\\n0&0&0&0&0&0&1\n\\end{bmatrix}$$\n\n### Embeddings\n\nOne-hot vectors are huge and carry no similarity structure. An **embedding matrix** $W_e$ maps each token to a dense vector: $W_e \\cdot e_{\\text{word}}$ simply **selects the column** of $W_e$ at that word's index.\n\nExam form (AS25 2.4): vocabulary $V = \\{\\text{taco}, \\text{broccoli}, \\text{pizza}, \\text{cupcake}\\}$,\n\n$$W_e = \\begin{bmatrix} 1&5&0&3 \\\\ 2&0&4&1 \\\\ 0&2&3&5 \\end{bmatrix}$$\n\n$e_{\\text{broccoli}} = (0,1,0,0)^\\top$ selects column 2 β†’ $(5,0,2)^\\top$;\n$e_{\\text{cupcake}} = (0,0,0,1)^\\top$ selects column 4 β†’ $(3,1,5)^\\top$.\n\n$$W_e e_{\\text{broccoli}} + W_e e_{\\text{cupcake}} = \\begin{bmatrix}5\\\\0\\\\2\\end{bmatrix} + \\begin{bmatrix}3\\\\1\\\\5\\end{bmatrix} = \\begin{bmatrix}8\\\\1\\\\7\\end{bmatrix}$$" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” one-hot, decoding, and the embedding lookup\nimport numpy as np\n\nvocab = [\"a\", \"c\", \"d\", \"e\", \"o\", \"r\"]\nidx = {c: i for i, c in enumerate(vocab)}\n\ndef one_hot(ch, V=vocab):\n v = np.zeros((len(V), 1), dtype=int); v[V.index(ch)] = 1\n return v\n\nprint(\"one-hot('e') =\", one_hot(\"e\").ravel())\n\nencoded = [2, 3, 1, 4, 2, 3, 5]\nprint(\"decoded =\", \"\".join(vocab[i] for i in encoded))\n\nX = np.hstack([one_hot(vocab[i]) for i in encoded])\nprint(\"X shape =\", X.shape, \"(vocab x sequence length)\")\nprint(X)\n\n# --- embedding lookup = column selection -----------------------------------\nW_e = np.array([[1, 5, 0, 3],\n [2, 0, 4, 1],\n [0, 2, 3, 5]])\nV = [\"taco\", \"broccoli\", \"pizza\", \"cupcake\"]\n\ndef embed(word):\n e = np.zeros(len(V)); e[V.index(word)] = 1\n return W_e @ e\n\nprint(\"\\nW_e . e_broccoli =\", embed(\"broccoli\"))\nprint(\"W_e . e_cupcake =\", embed(\"cupcake\"))\nprint(\"phrase 'broccoli cupcake' ->\", embed(\"broccoli\") + embed(\"cupcake\"))" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## 12.2 The Elman RNN\n\n$$h_t = \\tanh\\bigl(W_{ih}x_t + W_{hh}h_{t-1}\\bigr)$$\n\nor, with a bias term:\n\n$$h_t = \\tanh\\bigl(W_{hh}h_{t-1} + W_{xh}x_t + b_h\\bigr)$$\n\n### One step by hand (Assignment 9 Β§1.4)\n\n$$h_{t-1} = \\begin{bmatrix}0.2\\\\-0.1\\end{bmatrix},\\ x_t = \\begin{bmatrix}1\\\\0\\end{bmatrix},\\\nW_{hh} = \\begin{bmatrix}0.5&-0.2\\\\0.1&0.4\\end{bmatrix},\\\nW_{xh} = \\begin{bmatrix}0.3&0.8\\\\-0.5&0.2\\end{bmatrix},\\\nb_h = \\begin{bmatrix}0.1\\\\-0.2\\end{bmatrix}$$\n\n$$W_{hh}h_{t-1} = \\begin{bmatrix}0.12\\\\-0.02\\end{bmatrix},\\qquad\nW_{xh}x_t = \\begin{bmatrix}0.3\\\\-0.5\\end{bmatrix}$$\n\n$$\\text{sum} = \\begin{bmatrix}0.12\\\\-0.02\\end{bmatrix} + \\begin{bmatrix}0.3\\\\-0.5\\end{bmatrix} + \\begin{bmatrix}0.1\\\\-0.2\\end{bmatrix} = \\begin{bmatrix}0.52\\\\-0.72\\end{bmatrix}$$\n\n$$h_t = \\tanh\\left(\\begin{bmatrix}0.52\\\\-0.72\\end{bmatrix}\\right) \\approx \\begin{bmatrix}0.48\\\\-0.62\\end{bmatrix}$$" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” the RNN step by hand, then the three model variants\nimport numpy as np, torch, torch.nn as nn, torch.nn.functional as F\n\n# --- one step, verified -----------------------------------------------------\nh_prev = np.array([0.2, -0.1]); x_t = np.array([1, 0])\nW_hh = np.array([[0.5, -0.2], [0.1, 0.4]])\nW_xh = np.array([[0.3, 0.8], [-0.5, 0.2]])\nb_h = np.array([0.1, -0.2])\n\nprint(\"W_hh h_(t-1) =\", W_hh @ h_prev)\nprint(\"W_xh x_t =\", W_xh @ x_t)\nz = W_hh @ h_prev + W_xh @ x_t + b_h\nprint(\"sum =\", z)\nprint(\"h_t = tanh =\", np.round(np.tanh(z), 2))\n\n\n# --- (a) Elman RNN implemented from scratch, no nn.RNN ----------------------\nclass ElmanRNN(nn.Module):\n def __init__(self, num_tokens, embedding_dim, hidden_dim):\n super().__init__()\n self.hidden_dim = hidden_dim\n self.embedding = nn.Embedding(num_tokens, embedding_dim)\n self.W_ih = nn.Linear(embedding_dim, hidden_dim)\n self.W_hh = nn.Linear(hidden_dim, hidden_dim, bias=False) # no bias -> omit b\n self.out = nn.Linear(hidden_dim, num_tokens)\n\n def forward(self, x, hidden=None):\n embedded = self.embedding(x) # (batch, seq_len, emb)\n batch_size, seq_len, _ = embedded.size()\n h = (torch.zeros(batch_size, self.hidden_dim, device=embedded.device)\n if hidden is None else hidden[-1])\n\n outputs = []\n for t in range(seq_len): # unroll over time\n x_t = embedded[:, t, :]\n h = F.tanh(self.W_ih(x_t) + self.W_hh(h)) # the recurrence\n outputs.append(self.out(h))\n\n return torch.stack(outputs, dim=1), h.unsqueeze(0)\n\n\n# --- (b) built-in vanilla RNN ----------------------------------------------\nclass RNN(nn.Module):\n def __init__(self, num_tokens, embedding_dim, hidden_dim, num_layers=1):\n super().__init__()\n self.embedding = nn.Embedding(num_tokens, embedding_dim)\n self.rnn = nn.RNN(embedding_dim, hidden_dim, num_layers, batch_first=True)\n self.linear = nn.Linear(hidden_dim, num_tokens)\n\n def forward(self, x, hidden=None):\n embedded = self.embedding(x)\n output, hidden = self.rnn(embedded, hidden)\n return self.linear(output), hidden\n\n\n# --- (c) LSTM: identical except for one line -------------------------------\nclass LSTMModel(nn.Module):\n def __init__(self, num_tokens, embedding_dim, hidden_dim, num_layers=1):\n super().__init__()\n self.embedding = nn.Embedding(num_tokens, embedding_dim)\n self.rnn = nn.LSTM(embedding_dim, hidden_dim, num_layers, batch_first=True)\n self.linear = nn.Linear(hidden_dim, num_tokens)\n\n def forward(self, x, hidden=None):\n embedded = self.embedding(x)\n output, hidden = self.rnn(embedded, hidden)\n return self.linear(output), hidden\n\n\ntokens = torch.randint(0, 50, (4, 9)) # batch 4, sequence length 9\nfor cls in (ElmanRNN, RNN, LSTMModel):\n m = cls(num_tokens=50, embedding_dim=64, hidden_dim=128)\n logits, _ = m(tokens)\n print(f\"{cls.__name__:<10} logits {tuple(logits.shape)} # (batch, seq_len, vocab)\")" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "> `batch_first=True` makes the input shape `(batch, sequence, features)` instead of PyTorch's default `(sequence, batch, features)`. Use it everywhere and stay consistent.\n\n## 12.3 The text-generation pipeline\n\n### Tokenizer\n\nFour special tokens, always in this order at the front of the vocabulary: ``, ``, ``, ``." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” WordTokenizer, TextDataset, collate_sequences, training, generation\nimport torch, torch.nn as nn, torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.nn.utils.rnn import pad_sequence\n\n\nclass WordTokenizer:\n def __init__(self, corpus):\n words = sorted(set(word for line in corpus for word in line.strip().split()))\n words = [\"\", \"\", \"\", \"\"] + words\n\n self.word_to_index = {word: idx for idx, word in enumerate(words)}\n self.index_to_word = {idx: word for word, idx in self.word_to_index.items()}\n self.num_tokens = len(self.word_to_index)\n\n self.beg_token_id = self.word_to_index[\"\"]\n self.end_token_id = self.word_to_index[\"\"]\n self.pad_token_id = self.word_to_index[\"\"]\n self.unk_token_id = self.word_to_index[\"\"]\n\n def encode(self, text, add_beg_token=False, add_end_token=False):\n tokens = [self.word_to_index.get(w, self.unk_token_id) for w in text.strip().split()]\n if add_beg_token:\n tokens = [self.beg_token_id] + tokens\n if add_end_token:\n tokens.append(self.end_token_id)\n return tokens\n\n def decode(self, token_ids):\n return \" \".join(self.index_to_word[t] for t in token_ids)\n\n\nclass TextDataset(Dataset):\n def __init__(self, sequences):\n self.sequences = sequences\n\n def __len__(self):\n return len(self.sequences)\n\n def __getitem__(self, idx):\n return self.sequences[idx]\n\n\ndef collate_sequences(batch, pad_token_id):\n \"\"\"input = sequence without its LAST token; target = without its FIRST token.\n\n The target is the input shifted one position left: at every time step the\n model sees the current word and must predict the following one.\n \"\"\"\n inputs = [seq[:-1] for seq in batch if len(seq) > 1]\n targets = [seq[1:] for seq in batch if len(seq) > 1]\n inputs_padded = pad_sequence(inputs, batch_first=True, padding_value=pad_token_id)\n targets_padded = pad_sequence(targets, batch_first=True, padding_value=pad_token_id)\n return inputs_padded, targets_padded\n\n\ncorpus = [\n \"to be or not to be that is the question\",\n \"all the worlds a stage and all the men and women merely players\",\n \"romeo romeo wherefore art thou romeo\",\n \"now is the winter of our discontent\",\n \"the lady doth protest too much methinks\",\n \"if music be the food of love play on\",\n \"this above all to thine own self be true\",\n \"shall i compare thee to a summer day\",\n]\n\ntokenizer = WordTokenizer(corpus)\nprint(\"Vocab size:\", tokenizer.num_tokens)\nprint(\"First line:\", corpus[0])\nprint(\"Encoded:\", tokenizer.encode(corpus[0]))\n\nsequences = [\n torch.tensor(tokenizer.encode(line, add_beg_token=True, add_end_token=True),\n dtype=torch.long)\n for line in corpus\n if len(tokenizer.encode(line)) >= 2 # keep sequences of >= 2 words\n]\ndataset = TextDataset(sequences)\nprint(f\"Number of usable sequences: {len(sequences)}\")\n\nsample_inputs, sample_targets = collate_sequences([dataset[0], dataset[1]],\n tokenizer.pad_token_id)\nprint(\"Input :\", sample_inputs[0][:8].tolist())\nprint(\"Target:\", sample_targets[0][:8].tolist(), \" <- shifted left by one\")\nprint(\"shapes:\", tuple(sample_inputs.shape), tuple(sample_targets.shape))" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” train a small RNN and generate text\nimport torch, torch.nn as nn, torch.nn.functional as F\nfrom torch.utils.data import DataLoader\n\n\ndef train_model_rnn(model, dataset, tokenizer, batch_size=4, num_epochs=20,\n lr=0.005, device=\"cpu\"):\n model.to(device)\n pad_token_id = tokenizer.pad_token_id\n\n data_loader = DataLoader(\n dataset, batch_size=batch_size, shuffle=True,\n collate_fn=lambda b: collate_sequences(b, pad_token_id), # <- the key argument\n )\n\n optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n loss_fn = nn.CrossEntropyLoss(ignore_index=pad_token_id) # ignore padding!\n\n model.train()\n for epoch in range(num_epochs):\n total_loss = 0.0\n for inputs, targets in data_loader:\n inputs, targets = inputs.to(device), targets.to(device)\n optimizer.zero_grad()\n logits, hidden = model(inputs)\n # flatten batch+time so CE compares every predicted token to its target\n loss = loss_fn(logits.view(-1, tokenizer.num_tokens), targets.view(-1))\n loss.backward()\n optimizer.step()\n total_loss += loss.item()\n if (epoch + 1) % 5 == 0 or epoch == 0:\n print(f\"Epoch {epoch+1}/{num_epochs} | Loss: {total_loss / len(data_loader):.4f}\")\n\n\ndef generate_text_with_prompt(model, prompt_tokens, tokenizer, max_length=20,\n temperature=1.0, device=\"cpu\", stop_at_end=False):\n model.eval()\n input_tokens = torch.tensor([prompt_tokens], dtype=torch.long).to(device)\n hidden = None\n with torch.no_grad():\n logits, hidden = model(input_tokens) # consume the whole prompt first\n\n generated = prompt_tokens.copy()\n input_token = input_tokens[:, -1:]\n\n for _ in range(max_length):\n logits, hidden = model(input_token, hidden)\n logits = logits[:, -1, :] / temperature # temperature sharpens/flattens\n probabilities = F.softmax(logits, dim=-1)\n next_token = torch.multinomial(probabilities, num_samples=1).item()\n\n if stop_at_end and next_token == tokenizer.end_token_id:\n break\n\n generated.append(next_token)\n input_token = torch.tensor([[next_token]], dtype=torch.long).to(device)\n\n # strip every special token so the output reads cleanly\n generated = [t for t in generated if t not in (tokenizer.beg_token_id,\n tokenizer.end_token_id,\n tokenizer.pad_token_id)]\n return tokenizer.decode(generated)\n\n\nset_seed(42)\nembedding_dim, hidden_dim = 64, 128\nmodel_rnn = RNN(tokenizer.num_tokens, embedding_dim, hidden_dim)\ntrain_model_rnn(model_rnn, dataset, tokenizer, batch_size=4, num_epochs=20, device=\"cpu\")\n\nprompt_ids = tokenizer.encode(\"romeo romeo\") # lower-case avoids \nfor temp in (0.3, 1.0, 1.8):\n print(f\"\\nT={temp}: {generate_text_with_prompt(model_rnn, prompt_ids, tokenizer, max_length=12, temperature=temp)}\")" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "### Generation parameters\n\n- **`temperature`** β€” low (0.1–0.5) gives predictable, \"safe\" outputs from high-probability words, and more repetition. High (>1) gives diverse, creative outputs that are less coherent.\n- **`max_length`** β€” short outputs are usually more readable because the text has less time to drift off-topic; long outputs lose coherence and repeat more.\n\n### Judging generated text\n\nAnswer along four axes: **fluency** (grammatically natural?), **coherence** (does it hold together across lines?), **repetition** (too many repeated phrases?), **lyric-like quality** (does it read like the target genre?). For small RNNs the honest answer is: partially fluent, weakly coherent, noticeably repetitive, but with genre-appropriate vocabulary.\n\n### RNN + attention\n\n`simple_rnn_with_attention.py` shows the bridge to [Β§13](#s13): run an `nn.RNN`, then apply self-attention over its outputs and add a residual connection.\n\n```python\nself.rnn = nn.RNN(embedding_dim, hidden_dim, num_layers, batch_first=True)\nself.attention = nn.MultiheadAttention(embed_dim=hidden_dim, num_heads=1, batch_first=True)\nself.norm = nn.LayerNorm(hidden_dim)\n\noutput, hidden = self.rnn(embedded, hidden)\nattention, _ = self.attention(output, output, output) # Q = K = V -> self-attention\noutput = attention + output # residual connection\nlogits = self.linear(output)\n```" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 13. Attention & Transformers\n\n[↑ TOC](#toc)\n\n## 13.1 The intuition\n\nAttention is a soft lookup. A **query** asks a question; each item offers a **key** (what it is good at) and a **value** (what it contributes). Similarity between query and key decides how much of each value ends up in the answer.\n\nThe course's tutor analogy: you need strong theory plus some coding help, so $q = [0.8, 0.2]$.\n\n| | Key | Value |\n|---|---|---|\n| Tutor 1 (theory) | $k_1 = [1, 0]$ | $v_1 = [10, 0]$ |\n| Tutor 2 (coding) | $k_2 = [0, 1]$ | $v_2 = [0, 10]$ |\n| Tutor 3 (both) | $k_3 = [0.4, 0.3]$ | $v_3 = [5, 5]$ |\n\n## 13.2 Scaled dot-product attention, by hand\n\n$$\\text{Attention}(Q,K,V) = \\text{softmax}\\!\\left(\\frac{QK^\\top}{\\sqrt{d}}\\right)V$$\n\n**Step 1 β€” raw scores** ($q \\cdot k_i$): $\\ 0.8,\\ 0.2,\\ 0.38$\n**Step 2 β€” scale by $\\sqrt d$** with $d = 2$: $\\ 0.57,\\ 0.14,\\ 0.27$\n**Step 3 β€” softmax:** $e^{0.57}\\approx1.77$, $e^{0.14}\\approx1.15$, $e^{0.27}\\approx1.31$; sum $= 4.23$ β†’ weights $0.42,\\ 0.26,\\ 0.31$\n**Step 4 β€” weighted sum of values:**\n\n$$\\begin{aligned}\n\\text{output} &= 0.42v_1 + 0.26v_2 + 0.31v_3 \\\\\n&= [4.2, 0] + [0, 2.6] + [1.55, 1.55] \\\\\n&= [5.75, 4.25]\n\\end{aligned}$$\n\n**Interpretation to write:** the weights reflect how well each key aligns with the query. Tutor 1 gets the highest weight because their expertise matches the strong theoretical requirement; Tutor 3 gets a moderate weight for partially matching both; Tutor 2 the lowest for matching only the weaker component. **Attention distributes focus proportionally to similarity rather than picking a single best option.**\n\n**What softmax does:** it converts raw similarity scores into a probability distribution β€” normalising them to sum to 1 so they can be read as weights, and emphasising relative differences so larger scores become disproportionately more important.\n\n**Why divide by $\\sqrt d$:** without it, dot products grow with dimension, softmax saturates, and gradients vanish.\n\n> The numbers above carry the rounding used in the course solution. Computed exactly, the weights are $0.42 / 0.27 / 0.31$ and the output is $[5.72, 4.28]$. Either is accepted β€” the marks are for the four steps, so write them all out." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” attention by hand, step by step\nimport numpy as np\n\nq = np.array([0.8, 0.2])\nK = np.array([[1, 0], [0, 1], [0.4, 0.3]])\nV = np.array([[10, 0], [0, 10], [5, 5]])\nd = q.shape[-1]\n\nraw = K @ q\nprint(\"1. raw scores :\", np.round(raw, 3))\nscaled = raw / np.sqrt(d)\nprint(\"2. scaled :\", np.round(scaled, 2))\nexps = np.exp(scaled)\nprint(\"3. exponentials:\", np.round(exps, 2), \" sum =\", round(exps.sum(), 2))\nweights = exps / exps.sum()\nprint(\" softmax :\", np.round(weights, 2))\nprint(\"4. output :\", np.round(weights @ V, 2))" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## 13.3 Self-attention in code\n\n**Self-attention** means $Q$, $K$ and $V$ all come from the *same* sequence β€” the model learns how the tokens relate to each other.\n\nFor \"Alice is eating a green apple\", the token \"eating\" should attend most to **\"Alice\"** (the subject) and **\"apple\"** (the object)." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” self-attention on the course's toy sentence + attention heatmap\nimport torch, torch.nn.functional as F, matplotlib.pyplot as plt\n\ntorch.manual_seed(0)\nx = torch.tensor([[ # potential (!) embeddings of the words:\n [0.1, 0.2, 0.3, 0.4], # \"Alice\"\n [0.0, 0.1, 0.0, 0.3], # \"is\"\n [0.5, 0.6, 0.7, 0.8], # \"eating\"\n [0.0, 0.0, 0.2, 0.2], # \"a\"\n [0.1, 0.1, 0.2, 0.1], # \"green\"\n [0.6, 0.7, 0.8, 0.9], # \"apple\"\n]], dtype=torch.float)\n\nB, N, D = x.shape\nW_Q = torch.nn.Linear(D, D, bias=False) # projections; in practice they may\nW_K = torch.nn.Linear(D, D, bias=False) # project into different dimensions\nW_V = torch.nn.Linear(D, D, bias=False)\n\nQ, K, V = W_Q(x), W_K(x), W_V(x) # each (1, N, D)\n\nhead_dim = Q.shape[-1]\nraw_scores = torch.matmul(Q, K.transpose(1, 2)) / (head_dim ** 0.5) # (B, N, N)\nweights = F.softmax(raw_scores, dim=-1) # (B, N, N)\nattention = torch.matmul(weights, V) # (B, N, D)\n\nprint(\"Raw scores: \", raw_scores[0, 2])\nprint(\"Weights: \", weights[0, 2])\nprint(\"Attention: \", attention[0, 2]) # contextualised representation of \"eating\"\n\ntokens = [\"Alice\", \"is\", \"eating\", \"a\", \"green\", \"apple\"]\nplt.figure()\nplt.imshow(weights[0].detach().numpy(), cmap=\"viridis\")\nplt.xticks(range(len(tokens)), tokens, rotation=45)\nplt.yticks(range(len(tokens)), tokens)\nplt.colorbar(); plt.title(\"Attention Heatmap\")\nplt.xlabel(\"Key (who I look at)\"); plt.ylabel(\"Query (who is looking)\")\nplt.show()" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "### Reading that heatmap β€” the trap\n\nAll weights come out near $0.16$–$0.17$, i.e. $\\approx 1/6$: a **uniform** distribution. The projection matrices are **randomly initialised and untrained**, so no meaningful relationships exist yet. Whatever token appears brightest is noise.\n\nAfter training you would expect \"eating\" to attend to \"apple\" and \"Alice\". `attention[0, 2]` is a contextualised embedding of \"eating\" combining information from all tokens weighted by relevance β€” **currently meaningless due to random weights.**\n\n> πŸ“Œ This is a classic exam trap: you are asked whether the heatmap matches your prediction. The correct answer is *\"no, and here is why: the model is untrained.\"*\n\n## 13.4 Cross-attention\n\nIn self-attention, $Q$, $K$, $V$ come from the same sequence. In **cross-attention**, the queries come from one modality and the keys and values from another.\n\nFor text-guided image generation: the **image** representation produces the queries, the **text** encoder produces the keys and values. This lets the image generation process \"look up\" relevant information from the text embeddings β€” the attention scores decide which words matter when generating which parts of the image.\n\n## 13.5 Building blocks in PyTorch\n\n| Layer | What it gives you |\n|---|---|\n| `nn.MultiheadAttention(embed_dim, num_heads, batch_first=True)` | attention alone; call as `attn(q, k, v)` β†’ `(output, weights)` |\n| `nn.TransformerEncoderLayer(d_model, nhead, dim_feedforward, batch_first=True)` | self-attention + feedforward + norms + residuals, all in one |\n\nA transformer block contains, in order: **LayerNorm β†’ Multi-Head Self-Attention β†’ residual β†’ LayerNorm β†’ Feedforward (MLP) β†’ residual**." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” MultiheadAttention and a full TransformerEncoderLayer\nimport torch, torch.nn as nn\n\nx = torch.randn(2, 6, 32) # (batch, tokens, features)\n\nmha = nn.MultiheadAttention(embed_dim=32, num_heads=4, batch_first=True)\nout, attn_weights = mha(x, x, x) # Q = K = V -> self-attention\nprint(\"MHA output :\", tuple(out.shape), \"| weights:\", tuple(attn_weights.shape))\n\nenc = nn.TransformerEncoderLayer(d_model=32, nhead=4, dim_feedforward=16, batch_first=True)\nprint(\"Encoder out:\", tuple(enc(x).shape))\n\n# residual connection: reuse the original output alongside the attended one\nprint(\"residual :\", tuple((out + x).shape))" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "✨ **The single most important idea in this part of the course:** *Transformers are sequence models, not language models.* Any data that can be turned into a sequence of vectors β€” words, image patches, points in a cloud β€” can be fed to a transformer." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 14. Vision Transformers & DINOv2\n\n[↑ TOC](#toc)\n\n## 14.1 From pixels to tokens\n\nA ViT applies the transformer architecture to images. Language splits sentences into word tokens; a ViT splits the image into square **patches**, and each patch becomes one token.\n\n$$\\text{number of patches} = \\left(\\frac{\\text{image\\_size}}{\\text{patch\\_size}}\\right)^2$$\n\nA $112\\times112$ image with $14\\times14$ patches gives $(112/14)^2 = 8^2 = $ **64 patches**.\n\n| Patch size | Effect |\n|---|---|\n| **Smaller** | preserves more detail Β· longer sequences Β· **more computation** (self-attention compares every token with every other) |\n| **Larger** | less computation Β· **loses fine detail and small features** |\n\n## 14.2 The five-stage pipeline\n\n1. **Split** the image into patches\n2. **Embed** each patch into a vector\n3. **Prepend** the CLS token\n4. **Add** positional embeddings\n5. **Process** with transformer blocks, then **classify** from the CLS output\n\n**Patch embedding.** A $14\\times14$ RGB patch flattened is $14\\cdot14\\cdot3 = 588$ values. A linear layer turns that into a learned feature vector β€” exactly like a word embedding. In practice a `nn.Conv2d(3, embed_dim, kernel_size=patch_size, stride=patch_size)` does the splitting *and* the embedding in one operation, because kernel = stride = patch size produces non-overlapping patches.\n\n**CLS token.** A single learnable vector prepended to the sequence. Through self-attention it gathers information from every patch, and its final representation is used for classification.\n\n**Positional embeddings.** Transformers process all tokens simultaneously and have no built-in notion of space β€” without positional embeddings, shuffled patches would look identical. CNNs get spatial structure for free from the convolution operation; transformers must be told." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” the ViT pipeline, block by block\nimport torch, torch.nn as nn, torch.nn.functional as F\n\n# ---- 1) patches by hand, with unfold ---------------------------------------\nimage = torch.randn(3, 112, 112)\npatch_size = 14\npatches = image.unfold(1, patch_size, patch_size).unfold(2, patch_size, patch_size)\nprint(\"unfolded :\", tuple(patches.shape)) # (3, 8, 8, 14, 14)\npatches = patches.permute(1, 2, 0, 3, 4).reshape(-1, 3, patch_size, patch_size)\nprint(\"flattened sequence:\", tuple(patches.shape), \"-> 64 tokens\")\n\n\n# ---- 2) the transformer block ----------------------------------------------\nclass ViTBlock(nn.Module):\n \"\"\"LayerNorm -> Multi-Head Self-Attention -> residual\n LayerNorm -> Feedforward (MLP) -> residual\"\"\"\n\n def __init__(self, embedding_dimension, feedforward_dimension,\n num_attention_heads, dropout_probability):\n super().__init__()\n self.normalization_before_attention = nn.LayerNorm(embedding_dimension)\n self.self_attention = nn.MultiheadAttention(\n embedding_dimension, num_attention_heads, dropout_probability, batch_first=True)\n self.normalization_before_mlp = nn.LayerNorm(embedding_dimension)\n self.feedforward_network = nn.Sequential(\n nn.Linear(embedding_dimension, feedforward_dimension),\n nn.GELU(),\n nn.Linear(feedforward_dimension, embedding_dimension),\n )\n\n def forward(self, tokens):\n normalized = self.normalization_before_attention(tokens)\n attention_output, attention_weights = self.self_attention(\n normalized, normalized, normalized) # Q = K = V\n tokens = tokens + attention_output # residual on UNnormalised tokens\n normalized = self.normalization_before_mlp(tokens)\n tokens = tokens + self.feedforward_network(normalized)\n return tokens, attention_weights\n\n\n# ---- 3) the full backbone ---------------------------------------------------\nclass ViT(nn.Module):\n def __init__(self, patch_size, num_transformer_layers, embedding_dimension,\n num_attention_heads, dropout_probability, maximum_num_tokens=500):\n super().__init__()\n self.patch_size = patch_size\n self.embedding_dimension = embedding_dimension\n\n # kernel = stride = patch size -> NON-overlapping patches, embedded in one op\n self.patch_embedding = nn.Conv2d(3, embedding_dimension,\n kernel_size=patch_size, stride=patch_size, padding=0)\n self.classification_token = nn.Parameter(\n torch.randn(1, 1, embedding_dimension), requires_grad=True)\n self.position_embeddings = nn.Parameter(\n torch.randn(1, maximum_num_tokens, embedding_dimension), requires_grad=True)\n\n self.transformer_blocks = nn.ModuleList([\n ViTBlock(embedding_dimension, 4 * embedding_dimension,\n num_attention_heads, dropout_probability)\n for _ in range(num_transformer_layers)\n ])\n\n def forward(self, images):\n batch_size = images.size(0)\n num_patches_per_axis = images.size(2) // self.patch_size\n\n tokens = self.patch_embedding(images) # (B, E, H', W')\n tokens = tokens.permute(0, 2, 3, 1) # (B, H', W', E)\n tokens = tokens.reshape(batch_size, -1, self.embedding_dimension) # (B, N, E)\n\n cls = self.classification_token.expand(batch_size, -1, -1)\n tokens = torch.cat([cls, tokens], dim=1) # prepend CLS\n tokens = tokens + self.position_embeddings[:, :tokens.size(1)]\n\n for block in self.transformer_blocks:\n tokens, attention_weights = block(tokens)\n\n classification_features = tokens[:, 0] # the CLS output\n spatial_tokens = tokens[:, 1:]\n spatial_feature_map = (spatial_tokens\n .reshape(batch_size, num_patches_per_axis,\n num_patches_per_axis, self.embedding_dimension)\n .permute(0, 3, 1, 2))\n return classification_features, spatial_tokens, spatial_feature_map\n\n\nclass Model(nn.Module):\n \"\"\"Backbone + classification head on the CLS representation.\"\"\"\n\n def __init__(self, backbone, num_classes):\n super().__init__()\n self.backbone = backbone\n self.classifier = nn.Sequential(\n nn.LayerNorm(self.backbone.embedding_dimension),\n nn.Linear(self.backbone.embedding_dimension, num_classes),\n )\n\n def forward(self, images):\n cls_features, spatial_tokens, spatial_map = self.backbone(images)\n logits = self.classifier(cls_features)\n return logits, cls_features, spatial_tokens, spatial_map\n\n\nvit = ViT(patch_size=14, num_transformer_layers=2, embedding_dimension=64,\n num_attention_heads=8, dropout_probability=0.1)\nmodel = Model(backbone=vit, num_classes=10)\nlogits, cls, tok, smap = model(torch.randn(2, 3, 112, 112))\nprint(\"\\nlogits\", tuple(logits.shape), \"| cls\", tuple(cls.shape),\n \"| tokens\", tuple(tok.shape), \"| map\", tuple(smap.shape))" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# reference β€” ViT training loop with AdamW + cosine annealing # needs download\nimport torch, torch.nn as nn\nfrom torch.utils.data import DataLoader, random_split\nfrom tqdm.auto import tqdm\n\n\ndef train_vit(model, dataset, num_epochs, batch_size, learning_rate,\n val_size, device=\"cpu\"):\n train_size = int(len(dataset) * (1 - val_size))\n val_size = len(dataset) - train_size\n training_set, validation_set = random_split(dataset, [train_size, val_size])\n train_loader = DataLoader(training_set, batch_size, shuffle=True)\n val_loader = DataLoader(validation_set, batch_size, shuffle=False)\n\n optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)\n scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(\n optimizer, T_max=num_epochs * len(train_loader))\n criterion = nn.CrossEntropyLoss()\n\n model.to(device)\n for _ in tqdm(range(num_epochs), desc=\"Epochs\"):\n model.train()\n for x, y in tqdm(train_loader, desc=\"Training\", leave=False):\n x, y = x.to(device), y.to(device)\n out = model(x)[0] # [0] -> the logits\n loss = criterion(out, y)\n optimizer.zero_grad(); loss.backward(); optimizer.step(); scheduler.step()\n\n with torch.no_grad():\n model.eval()\n loss = accuracy = 0\n for x, y in tqdm(val_loader, desc=\"Validation\", leave=False):\n x, y = x.to(device), y.to(device)\n out = model(x)[0]\n loss += criterion(out, y)\n accuracy += (torch.argmax(out, dim=1) == y).to(torch.float32).mean()\n print(f\"val loss {loss / len(val_loader):.4f} acc {accuracy / len(val_loader):.4f}\")\n\n\n# CIFAR10 pipeline the assignment uses\n# transform = v2.Compose([\n# v2.ToImage(),\n# v2.ToDtype(torch.float32, scale=True),\n# v2.Resize((112, 112)), # bigger -> more patches\n# v2.Normalize(mean=(0.4914, 0.4822, 0.4465), std=(0.2023, 0.1994, 0.2010)),\n# v2.RandomHorizontalFlip(p=0.5),\n# ])\n# dataset = datasets.CIFAR10(root=\"data\", train=True, transform=transform, download=True)\n# train_vit(model, dataset, num_epochs=1, batch_size=256,\n# learning_rate=0.0001, val_size=0.1, device=device)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## 14.3 ViT vs CNN\n\n| | ViT | CNN |\n|---|---|---|\n| Long-range relationships | βœ… self-attention connects any two regions directly | limited by receptive field |\n| Inductive bias | none image-specific β†’ needs lots of data | locality + translation equivariance built in |\n| Small datasets | struggles | **wins** |\n| Large datasets / pretraining | **state of the art** | plateaus earlier |\n\n*Advantages of ViTs:* they model long-range relationships between image regions through self-attention, scale very well with large datasets, and reach state-of-the-art performance when pretrained on massive data.\n*Why CNNs win on small datasets:* their inductive biases help learning when data is limited.\n\n## 14.4 Attention maps\n\nYou can visualise where the model looks by measuring the **cosine similarity between the CLS representation and each patch token**, then folding that vector back into a square grid." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” CLS-to-patch similarity map\nimport torch, torch.nn.functional as F, matplotlib.pyplot as plt\n\n\ndef create_cls_map(classification_features, spatial_tokens):\n \"\"\"CLS-to-patch similarity, reshaped into a square map.\n\n classification_features: (B, E) spatial_tokens: (B, num_patches, E)\n \"\"\"\n classification_features = F.normalize(classification_features, dim=-1)\n spatial_tokens = F.normalize(spatial_tokens, dim=-1)\n cls_map = torch.sum(classification_features.unsqueeze(1) * spatial_tokens, dim=-1)\n batch_size = cls_map.size(0)\n n = int(cls_map.size(1) ** 0.5)\n return cls_map.reshape(batch_size, n, n)\n\n\ndef to_image(image_tensor):\n \"\"\"Normalised tensor -> displayable uint8 image.\"\"\"\n image_tensor = torch.clamp(image_tensor, -1, 1)\n image_tensor = image_tensor.permute(1, 2, 0)\n image_tensor = (1 + image_tensor) * 127.5\n return image_tensor.to(torch.uint8)\n\n\nimg = torch.randn(1, 3, 112, 112)\nwith torch.no_grad():\n logits, cls_features, spatial_tokens, _ = model(img)\nattention_map = create_cls_map(cls_features, spatial_tokens)[0]\n\nfig, ax = plt.subplots(1, 2, figsize=(8, 4))\nax[0].imshow(to_image(img[0])); ax[0].set_title(\"Input\"); ax[0].axis(\"off\")\nax[1].imshow(attention_map); ax[1].set_title(\"CLS attention map\"); ax[1].axis(\"off\")\nplt.show()\nprint(\"map shape:\", tuple(attention_map.shape), \"-> 8x8 patch grid\")" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "**Reading attention maps.** Strong CLS attention to certain patches suggests the model considers those regions related or important to the prediction. Attention maps make the model interpretable: you can tell whether a wrong prediction came from an **ambiguous image** (attention spread almost uniformly over the whole frame) or from **looking at the wrong region** (attention concentrated on the background).\n\n## 14.5 DINOv2 β€” self-supervised transfer learning\n\nTraining a ViT from scratch is expensive. **DINOv2** is a ViT pretrained with **self-supervised** learning: no labels at all, learning useful visual representations from large amounts of unlabelled data. Freeze the backbone, train only the classifier.\n\nDINOv2 generalises better than a small ViT trained for one epoch because it was pretrained on massive datasets and has already learned strong, transferable visual representations." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# reference β€” DINOv2 backbone + frozen-backbone training # needs torch.hub download\nimport torch, torch.nn as nn, torch.nn.functional as F\n\n\nclass DINOv2Backbone(nn.Module):\n def __init__(self):\n super().__init__()\n self.dino = torch.hub.load(\"facebookresearch/dinov2\", \"dinov2_vits14\")\n self.embedding_dimension = self.dino.embed_dim\n\n def forward(self, images, scale=1):\n # DINOv2 uses 14x14 patches, so the image size must be divisible by 14\n resized = F.interpolate(images, size=(scale * 224, scale * 224),\n mode=\"bilinear\", align_corners=False)\n outputs = self.dino.get_intermediate_layers(\n resized, n=1, reshape=False, return_class_token=True)[0]\n patch_tokens, classification_features = outputs[0], outputs[1]\n\n batch_size = patch_tokens.size(0)\n n = int(patch_tokens.size(1) ** 0.5)\n spatial_feature_map = (patch_tokens\n .reshape(batch_size, n, n, self.embedding_dimension)\n .permute(0, 3, 1, 2))\n return classification_features, patch_tokens, spatial_feature_map\n\n\n# dinov2_backbone = DINOv2Backbone()\n# model_dino = Model(backbone=dinov2_backbone, num_classes=10)\n#\n# for parameter in model_dino.parameters(): # freeze everything\n# parameter.requires_grad = False\n# for layer_name, layer in model_dino.named_children():\n# if layer_name == \"classifier\": # unfreeze only the head\n# for parameter in layer.parameters():\n# parameter.requires_grad = True\n#\n# train_vit(model_dino, dataset, num_epochs=1, batch_size=256,\n# learning_rate=0.0001, val_size=0.1, device=device)" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# reference β€” bonus: unsupervised segmentation from DINOv2 features via PCA\nimport torch, torch.nn as nn, torch.nn.functional as F\nfrom sklearn.decomposition import PCA\n\n\n@torch.no_grad()\ndef segment_with_dinov2(images, dino_backbone, scale=1, device=torch.device(\"cpu\")):\n \"\"\"Foreground/background masks with NO segmentation labels.\n\n The first principal component of the patch features separates object from\n background - evidence that pretrained transformers learn real visual structure.\n \"\"\"\n dino_backbone.to(device).eval()\n images = images.to(device)\n\n (_, _, spatial_feature_map) = dino_backbone(images, scale=scale)\n (batch_size, num_channels, height, width) = spatial_feature_map.shape\n\n spatial_feature_map = spatial_feature_map.permute(0, 2, 3, 1)\n spatial_feature_map = spatial_feature_map.reshape(batch_size * height * width, num_channels)\n\n pca = PCA(n_components=1)\n first_pc = torch.from_numpy(pca.fit_transform(spatial_feature_map.cpu()))\n first_pc = first_pc.reshape(batch_size, height, width)\n\n return (first_pc > 0).to(torch.float32)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 15. Sets & point clouds (DeepSets)\n\n[↑ TOC](#toc)\n\nA point cloud is an **unordered set**. Shuffle the points and it is the same shape β€” so the model's output must not depend on the order. That property is **permutation invariance**, and you get it from one line:\n\n```python\nx = torch.sum(x, dim=1) # aggregate across points -> order no longer matters\n```\n\nEncode each point independently, then **sum** (or mean/max) across the point dimension, then classify. That is the DeepSets recipe.\n\n**Why attention beats RNNs/LSTMs here.** Attention lets each point interact directly with every other point regardless of position. An RNN processes data sequentially, imposing an artificial ordering and making global relationships hard to capture. For unordered spatial data, attention learns pairwise and global geometric structure β€” which is what distinguishes a sphere from a torus." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” DeepSets classifiers, with and without a transformer encoder\nimport numpy as np, torch, torch.nn as nn\nfrom torch.utils.data import Dataset, DataLoader\n\n\ndef sample_from_sphere(n=100, d=2, r=1, seed=None):\n \"\"\"Sample n points from a d-sphere in d+1 dimensions.\"\"\"\n rng = np.random.default_rng(seed)\n X = rng.standard_normal((n, d + 1))\n X = r * X / np.sqrt(np.sum(X ** 2, 1)[:, None]) # project onto the sphere\n return torch.as_tensor(X, dtype=torch.float)\n\n\ndef sample_from_torus(n, r=1.0, R=2.0, seed=None):\n \"\"\"Sample points uniformly from a torus embedded in 3D (rejection sampling).\"\"\"\n rng = np.random.default_rng(seed)\n angles = []\n while len(angles) < n:\n x = rng.uniform(0, 2 * np.pi)\n y = rng.uniform(0, 1 / np.pi)\n if y < (1.0 + (r / R) * np.cos(x)) / (2 * np.pi):\n angles.append((x, rng.uniform(0, 2 * np.pi)))\n X = [( (R + r * np.cos(t)) * np.cos(p),\n (R + r * np.cos(t)) * np.sin(p),\n r * np.sin(t) ) for t, p in angles]\n return torch.as_tensor(np.asarray(X), dtype=torch.float)\n\n\nclass SphereVersusTorus(Dataset):\n \"\"\"Point clouds sampled from spheres (label 0) and tori (label 1).\"\"\"\n\n def __init__(self, n_point_clouds=500, n_samples=100, ratio=1, shuffle=True, seed=None):\n self.n_point_clouds, self.n_samples = n_point_clouds, n_samples\n rng = np.random.default_rng(seed)\n n_spheres = int(n_point_clouds * (ratio / (ratio + 1)))\n n_tori = n_point_clouds - n_spheres\n\n spheres = torch.stack([sample_from_sphere(n_samples, r=1.0, seed=rng)\n for _ in range(n_spheres)])\n tori = torch.stack([sample_from_torus(n_samples, r=0.431, R=0.862, seed=rng)\n for _ in range(n_tori)])\n\n self.data = torch.vstack((spheres, tori))\n self.labels = torch.as_tensor([0] * n_spheres + [1] * n_tori, dtype=torch.long)\n if shuffle:\n perm = torch.randperm(self.n_point_clouds)\n self.data, self.labels = self.data[perm], self.labels[perm]\n\n def __getitem__(self, index):\n return self.data[index], self.labels[index]\n\n def __len__(self):\n return len(self.data)\n\n\nclass DeepSetClassifier(nn.Module):\n \"\"\"V1 plain Β· V2 adds BatchNorm Β· V3/V4 add LayerNorm after the sum + Dropout.\"\"\"\n\n def __init__(self, input_dim, hidden_dim, output_dim, variant=4):\n super().__init__()\n self.variant = variant\n self.norm = nn.LayerNorm(hidden_dim)\n self.encoder = nn.Sequential(\n nn.Linear(input_dim, hidden_dim // 2), nn.ReLU(),\n nn.Linear(hidden_dim // 2, hidden_dim), nn.ReLU(),\n )\n head = ([nn.BatchNorm1d(hidden_dim)] if variant == 2 else [])\n head += [nn.Linear(hidden_dim, hidden_dim // 2)]\n head += ([nn.Dropout()] if variant == 4 else [])\n head += [nn.ReLU(), nn.Linear(hidden_dim // 2, output_dim)]\n self.classifier = nn.Sequential(*head)\n\n def forward(self, x):\n x = self.encoder(x)\n x = torch.sum(x, dim=1) # <- permutation invariance\n if self.variant in (3, 4):\n x = self.norm(x)\n return self.classifier(x)\n\n\nclass DeepSetTransformerClassifier(nn.Module):\n \"\"\"Same idea, but points attend to each other before being summed.\"\"\"\n\n def __init__(self, input_dim, hidden_dim, num_classes):\n super().__init__()\n self.norm = nn.LayerNorm(hidden_dim)\n self.encoder = nn.Sequential(\n nn.Linear(input_dim, hidden_dim // 2), nn.ReLU(),\n nn.Linear(hidden_dim // 2, hidden_dim), nn.ReLU(),\n nn.TransformerEncoderLayer(d_model=hidden_dim, nhead=1,\n dim_feedforward=hidden_dim // 2, batch_first=True),\n )\n self.classifier = nn.Sequential(nn.ReLU(), nn.Linear(hidden_dim, num_classes))\n\n def forward(self, x):\n x = self.encoder(x)\n x = torch.sum(x, dim=1) # <- permutation invariance\n x = self.norm(x)\n return self.classifier(x)\n\n\ndata = SphereVersusTorus(40, 60, seed=42)\nloader = DataLoader(data, batch_size=8, shuffle=True)\nbatch, labels = next(iter(loader))\nprint(\"point-cloud batch:\", tuple(batch.shape), \"(batch, points, xyz)\")\n\nfor m in (DeepSetClassifier(3, 32, 2), DeepSetTransformerClassifier(3, 32, 2)):\n print(f\"{m.__class__.__name__:<32} -> {tuple(m(batch).shape)}\")\n\n# permutation invariance, demonstrated\nm = DeepSetClassifier(3, 32, 2).eval()\nwith torch.no_grad():\n a = m(batch)\n b = m(batch[:, torch.randperm(batch.size(1)), :]) # shuffle the points\nprint(\"\\nmax |f(x) - f(shuffled x)| =\", (a - b).abs().max().item(), \" (~0 -> invariant)\")" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "πŸ“Œ **Exam pattern.** Assignment 10 Β§2 shows the point-cloud script and asks three things: *which layer is responsible for attention* (`nn.TransformerEncoderLayer` β€” it lets each point attend to all others and compute a weighted combination of their features), *what `torch.sum(x, dim=1)` is for* (it aggregates across all points and enforces permutation invariance, because point clouds are unordered sets), and an interpretation of the loss/accuracy curves: training loss falls steadily; validation loss spikes early because parameters are still random and updates are aggressive; around epoch 8–10 both drop to near zero and accuracy jumps to ~100 %, meaning the model discovered a strongly discriminative feature; **no overfitting**, because train and validation converge together." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 16. Exam patterns & drills\n\n[↑ TOC](#toc)\n\nEverything below is reconstructed from the 2026 midterm, the mock exam and the two 2025 finals. The questions repeat with different numbers β€” the *shapes* are what to memorise." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 16.1 Debug drill A β€” six syntax errors (6 pts)\n\nYou are given a `NeuralNetworkBroken` class with exactly **six** planted mistakes and told not to change the structure. Fix each one and explain it **as a code comment**. When it is clean the next cell prints `βœ… Success!`.\n\n### The broken code (SS25 version)\n\n```python\nclass NeuralNetworkBroken(nn.Module):\n def __init__(self, input_dim, hidden_dim, output_dim):\n super().__init__()\n self.model = nn.Sequential(\n nn.Linear(input_din), # ← 2 bugs\n nn.Linear(hidden_dim, hidden_dim) # ← missing comma\n nn.Linear(output_dim, output_dim), # ← wrong input dimension\n )\n\n def forward(self, x):\n return self.model(theta) # ← undefined variable\n\ndevice = device\nmodel = NeuralNetworkBroken(input_dim=10, hidden_dim=8, output_dim=2)\nmodel.to('gpu-cluster') # ← invalid device\n```\n\n### The six bugs, every time\n\n| # | Bug | Fix | Why |\n|---|---|---|---|\n| 1 | Missing comma between layers | add `,` | `nn.Sequential` takes comma-separated arguments |\n| 2 | Typo `input_din` | `input_dim` | that name is never defined |\n| 3 | `nn.Linear` called with one argument | `nn.Linear(input_dim, hidden_dim)` | `Linear` needs `in_features` **and** `out_features` |\n| 4 | `model.to('gpu-cluster')` / `'gpu1'` | `model.to(device)` | valid devices are `'cpu'`, `'cuda'` or the `device` object |\n| 5 | `forward` returns `self.model(theta)` / `(sigmoid)` | `self.model(x)` | `x` is the argument the method receives |\n| 6 | Last layer's `in_features` wrong | `nn.Linear(hidden_dim, output_dim)` | the input of a layer must equal the output of the previous one |\n\nWatch also for a stray `;` at the end of a layer line (AS25 planted one)." + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” the FIXED network, with the six explanatory comments\nimport torch, torch.nn as nn\nfrom torch.utils.data import TensorDataset, DataLoader\n\n\nclass NeuralNetworkBroken(nn.Module):\n def __init__(self, input_dim, hidden_dim, output_dim):\n super().__init__()\n self.model = nn.Sequential(\n nn.Linear(input_dim, hidden_dim), # 2) input_din -> input_dim, 3) add hidden_dim\n nn.Linear(hidden_dim, hidden_dim), # 1) add comma, 6) correct output dimension\n nn.Linear(hidden_dim, output_dim),\n )\n\n def forward(self, x):\n return self.model(x) # 5) change from `theta`/`sigmoid` -> x\n\n\nmodel = NeuralNetworkBroken(input_dim=10, hidden_dim=8, output_dim=2)\nmodel.to(device) # 4) move to a VALID device\n\n# --- the DO NOT EDIT cell that must now run --------------------------------\nX = torch.randn(32, 10)\ny = torch.randint(0, 2, (32,))\ndataloader = DataLoader(TensorDataset(X, y), batch_size=32)\n\nloss_fn = nn.CrossEntropyLoss()\noptimizer = torch.optim.Adam(model.parameters())\n\nmodel.train()\nfor inputs, targets in dataloader:\n outputs = model(inputs.to(device))\n loss = loss_fn(outputs, targets.to(device))\n break\n\nprint('βœ… Success!')" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 16.2 Debug drill B β€” three conceptual errors (6 pts)\n\nSame idea, but the mistakes are *conceptual*: some crash, some let the code run while silently preventing learning.\n\n### The broken setup\n\n```python\nnet = NeuralNetwork(input_dim=12, hidden_dim=16, output_dim=4)\nnet.to(device)\nloss_fn = nn.BCEWithLogitsLoss() # ← bug 1\noptimizer = torch.optim.Adam(model.parameters(), lr=5e-25) # ← bugs 2 and 3\n```\n\n| # | Bug | Fix | Why |\n|---|---|---|---|\n| 1 | `BCEWithLogitsLoss` / `MSELoss` on a 4-class problem | `nn.CrossEntropyLoss()` | multi-class classification with integer labels needs cross-entropy; BCE expects float targets shaped like the output |\n| 2 | Optimizer built on `model.parameters()` | `net.parameters()` | the optimizer must own the parameters of the network actually being trained β€” otherwise nothing updates |\n| 3 | Absurd learning rate (`5e-25` or `1000`) | `lr=0.001` | far too small = no movement; far too large = divergence |" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” the FIXED training setup\nimport torch, torch.nn as nn\nfrom torch.utils.data import TensorDataset, DataLoader\n\nX = torch.randn(32, 12)\ny = torch.randint(0, 4, (32,))\ndataloader = DataLoader(TensorDataset(X, y), batch_size=16, shuffle=True)\n\n\nclass NeuralNetwork(nn.Module):\n def __init__(self, input_dim, hidden_dim, output_dim, dropout_rate=0.3):\n super().__init__()\n self.fc1 = nn.Linear(input_dim, hidden_dim)\n self.relu = nn.ReLU()\n self.dropout = nn.Dropout(dropout_rate)\n self.fc2 = nn.Linear(hidden_dim, output_dim)\n\n def forward(self, x):\n x = self.fc1(x)\n x = self.relu(x)\n x = self.dropout(x)\n return self.fc2(x)\n\n\nnet = NeuralNetwork(input_dim=12, hidden_dim=16, output_dim=4).to(device)\nloss_fn = nn.CrossEntropyLoss() # 1) multi-class -> CE\noptimizer = torch.optim.Adam(net.parameters(), lr=0.001) # 2) net, 3) sane lr\n\nnet.train()\nfor inputs, targets in dataloader:\n outputs = net(inputs.to(device))\n loss = loss_fn(outputs, targets.to(device))\n break\n\nprint('Loop runs! loss =', round(loss.item(), 4))" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "### The full bug checklist to run through mentally\n\n**Crashes**\n- missing comma / stray semicolon in `nn.Sequential`\n- `nn.Linear` with the wrong number of arguments\n- mismatched dimensions between consecutive layers\n- undefined variable in `forward`\n- invalid device string\n- missing `super().__init__()`\n- forgetting `nn.Flatten()` before the first `Linear` on image data\n- `BCELoss` with integer class labels\n\n**Silent failures**\n- optimizer built on the wrong model's parameters\n- `optimizer.zero_grad()` after `loss.backward()`\n- missing `optimizer.step()`\n- learning rate off by many orders of magnitude\n- `nn.Softmax` before `nn.CrossEntropyLoss`\n- `model.eval()` never called β†’ Dropout/BatchNorm still in training mode at evaluation\n- validation set built from the training tensors (data leakage)\n- forgetting to reinitialise the optimizer after changing the model" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 16.3 True/false theory bank\n\n20 of 64 points. Each question has 4 statements; **4 correct = 2 pts, 3 correct = 1 pt, ≀2 = 0 pts.** Guessing on a single statement can zero the whole question, so reason about each one independently.\n\n### Overfitting & underfitting\n- βœ… Overfitting = performs well on training data but poorly on new data\n- βœ… Underfitting = the model is too simple to capture patterns\n- ❌ \"High training loss and low validation loss indicates overfitting\" β€” backwards\n- βœ… Regularization like dropout reduces overfitting\n- βœ… Zero training loss β†’ the model **might** be overfitting; ❌ it does *not* mean it generalised, and ❌ the test loss is *not* guaranteed low\n\n### Universal Approximation Theorem\n- βœ… A sufficiently large single hidden layer can approximate any continuous function\n- βœ… It does **not** guarantee that learning is efficient or practical\n- ❌ It does **not** guarantee perfect generalisation on unseen data\n- βœ… Non-linear activations are necessary\n\n### Train / validation / test\n- βœ… Training set updates the parameters\n- βœ… Validation set evaluates performance during training\n- βœ… You must **never** use the test set to choose the best model\n- ❌ The test set is not for tuning hyperparameters\n- βœ… Splitting prevents overfitting and lets you assess generalisation\n\n### Cross-validation\n- βœ… More reliable performance estimate, especially on small datasets\n- βœ… Every data point is used for both training and validation, reducing evaluation instability\n- βœ… Usable for hyperparameter tuning\n- ❌ Does **not** minimise computation time\n- ❌ Does **not** remove the need for a separate test set\n- ❌ Does **not** train on the full dataset every time\n\n### Activation functions\n- βœ… ReLU, GELU, Tanh are activations Β· ❌ **MSE is a loss**\n- βœ… ReLU outputs zero for all negative inputs\n- βœ… Leaky ReLU allows a small non-zero gradient for negative inputs\n- ❌ Leaky ReLU is *not* \"ReLU shifted upward\"\n- ❌ Their outputs are *not* confined to $[0,1]$\n\n### Loss selection\n- βœ… MSE for continuous targets (test scores)\n- βœ… Categorical CE for multi-class, one class per image\n- βœ… Binary CE for two-class problems (cancer / no cancer)\n- βœ… Binary CE for thresholded regression (price above / below \\$200 000)\n\n### MLP architecture\n- βœ… Input layer, one or more hidden layers, output layer\n- βœ… Fully connected: each neuron connects to every neuron in the next layer\n- ❌ The output activation is **not** always softmax\n- ❌ An MLP **does** use activations between layers\n\n### Single-layer perceptron\n- βœ… Only reliably classifies linearly separable data\n- βœ… Converges if the data is linearly separable\n- βœ… Stops updating when no examples are misclassified\n- ❌ Does **not** always find the *best* hyperplane\n- ❌ Cannot approximate any function given enough data\n\n### Learning rate\n- βœ… Controls how much the weights update each step\n- βœ… Balances fast convergence against stable learning\n- βœ… Small = slow training; large = may overshoot the minimum\n- βœ… Gradient descent updates weights in the direction of the **negative** gradient\n- ❌ Does not change the number of layers, and guarantees nothing about accuracy\n- ❌ Gradient descent does **not** always find the global minimum\n\n### Imbalanced datasets\n- βœ… The model may be biased toward the majority class\n- βœ… Standard accuracy may be misleading\n- βœ… Augmenting the minority class may be needed\n- ❌ Not always solvable by simply adding more data\n\n### Unsupervised learning\n- βœ… Grouping customers by purchase behaviour only\n- ❌ Classifying labelled animal images Β· ❌ Spam classification Β· ❌ House-price prediction (all supervised)\n\n### Classification algorithms\n- βœ… Assign discrete class labels\n- βœ… Evaluated with precision, recall, F1\n- ❌ Do **not** primarily minimise MSE\n- ❌ Are **not** limited to two classes\n\n### PCA, autoencoders, dimensionality reduction\n- βœ… PCA finds directions of maximum variance\n- βœ… Both PCA and autoencoders reduce dimensionality\n- ❌ Autoencoders do **not** require labelled data\n- ❌ The latent dimension is **not** required to be a multiple of two\n\n### Normalisation\n- βœ… BatchNorm normalises across the batch dimension\n- βœ… LayerNorm normalises across features per sample\n- βœ… BatchNorm behaves differently in training and inference\n- βœ… LayerNorm behaves the same in both\n\n### One-hot encoding & feature types\n- βœ… Appropriate for categorical inputs with no inherent order (species)\n- ❌ Not appropriate when categories have a meaningful order (T-shirt sizes)\n- ❌ Not appropriate as a regression target\n- ❌ Four species need a one-hot of size **four**, not two\n- βœ… Height in cm = numerical Β· βœ… Name = categorical Β· βœ… Marathon placement = ordinal Β· ❌ Number of legs = numerical (a count), not ordinal\n\n### Neural networks in general\n- βœ… They learn feature representations from raw data, removing manual feature engineering\n- ❌ They are not simple or cheap to train\n- ❌ They do not always beat decision trees / SVMs\n- ❌ They are not inherently interpretable\n\n### Reading curves\n- βœ… Train loss ↓ while val loss ↑ β†’ overfitting, and regularization (Dropout, weight decay) could help\n- ❌ ...therefore *not* \"generalising well\" and *not* \"validation loss is improving\"" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 16.4 By-hand computation drills\n\nWork these without looking at the answers, then check with the code cell.\n\n1. Which of $\\mathbf{a}=(5,2)$, $\\mathbf{b}=(4,-10)$, $\\mathbf{c}=(4,-12)$, $\\mathbf{d}=(1,3)$ are orthogonal?\n2. Two 2-D vectors using only $\\pm1$ with cosine similarity $-1$; show the full computation.\n3. $f(x,y)=2x^2+y^2$, start $(2,1)$, $\\eta=0.1$: gradient, one step, both function values, interpretation.\n4. Perceptron: $x=[1.5,-2.0,3.0]$, $w=[0.7,1.2,-0.8]$, $b=-0.3$, ReLU.\n5. BCE for the 8 (label, probability) pairs in [Β§2.3](#s2-3).\n6. MSE for $y=(1,1,1,0,0,0,1,1)$, $\\hat y=(1,0,1,1,0,0,0,0)$.\n7. Confusion matrix for $y=(+1,-1,+1,-1,+1,-1)$, $\\hat y=(+1,-1,+1,+1,+1,-1)$.\n8. Shape trace: input $(1,28,28)$ β†’ Conv(8, k3, s1, p0) β†’ ReLU β†’ MaxPool(k2) β†’ Conv(16, k3, s1, p1) β†’ ReLU β†’ MaxPool(k3, s2) β†’ Flatten. What is `in_features`?\n9. Embedding: $W_e$ from [Β§12.1](#s12), compute $W_e e_{\\text{popcorn}} + W_e e_{\\text{coffee}}$ for $V=\\{\\text{coffee},\\text{tea},\\text{popcorn},\\text{waffle}\\}$, $W_e = \\begin{bmatrix}1&2&0&3\\\\4&1&2&0\\\\0&5&1&1\\end{bmatrix}$.\n10. Attention: $q=[0.8,0.2]$ with the three tutor keys/values from [Β§13](#s13).\n11. RNN step: the $h_t$ computation from [Β§12.2](#s12).\n12. How many $14\\times14$ patches from a $112\\times112$ image? And from a $224\\times224$ image?" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": "# runnable β€” answers to all twelve drills\nimport numpy as np\nfrom math import log\n\nprint(\"1. aΒ·b =\", np.dot([5,2],[4,-10]), \"-> a βŸ‚ b\")\nprint(\"2. u=(1,1), v=(-1,-1): dot=-2, |u|=|v|=√2, cos = -2/2 = -1\")\n\nx0 = np.array([2., 1.]); g = np.array([4 * 2., 2 * 1.]); x1 = x0 - 0.1 * g\nf = lambda p: 2 * p[0] ** 2 + p[1] ** 2\nprint(f\"3. βˆ‡f(2,1)=({g[0]:.0f}, {g[1]:.0f}) step->({x1[0]:.1f}, {x1[1]:.1f})\"\n f\" f: {f(x0):.0f} -> {f(x1):.2f} (decreases)\")\n\nz = sum(w*xi for w, xi in zip([0.7,1.2,-0.8], [1.5,-2.0,3.0])) - 0.3\nprint(f\"4. z = {z:.2f} ReLU(z) = {max(0.0, z):.2f}\")\n\npairs = [(1,0.9),(1,0.7),(1,0.4),(0,0.3),(0,0.1),(0,0.2),(1,0.8),(0,0.6)]\nprint(f\"5. BCE = {np.mean([-(log(p) if t else log(1-p)) for t,p in pairs]):.2f}\")\n\nyv = np.array([1,1,1,0,0,0,1,1]); yh = np.array([1,0,1,1,0,0,0,0])\nprint(f\"6. MSE = {np.mean((yv-yh)**2)} = {int(((yv-yh)**2).sum())}/8\")\n\ny = np.array([1,-1,1,-1,1,-1]); p = np.array([1,-1,1,1,1,-1])\nprint(f\"7. TP={int(((y==1)&(p==1)).sum())} FP={int(((y==-1)&(p==1)).sum())} \"\n f\"TN={int(((y==-1)&(p==-1)).sum())} FN={int(((y==1)&(p==-1)).sum())}\")\n\nprint(\"8. 26 -> 13 -> 13 -> 6 ; in_features = 16*6*6 =\", 16*6*6)\n\nW_e = np.array([[1,2,0,3],[4,1,2,0],[0,5,1,1]]); V = [\"coffee\",\"tea\",\"popcorn\",\"waffle\"]\nemb = lambda w: W_e[:, V.index(w)]\nprint(\"9. popcorn+coffee =\", emb(\"popcorn\") + emb(\"coffee\"))\n\nq = np.array([0.8,0.2]); K = np.array([[1,0],[0,1],[0.4,0.3]]); Vv = np.array([[10,0],[0,10],[5,5]])\nw = np.exp((K@q)/np.sqrt(2)); w = w/w.sum()\nprint(\"10. weights =\", np.round(w,2), \" output =\", np.round(w@Vv, 2))\n\nh = np.tanh(np.array([[0.5,-0.2],[0.1,0.4]])@np.array([0.2,-0.1])\n + np.array([[0.3,0.8],[-0.5,0.2]])@np.array([1,0]) + np.array([0.1,-0.2]))\nprint(\"11. h_t =\", np.round(h, 2))\n\nprint(\"12. (112/14)Β² =\", (112//14)**2, \" | (224/14)Β² =\", (224//14)**2)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n## 16.5 Exam-day checklist\n\n**Before you start**\n- Turn off every AI feature; only the notebook and a PDF viewer open.\n- Download the notebook in the 5-minute window. Save often (`Ctrl+S`).\n- Have this summary, the assignment solutions and your cheat sheet open as **local PDFs**.\n\n**While answering**\n- Read the instruction literally. Wrong dataset, wrong variable name, wrong output format = 0.\n- Markdown cells for text (**in English**), code cells for code, directly below each question.\n- **Show every step in $\\LaTeX$** on Hands-On questions.\n- Name variables exactly as instructed (`history_basic`, `history_aug`, `history_aug2`, `stu_model`, …).\n- Don't rename or delete provided cells. Don't hard-code results.\n- Use `help(nn.Flatten)` or `nn.Flatten?` when you forget an argument β€” documentation is allowed, search engines are not.\n- If a model won't work, use the **backup model** and keep going: the training/plotting/evaluation points are still available.\n\n**Before you submit**\n- **Restart Session & Run All.** Code that does not run costs 50 % of that question.\n- Check every printed output matches the requested format (`Validation loss: x.xx, validation accuracy: x.xx`).\n- Fill in your name at the top.\n- Upload within the 5-minute window after the end." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 17. Quick reference sheets\n\n[↑ TOC](#toc)\n\n## 17.1 Formulas\n\n| Concept | Formula |\n|---|---|\n| Euclidean norm | $\\parallel\\mathbf{v}\\parallel = \\sqrt{\\sum_i v_i^2}$ |\n| Cosine similarity | $\\dfrac{\\mathbf{u}\\cdot\\mathbf{v}}{\\parallel\\mathbf{u}\\parallel\\parallel\\mathbf{v}\\parallel}$ |\n| Projection | $\\dfrac{\\mathbf{a}\\cdot\\mathbf{b}}{\\mathbf{b}\\cdot\\mathbf{b}}\\mathbf{b}$ |\n| Gradient descent step | $\\mathbf{x} \\leftarrow \\mathbf{x} - \\eta\\nabla f(\\mathbf{x})$ |\n| Chain rule | $\\frac{df}{dt} = \\sum_i \\frac{\\partial f}{\\partial x_i}\\frac{dx_i}{dt}$ |\n| MSE | $\\frac1n\\sum(y_i-\\hat y_i)^2$ |\n| BCE | $-\\frac1n\\sum[y\\log p + (1-y)\\log(1-p)]$ |\n| Neuron | $z = \\sum_i w_i x_i + b$ |\n| Conv output | $\\lfloor\\frac{W+2P-K}{S}\\rfloor+1$ |\n| Pool output | $\\lfloor\\frac{W-K}{S}\\rfloor+1$ |\n| Precision / Recall | $\\frac{TP}{TP+FP}$ / $\\frac{TP}{TP+FN}$ |\n| F1 | $2\\frac{P\\cdot R}{P+R}$ |\n| Softmax | $\\frac{e^{s_i}}{\\sum_j e^{s_j}}$ |\n| Attention | $\\text{softmax}\\!\\left(\\frac{QK^\\top}{\\sqrt d}\\right)V$ |\n| RNN step | $h_t = \\tanh(W_{ih}x_t + W_{hh}h_{t-1} + b_h)$ |\n| VAE sampling | $z = \\mu + \\varepsilon\\sigma$, $\\sigma = e^{\\frac12\\log\\sigma^2}$ |\n| VAE KL term | $-\\frac12\\sum(1+\\log\\sigma^2-\\mu^2-\\sigma^2)$ |\n| ViT patches | $(\\text{image\\_size}/\\text{patch\\_size})^2$ |\n\n## 17.2 PyTorch API\n\n| Task | Code |\n|---|---|\n| Device | `device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")` |\n| Move | `model.to(device)`, `x.to(device)` |\n| Train / eval mode | `model.train()` / `model.eval()` |\n| No gradients | `with torch.no_grad():` |\n| Predicted class | `output.argmax(dim=1)` or `torch.max(output, 1)` |\n| Flatten in a model | `nn.Flatten()` |\n| Flatten manually | `x.view(x.shape[0], -1)` |\n| Infer input size | `nn.LazyLinear(out_features)` |\n| Pass-through layer | `nn.Identity()` |\n| Save / load | `torch.save(model.state_dict(), 'f.pth')` / `model.load_state_dict(...)` |\n| Deep copy weights | `copy.deepcopy(model.state_dict())` |\n| Freeze | `for p in model.parameters(): p.requires_grad = False` |\n| Inspect trainable | `for n, p in model.named_parameters(): print(n, p.requires_grad)` |\n| Top-level blocks | `model.named_children()` |\n| LR scheduler | `torch.optim.lr_scheduler.ReduceLROnPlateau(opt, mode='min', factor=0.5, patience=4)` |\n| Cosine schedule | `torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=...)` |\n| Pad sequences | `pad_sequence(seqs, batch_first=True, padding_value=pad_id)` |\n| Custom batching | `DataLoader(..., collate_fn=lambda b: collate_sequences(b, pad_id))` |\n\n## 17.3 Defaults worth memorising\n\n| Setting | Value |\n|---|---|\n| `Adam` learning rate | `0.001` |\n| `SGD` learning rate | `0.01` |\n| Batch size | 32 / 64 / 128 |\n| `MaxPool2d` stride | defaults to `kernel_size` |\n| Normalize to $[-1,1]$ | `mean=(0.5,), std=(0.5,)` |\n| MNIST statistics | `mean=(0.1307,), std=(0.3081,)` |\n| CIFAR10 statistics | `mean=(0.4914,0.4822,0.4465), std=(0.2023,0.1994,0.2010)` |\n| MNIST / FashionMNIST | 28Γ—28, 10 classes, 60 000 / 10 000 |\n| USPS | 16Γ—16 β†’ 256 inputs, 10 classes |\n| CIFAR10 | 32Γ—32 RGB, 10 classes |\n| Titanic after one-hot | `input_dim = 14`, `output_dim = 2` |\n| Dropout | 0.3 – 0.5 |\n| Weight decay | `1e-4` |\n| Early-stopping patience | 10, `min_delta=0.001` |\n\n## 17.4 Datasets used in the course\n\n| Dataset | Shape | Classes | Where |\n|---|---|---|---|\n| Iris | 4 features | 3 (or 2 binarised) | A3 |\n| FashionMNIST | 1Γ—28Γ—28 | 10 | A4 |\n| Titanic (CSV) | 14 features after encoding | 2 | A5 |\n| MNIST | 1Γ—28Γ—28 | 10 | A6, A7, A8 |\n| dogs vs cats | 3Γ—100Γ—100 | 2 | A8.2 |\n| lyrics.txt / shakespeare.txt | word sequences | vocab-sized | A9 |\n| Sphere vs Torus | (N points, 3) | 2 | A10 |\n| CIFAR10 | 3Γ—112Γ—112 (resized) | 10 | A11 |\n| USPS | 1Γ—16Γ—16 | 10 | mock exam, both finals |\n| blobs_data.csv | 200Γ—4 | 4 clusters | mock exam |" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "\n\n---\n\n# 18. Source map\n\n[↑ TOC](#toc)\n\nEvery section of this notebook traces back to specific files in the course folder. The companion file **`ML_Course_Source_Map.md`**, saved next to this notebook, gives the full two-way mapping:\n\n- **source file β†’ sections it feeds**, with the topics each contributed\n- **section β†’ source files**, so you can jump from a summary back to the original solution\n- a coverage table confirming all 17 notebooks, 7 scripts and 2 data files were read\n\nOpen it alongside this notebook when you want the original worked solution for something you see here.\n\n---\n\n*Built from the contents of the `MachineLearning` folder. Every assignment solution, exam and script in that folder is represented above.*" + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.11", + "mimetype": "text/x-python", + "file_extension": ".py", + "pygments_lexer": "ipython3", + "nbconvert_exporter": "python", + "codemirror_mode": { + "name": "ipython", + "version": 3 + } + }, + "colab": { + "provenance": [], + "toc_visible": true + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/Midterm 2026_solution.ipynb b/Midterm 2026_solution.ipynb new file mode 100644 index 0000000..c349f2f --- /dev/null +++ b/Midterm 2026_solution.ipynb @@ -0,0 +1 @@ +{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[],"gpuType":"T4"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"1cf71a12"},"source":["**FILL IN YOUR NAME HERE**: [LASTNAME] [FIRSTNAME]"]},{"cell_type":"markdown","metadata":{"id":"65323689"},"source":["# Machine Learning Midterm Exam\n","Spring 2026 - 30 March 2026"]},{"cell_type":"markdown","metadata":{"id":"7207e6ad"},"source":["---\n","\n","## Exam Rules and Conduct\n","\n","- You must use **Colab** with **all AI assistance turned off**\n","\n","### Allowed\n","- Built-in IDE documentation\n","- Standard autocomplete (e.g., showing available functions when typing `torch.nn`)\n","- Use of `help()` and `dir()` functions\n","- Locally saved files (including course materials, assignments, PDFs, summaries, books, etc.)\n","- PDF editor/viewer\n","- A locally installed dictionary app (English language with no translations)\n","\n","### Not Allowed\n","- **No AI coding assistance**: no chatbots/LLMs, code generation, or explanation tools\n","- **No external help**: no internet, communication apps, translation apps, or additional browser tabs\n","\n","---\n","\n","## Timing\n","\n","- You have **5 minutes before** the start to download the file and **5 minutes after** the end to upload it\n","- **Late submissions** will not be accepted without a documented reason\n","\n","---\n","\n","## Answer Format\n","\n","- Answer directly below each question\n"," - Use **Markdown cells** for text (in English)\n"," - Use **code cells** for programming\n","- **0 points** for incorrect or incomplete answers\n"," - Partial credit only if the core idea is correct **and** instructions are followed\n"," - Ignoring instructions = **0 points**, regardless of effort\n","- Be clear and conciseβ€”extra detail doesn’t earn more points\n","\n","---\n","\n","## Coding Guidelines\n","\n","- Use **only packages covered in the course and assignments**\n","- Follow dataset instructions. Using a different dataset than instructed will result in **0 points** for that question\n","- Code that **does not run** = **50% deduction**\n","- Code quality (correctness, clarity, efficiency) counts\n","\n","---\n","\n","## Academic Integrity\n","\n","Violations (e.g., using AI tools or outside help) will result in:\n","- A **failing grade (1.0)**\n","- A **disciplinary report** to the Faculty of Science and Medicine\n","\n","---\n","\n","## **Good luck! πŸ€**\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"b952307b"},"outputs":[],"source":["points = 64"]},{"cell_type":"markdown","source":["# 1. Theory (20 points)\n","Complete the quiz provided on Moodle.\n","\n","For each topic, identify which statements are TRUE βœ… and which are FALSE ❌. Each question may have 1, 2, 3, or 4 correct answers.\n","\n","- 4 correct answers: 2 points\n","- 3 correct answers: 1 point\n","- 2 or fewer correct answers: 0 points"],"metadata":{"id":"rzBG4UGk8Nq1"}},{"cell_type":"markdown","source":["---\n","# 2. Hands-On (20 points)\n","You may use Python as a calculator but all calculations must be shown in $\\LaTeX$"],"metadata":{"id":"m03ByxIZ8ff3"}},{"cell_type":"markdown","source":["## 2.1 **(2pts)** Which pair of vectors are orthogonal/perpendicular?\n","\n","Let:\n","\n","$\n","\\mathbf{a}=\n","\\begin{pmatrix}\n","5\\\\\n","2\n","\\end{pmatrix},\n","\\quad\n","\\mathbf{b}=\n","\\begin{pmatrix}\n","4\\\\\n","-10\n","\\end{pmatrix},\n","\\quad\n","\\mathbf{c}=\n","\\begin{pmatrix}\n","4\\\\\n","-12\n","\\end{pmatrix},\n","\\quad\n","\\mathbf{d}=\n","\\begin{pmatrix}\n","1\\\\\n","3\n","\\end{pmatrix}\n","$"],"metadata":{"id":"V4nn34V09shq"}},{"cell_type":"markdown","source":["2.1.1 **(1pt)** **Perpendicular vectors:** $\\mathbf{a}$ & $\\mathbf{b}$"],"metadata":{"id":"RulRavZXEl_u"}},{"cell_type":"markdown","source":["2.1.2 **(1pt)** **Calculation with $\\LaTeX$**:\n","\n","$\n","\\begin{pmatrix}\n","5\\\\\n","2\n","\\end{pmatrix}\n","\\cdot\n","\\begin{pmatrix}\n","4\\\\\n","-10\n","\\end{pmatrix} = 20 - 20 = 0\n","$"],"metadata":{"id":"VF3VufMbRjiB"}},{"cell_type":"markdown","source":["## 2.2 **(5pts)** Give an example of two 2D vectors whose cosine similarity is `-1`.\n","- Each vector should contain only the values `1` and `-1`.\n","- Then, show your work by computing the dot product, the magnitude of each vector, and the cosine similarity to verify that the result is `-1`."],"metadata":{"id":"HQR1a5rALeRv"}},{"cell_type":"markdown","source":["2.2.1 **(1pt)** Two vectors:\n","\n","$\n","\\mathbf{u}=\n","\\begin{pmatrix}\n","1\\\\\n","1\n","\\end{pmatrix},\n","\\quad\n","\\mathbf{v}=\n","\\begin{pmatrix}\n","-1\\\\\n","-1\n","\\end{pmatrix}\n","$"],"metadata":{"id":"zPzReiRjLyhE"}},{"cell_type":"markdown","source":["2.2.2 **(4pt)** Cosine Similarity Calculation with $\\LaTeX$:\n","\n","$\n","\\begin{aligned}\n","\\mathbf{u} \\cdot \\mathbf{v} &=\n","\\begin{pmatrix}\n","1 \\\\\n","1\n","\\end{pmatrix} \\cdot\n","\\begin{pmatrix}\n","-1 \\\\\n","-1\n","\\end{pmatrix}\n","= 1 \\cdot -1 + 1 \\cdot -1 = -1 -1 = -2\n","\\end{aligned}\n","$\n","\n","$\\begin{aligned}\n","\\parallel \\mathbf{u} \\parallel &= \\sqrt{1^2 + 1^2} = \\sqrt{2}\n","\\\\\n","\\parallel \\mathbf{v} \\parallel &= \\sqrt{(-1)^2 + (-1)^2} = \\sqrt{2}\n","\\end{aligned}\n","$\n","\n","$\\cos(\\theta) = \\frac{-2}{\\sqrt{2} \\cdot \\sqrt{2}} = \\frac{-2}{2} = -1\n","$"],"metadata":{"id":"TaqIRv18RgyF"}},{"cell_type":"markdown","source":["## 2.3 **(6pts)** Compute one step of gradient descent.\n","\n","Let: $f(x,y)=2x^2+y^2$\n","\n","Suppose we start at $(x_0, y_0) = (2, 1)$ with learning rate $\\eta = 0.1$"],"metadata":{"id":"cKmj8X7SQW01"}},{"cell_type":"markdown","source":["2.3.1. **(1pt)** Determine the partial derivatives\n","\n","Partial derivatives: $\\frac{\\partial f}{\\partial x} = 4x, \\quad \\frac{\\partial f}{\\partial y} = 2y$"],"metadata":{"id":"QWpoB-I7RJV8"}},{"cell_type":"markdown","source":["2.3.2 **(1pt)** Write down the gradient at $(2, 1)$\n","\n","So, $\\nabla f(2,1) =\n","\\begin{pmatrix}\n","4 \\cdot 2 \\\\\n","2 \\cdot 1\n","\\end{pmatrix} =\n","\\begin{pmatrix}\n","8 \\\\\n","2\n","\\end{pmatrix}\n","$\n"],"metadata":{"id":"k9-5VKhuSxEu"}},{"cell_type":"markdown","source":["2.3.3 **(1pt)** Compute one step of gradient descent\n","\n","$(x_1,y_1) = (2,1) - 0.1 \\cdot (8,2) = (1.2,0.8)$\n"],"metadata":{"id":"PARwKrM1R7m2"}},{"cell_type":"markdown","source":["2.3.4 **(2pts)** Compute the function value before and after the step (plug the original and first step $(x,y)$ into the function)\n","\n","$f(2,1) = 2(2)^2 + (1)^2 = 8 + 1 = 9$\n","\n","$f(1.2,0.8) = 2(1.2)^2 + (0.8)^2 = 2.88 + (0.64) = 3.52$"],"metadata":{"id":"RAgdGg9RR8lt"}},{"cell_type":"markdown","source":["2.3.5 **(1pt)** Interpret the change in function values by reporting the initial value, the new value after one gradient descent step, and the direction of the change\n","\n","The function value decreases from $9$ to $3.52$, meaning the step moved downhill"],"metadata":{"id":"-G7bEkbcR929"}},{"cell_type":"markdown","metadata":{"id":"9c2026b2"},"source":["## 2.4 **(6pts)** Activate this single-layer Perceptron with a LeakyReLU activation function.\n","You are working with a single-layer perceptron that has:\n","- 3 input units (features)\n","- 1 output unit (neuron)\n","\n","These are your parameters:\n","- $x=[2.0, -2.0, 1.0]$\n","- $w=[1.0, -1.0, 1.0]$\n","- $b=-2$\n","- $\\alpha$ = 0.01\n"]},{"cell_type":"markdown","metadata":{"id":"744c9753"},"source":["2.4.1 **(2pts)** Write the formula for the **weighted sum** of inputs.\n","\n","$z = \\sum_{i=1}^n w_ix_i + b$"]},{"cell_type":"markdown","metadata":{"id":"8b178fd1"},"source":["2.4.2 **(2pts)** Show your calculations and give the final answer rounded to 1 decimal point.\n","\n","$\n","\\begin{aligned}\n","z &= (1*2) + (-1*-2) + (1*1) - 2 \\\\\n","z &= 2 + 2 + 1 -2 \\\\\n","z &= 3.0\n","\\end{aligned}$"]},{"cell_type":"markdown","metadata":{"id":"d867c06e"},"source":["2.4.3 **(2pts)** Apply the **LeakyReLU** activation function to determine the final output of the perceptron.\n","\n","Expected answer format $\\sigma(x.x) = x.x$\n","\n","$\\sigma(3.0) = 3.0$"]},{"cell_type":"markdown","source":["# 3. Coding (24 points)"],"metadata":{"id":"wc9-2sKe8wJl"}},{"cell_type":"code","execution_count":1,"metadata":{"id":"B2DEJ7KInHgV","executionInfo":{"status":"ok","timestamp":1774774775167,"user_tz":-120,"elapsed":13031,"user":{"displayName":"Olivia Lecomte","userId":"07830699269148802057"}}},"outputs":[],"source":["### DO NOT EDIT THIS CELL ###\n","\n","import torch\n","import torch.nn as nn\n","import torch.optim as optim\n","import torchvision.transforms as transforms\n","import torchvision\n","from torch.utils.data import Subset, DataLoader, TensorDataset\n","\n","import numpy as np\n","import random\n","import matplotlib.pyplot as plt\n","from tqdm import tqdm\n","\n","\n","# Set random seed\n","def set_seed(seed):\n"," random.seed(seed)\n"," np.random.seed(seed)\n"," torch.manual_seed(seed)\n"," torch.cuda.manual_seed(seed)\n"," torch.cuda.manual_seed_all(seed)\n"," torch.backends.cudnn.deterministic = True\n"," torch.backends.cudnn.benchmark = False\n"," torch.use_deterministic_algorithms(True)\n","\n","set_seed(0)\n","\n","g = torch.Generator().manual_seed(42)\n","\n","device = torch.device(\"cpu\")\n","\n","#############################"]},{"cell_type":"markdown","metadata":{"id":"8cf4182b"},"source":["## 3.1 **(6pts)** Debug the following Neural Network initialization.\n","The following code defines and trains a simple neural network with 3 linear layers, an `__init__` and `forward` method. However, **it contains six syntax errors** that cause the code to crash during training. There are no structural problems so please **do not edit the general structure**.\n","\n","Your task is to:\n","\n","- Identify all 6 mistakes\n","- Fix each one and briefly explain why the change was necessary as a code comment\n","\n","Once the script is error-free, the script will print `βœ… Success!`."]},{"cell_type":"code","execution_count":22,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":30,"status":"ok","timestamp":1774775142452,"user":{"displayName":"Olivia Lecomte","userId":"07830699269148802057"},"user_tz":-120},"id":"ctFJSqRUTwjW","outputId":"1f05731d-9565-48fe-85f1-d032be495767"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["NeuralNetworkBroken(\n"," (model): Sequential(\n"," (0): Linear(in_features=10, out_features=8, bias=True)\n"," (1): Linear(in_features=8, out_features=8, bias=True)\n"," (2): Linear(in_features=8, out_features=2, bias=True)\n"," )\n",")"]},"metadata":{},"execution_count":22}],"source":["class NeuralNetworkBroken(nn.Module):\n"," def __init__(self, input_dim, hidden_dim, output_dim):\n"," super().__init__()\n"," self.model = nn.Sequential(\n"," nn.Linear(input_dim, hidden_dim), # 2) input_din -> input_dim, 3) add hidden_dim\n"," nn.Linear(hidden_dim, hidden_dim), # 1) add comma, 6) correct output dimension\n"," nn.Linear(hidden_dim, output_dim),\n"," )\n","\n"," def forward(self, x):\n"," return self.model(x) # 5) change from sigmoid -> x\n","\n","device = device\n","model = NeuralNetworkBroken(input_dim=10, hidden_dim=8, output_dim=2)\n","model.to(device) # 4) move to device"]},{"cell_type":"code","execution_count":23,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":37,"status":"ok","timestamp":1774775143726,"user":{"displayName":"Olivia Lecomte","userId":"07830699269148802057"},"user_tz":-120},"id":"9e49b08a","outputId":"38acf34f-227d-467d-f71c-6dc0c72efa1c"},"outputs":[{"output_type":"stream","name":"stdout","text":["βœ… Success!\n"]}],"source":["### DO NOT EDIT THIS CELL ###\n","\n","X = torch.randn(32, 10)\n","y = torch.randint(0, 2, (32,))\n","\n","dataset = TensorDataset(X, y)\n","dataloader = DataLoader(dataset, batch_size=32)\n","\n","# Model setup\n","loss_fn = nn.CrossEntropyLoss()\n","optimizer = torch.optim.Adam(model.parameters())\n","\n","# Minimal training loop to trigger crash\n","model.train()\n","\n","for inputs, targets in dataloader:\n"," outputs = model(inputs)\n"," loss = loss_fn(outputs, targets)\n"," break\n","\n","print('βœ… Success!')\n","\n","#############################"]},{"cell_type":"markdown","source":["## 3.2 **(12pts)** Implement an MLP for Random Data in PyTorch.\n","You are given training and validation datasets for a 10-class image classification problem. Each sample is a randomly generated grayscale image of shape 1x28x28 and a randomly assigned label in `{0,1,...,9}`."],"metadata":{"id":"mf7OYIyzZZiR"}},{"cell_type":"code","source":["## DO NOT EDIT THIS CELL\n","# number of samples\n","n_train = 100\n","n_val = 100\n","\n","# define random training images (28x28) and labels (between 0-9)\n","X_train = torch.randn(n_train, 1, 28, 28)\n","y_train = torch.randint(0, 10, (n_train,))\n","\n","# define random validation images (28x28) and labels (between 0-9)\n","X_val = torch.randn(n_val, 1, 28, 28)\n","y_val = torch.randint(0, 10, (n_val,))\n","\n","# create datasets\n","train_dataset = TensorDataset(X_train, y_train)\n","val_dataset = TensorDataset(X_train, y_train)\n","\n","# create data loaders\n","train_loader = DataLoader(train_dataset, batch_size=10, shuffle=True)\n","val_loader = DataLoader(val_dataset, batch_size=10, shuffle=False)"],"metadata":{"id":"Yh0aQ_xeat-6"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["3.2.1 **(4 pts)** Implement the following MLP in PyTorch:\n","- Use a sequential model with:\n"," - Flatten\n"," - Linear(28x28 β†’ 128), ReLU\n"," - Linear(128 β†’ 64), ReLU\n"," - Linear(64 β†’ 10)\n","- Define the `forward` function"],"metadata":{"id":"jQzWKbzsoklV"}},{"cell_type":"code","source":["class MLP(nn.Module):\n"," def __init__(self):\n"," super().__init__()\n","\n"," self.model = nn.Sequential(\n"," nn.Flatten(),\n"," nn.Linear(28*28, 128),\n"," nn.ReLU(),\n"," nn.Linear(128, 64),\n"," nn.ReLU(),\n"," nn.Linear(64, 10)\n"," )\n","\n"," def forward(self, x):\n"," return self.model(x)"],"metadata":{"id":"RQDPF5N3aQrJ"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["3.2.2 **(3pts)** Instantiate the model, move it to the `device`, and print the model."],"metadata":{"id":"mutVol-DpWN3"}},{"cell_type":"code","source":["model = MLP()\n","model.to(device)\n","\n","print(model)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"N0A2nJvzaYiw","executionInfo":{"status":"ok","timestamp":1774458151343,"user_tz":-60,"elapsed":14,"user":{"displayName":"Olivia Lecomte","userId":"07830699269148802057"}},"outputId":"93fd3034-42b2-464d-e146-22378a3538ce"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["MLP(\n"," (model): Sequential(\n"," (0): Flatten(start_dim=1, end_dim=-1)\n"," (1): Linear(in_features=784, out_features=128, bias=True)\n"," (2): ReLU()\n"," (3): Linear(in_features=128, out_features=64, bias=True)\n"," (4): ReLU()\n"," (5): Linear(in_features=64, out_features=10, bias=True)\n"," )\n",")\n"]}]},{"cell_type":"code","source":["### GRADING TOOL ###\n","# paste printed output here\n","stu_model=\"\"\"\n","CNNModel(\n"," (features): Sequential(\n"," (0): Conv2d(1, 32, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))\n"," (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n"," (2): ReLU()\n"," (3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n"," (4): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2))\n"," (5): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n"," (6): ReLU()\n"," (7): Flatten(start_dim=1, end_dim=-1)\n"," )\n"," (classifier): Sequential(\n"," (0): LazyLinear(in_features=0, out_features=128, bias=True)\n"," (1): ReLU()\n"," (2): Dropout(p=0.5, inplace=False)\n"," (3): Linear(in_features=128, out_features=10, bias=True)\n"," )\n",")\n","\"\"\"\n","\n","# reference to compare to\n","ref_model = \"\"\"\n","MLP(\n"," (model): Sequential(\n"," (0): Flatten(start_dim=1, end_dim=-1)\n"," (1): Linear(in_features=784, out_features=128, bias=True)\n"," (2): ReLU()\n"," (3): Linear(in_features=128, out_features=64, bias=True)\n"," (4): ReLU()\n"," (5): Linear(in_features=64, out_features=10, bias=True)\n"," )\n",")\n","\"\"\"\n","\n","def compare_model_strings(stu, ref):\n"," if ref.strip() == stu.strip():\n"," print('βœ… Correct')\n"," else:\n"," print('❌ Incorrect')\n"," # Optional: show the first point of difference\n"," import difflib\n"," diff = difflib.unified_diff(\n"," ref.strip().splitlines(),\n"," stu.strip().splitlines(),\n"," fromfile='Reference',\n"," tofile='Student',\n"," lineterm=''\n"," )\n"," print(\"\\n\".join(diff))\n","\n","# Run the comparison\n","compare_model_strings(stu_model, ref_model)"],"metadata":{"id":"VGOjBqMX_y0M"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["3.2.3 **(4pts)** Define the hyperparameters.\n","- Use:\n"," - The cross entropy loss function\n"," - `Adam` optimizer with learning rate `0.001`\n"," - 20 training epochs"],"metadata":{"id":"mnQ1GW90ppQU"}},{"cell_type":"code","source":["criterion = nn.CrossEntropyLoss()\n","optimizer = optim.Adam(model.parameters(), lr=0.001)\n","n_epochs = 20"],"metadata":{"id":"jtQU-CYJfWR9"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["3.2.4 **(1pt)** Train, plot, and evaluate the model."],"metadata":{"id":"OsHAWwZTqJ9x"}},{"cell_type":"code","source":["### DO NOT EDIT THIS CELL ###\n","\n","def train_epoch(model, train_dataloader, optimizer, loss_fn):\n"," losses = []\n"," correct_predictions = 0\n"," # Iterate mini batches over training dataset\n"," for features, labels in tqdm(train_dataloader):\n"," features = features.to(device)\n"," labels = labels.to(device)\n"," # Run predictions\n"," output = model(features)\n"," # Set gradients to zero\n"," optimizer.zero_grad()\n"," # Compute loss\n"," loss = loss_fn(output, labels)\n"," # Backpropagate (compute gradients)\n"," loss.backward()\n"," # Make an optimization step (update parameters)\n"," optimizer.step()\n"," # Log metrics\n"," losses.append(loss.item())\n"," predicted_labels = output.argmax(dim=1)\n"," correct_predictions += (predicted_labels == labels).sum().item()\n"," accuracy = 100.0 * correct_predictions / len(train_dataloader.dataset)\n"," # Return loss values for each iteration and accuracy\n"," mean_loss = np.array(losses).mean()\n"," return mean_loss, accuracy\n","\n","def evaluate(model, dataloader, loss_fn):\n"," losses = []\n"," correct_predictions = 0\n"," with torch.no_grad():\n"," for features, labels in dataloader:\n"," features = features.to(device)\n"," labels = labels.to(device)\n"," # Run predictions\n"," output = model(features)\n"," # Compute loss\n"," loss = loss_fn(output, labels)\n"," # Save metrics\n"," predicted_labels = output.argmax(dim=1)\n"," correct_predictions += (predicted_labels == labels).sum().item()\n"," losses.append(loss.item())\n"," mean_loss = np.array(losses).mean()\n"," accuracy = 100.0 * correct_predictions / len(dataloader.dataset)\n"," # Return mean loss and accuracy\n"," return mean_loss, accuracy\n","\n","def train(model, train_dataloader, val_dataloader, optimizer, n_epochs, loss_fn):\n"," # We will monitor loss functions as the training progresses\n"," train_losses = []\n"," val_losses = []\n"," train_accuracies = []\n"," val_accuracies = []\n","\n"," for epoch in range(n_epochs):\n"," model.train()\n"," train_loss, train_accuracy = train_epoch(model, train_dataloader, optimizer, loss_fn)\n"," model.eval()\n"," val_loss, val_accuracy = evaluate(model, val_dataloader, loss_fn)\n"," train_losses.append(train_loss)\n"," val_losses.append(val_loss)\n"," train_accuracies.append(train_accuracy)\n"," val_accuracies.append(val_accuracy)\n"," print('Epoch {}/{}: train_loss: {:.4f}, train_accuracy: {:.4f}, val_loss: {:.4f}, val_accuracy: {:.4f}'.format(epoch+1, n_epochs,\n"," train_losses[-1],\n"," train_accuracies[-1],\n"," val_losses[-1],\n"," val_accuracies[-1]))\n"," return train_losses, val_losses, train_accuracies, val_accuracies\n","\n","def plot(train_losses, val_losses, train_accuracies, val_accuracies, title):\n"," plt.figure()\n"," plt.plot(np.arange(len(train_losses)), train_losses)\n"," plt.plot(np.arange(len(val_losses)), val_losses)\n"," plt.legend(['train_loss', 'val_loss'])\n"," plt.xlabel('epoch')\n"," plt.xticks(np.arange(len(train_losses)), np.arange(1, len(train_losses)+1))\n"," plt.ylabel('loss value')\n"," plt.title('{}: Train/val loss'.format(title));\n","\n"," plt.figure()\n"," plt.plot(np.arange(len(train_accuracies)), train_accuracies)\n"," plt.plot(np.arange(len(val_accuracies)), val_accuracies)\n"," plt.legend(['train_acc', 'val_acc'])\n"," plt.xlabel('epoch')\n"," plt.xticks(np.arange(len(train_losses)), np.arange(1, len(train_losses)+1))\n"," plt.ylabel('accuracy')\n"," plt.title('{}: Train/val accuracy'.format(title));\n","\n"],"metadata":{"id":"ykvLWhdOas7F"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# train model\n","train_losses, val_losses, train_acc, val_acc = train(model, train_loader, val_loader, optimizer, n_epochs, criterion)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"eKHQbaJDa3vj","executionInfo":{"status":"ok","timestamp":1774458151907,"user_tz":-60,"elapsed":361,"user":{"displayName":"Olivia Lecomte","userId":"07830699269148802057"}},"outputId":"49e2e3e4-6ead-4cac-f837-fd1b1d603d0b"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 48.54it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 1/20: train_loss: 2.3117, train_accuracy: 10.0000, val_loss: 2.0470, val_accuracy: 55.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 361.37it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 2/20: train_loss: 1.9682, train_accuracy: 61.0000, val_loss: 1.7530, val_accuracy: 83.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 527.82it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 3/20: train_loss: 1.6262, train_accuracy: 83.0000, val_loss: 1.3396, val_accuracy: 92.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 481.36it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 4/20: train_loss: 1.1762, train_accuracy: 95.0000, val_loss: 0.8571, val_accuracy: 99.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 560.92it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 5/20: train_loss: 0.7054, train_accuracy: 99.0000, val_loss: 0.4589, val_accuracy: 99.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 547.37it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 6/20: train_loss: 0.3450, train_accuracy: 100.0000, val_loss: 0.2015, val_accuracy: 100.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 512.14it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 7/20: train_loss: 0.1437, train_accuracy: 100.0000, val_loss: 0.0839, val_accuracy: 100.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 501.06it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 8/20: train_loss: 0.0645, train_accuracy: 100.0000, val_loss: 0.0363, val_accuracy: 100.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 498.08it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 9/20: train_loss: 0.0279, train_accuracy: 100.0000, val_loss: 0.0191, val_accuracy: 100.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 444.17it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 10/20: train_loss: 0.0159, train_accuracy: 100.0000, val_loss: 0.0121, val_accuracy: 100.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 526.41it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 11/20: train_loss: 0.0109, train_accuracy: 100.0000, val_loss: 0.0090, val_accuracy: 100.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 550.59it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 12/20: train_loss: 0.0083, train_accuracy: 100.0000, val_loss: 0.0073, val_accuracy: 100.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 573.20it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 13/20: train_loss: 0.0069, train_accuracy: 100.0000, val_loss: 0.0062, val_accuracy: 100.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 556.41it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 14/20: train_loss: 0.0059, train_accuracy: 100.0000, val_loss: 0.0053, val_accuracy: 100.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 569.38it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 15/20: train_loss: 0.0051, train_accuracy: 100.0000, val_loss: 0.0047, val_accuracy: 100.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 521.82it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 16/20: train_loss: 0.0045, train_accuracy: 100.0000, val_loss: 0.0042, val_accuracy: 100.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 603.51it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 17/20: train_loss: 0.0040, train_accuracy: 100.0000, val_loss: 0.0037, val_accuracy: 100.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 515.83it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 18/20: train_loss: 0.0036, train_accuracy: 100.0000, val_loss: 0.0033, val_accuracy: 100.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 449.32it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 19/20: train_loss: 0.0032, train_accuracy: 100.0000, val_loss: 0.0030, val_accuracy: 100.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 431.87it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 20/20: train_loss: 0.0029, train_accuracy: 100.0000, val_loss: 0.0028, val_accuracy: 100.0000\n"]}]},{"cell_type":"code","source":["# visualize results\n","plot(train_losses, val_losses, train_acc, val_acc, title='MLP')\n"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":927},"id":"wsH1dPEpbLYz","executionInfo":{"status":"ok","timestamp":1774458152463,"user_tz":-60,"elapsed":552,"user":{"displayName":"Olivia Lecomte","userId":"07830699269148802057"}},"outputId":"a61522f3-ff67-4de1-8c22-5608f4555f6e"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["
"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAbBdJREFUeJzt3Xd0VNXexvHvmfQeAqn03jsoTRBBqlQVVBSxe4WriJXrxYL3ivgqVgT1itgQVIoI0iUgRVoITQidUJLQk5BA2sz7x5BIqJlkJpNMns9as3Jy5uw9vwTHeXLOPnsbFovFgoiIiIiLMDm7ABERERF7UrgRERERl6JwIyIiIi5F4UZERERcisKNiIiIuBSFGxEREXEpCjciIiLiUhRuRERExKUo3IiIiIhLUbgRkVKrWrVqDBs2zNllXNPBgwcxDIOpU6de97jo6GgMwyA6OrpY6hJxdQo3IqXA1KlTMQwDwzBYtWrVFc9bLBYqV66MYRjccccd+Z4zDIMRI0Zct/9bb701r3/DMAgJCaF169ZMmTIFs9lsU625H9QFeZQEv/76KyaTicTERGeXIiJ24u7sAkSk4Ly9vZk2bRodOnTIt3/FihUcOXIELy+vQvddqVIlxo0bB8CJEyf45ptveOSRR9i9ezdvv/12gfupX78+3377bb59o0ePxt/fn1deeaXQ9V1NXFwcJlPR/kabP38+LVu2JCIiwk5ViYizKdyIlCK9evXip59+4qOPPsLd/e+377Rp02jZsiUnT54sdN9BQUHcf//9ed8/8cQT1K1bl08++YQ333wTDw+PAvUTHh6erx+At99+mwoVKlyx/1Jms5nMzEy8vb0LXHNRwlyu3377jYcffrjI/YhIyaHLUiKlyL333supU6dYsmRJ3r7MzEx+/vln7rvvPru+lq+vL23atCEtLY0TJ04AkJ6ezq5du4oUonLlXi77/vvvadiwIV5eXixcuBCAd999l3bt2lG+fHl8fHxo2bIlP//88xV9XD7mJvfy3erVqxk1ahShoaH4+fkxYMCAvJ/hUtu2bePw4cP07t2bpKQk3N3deeONN644Li4uDsMw+OSTTwA4ffo0zz//PI0bN8bf35/AwEB69uzJli1bivx7udRPP/1Ey5Yt8fHxyQuHR48ezXdMYmIiDz30EJUqVcLLy4vIyEj69evHwYMH847ZuHEj3bt3p0KFCvj4+FC9enUFOnFpCjcipUi1atVo27YtP/zwQ96+BQsWkJyczD333GP319u/fz9ubm4EBwcDsH79eurXr5/3IV9Uv//+O88++yyDBw/mww8/pFq1agB8+OGHNG/enLFjx/LWW2/h7u7O3Xffzfz58wvU7z//+U+2bNnCa6+9xj/+8Q9+/fXXq447+u233wgLC6NVq1aEh4fTqVMnfvzxxyuOmzFjBm5ubtx9992A9fcyZ84c7rjjDiZMmMALL7zAtm3b6NSpE8eOHSv8L+QSU6dOZdCgQbi5uTFu3Dgee+wxZs2aRYcOHTh79mzecXfeeSezZ8/moYce4tNPP+Xpp58mNTWV+Ph4AI4fP063bt04ePAgL7/8Mh9//DFDhgzhzz//tEudIiWRLkuJlDL33Xcfo0eP5vz58/j4+PD999/TqVMnoqKiitRvTk5O3hmZkydPMmnSJGJiYujTpw++vr72KP0KcXFxbNu2jQYNGuTbv3v3bnx8fPK+HzFiBC1atGDChAn07t37hv2WL1+exYsX5w1aNpvNfPTRRyQnJxMUFJR33Pz58+nZs2fecYMHD+aJJ55g+/btNGrUKO+4GTNm0KlTJ8LDwwFo3Lgxu3fvzjfe54EHHqBevXp8+eWXjBkzphC/jb9lZWXx0ksv0ahRI1auXJl3qa5Dhw7ccccdvP/++7zxxhucPXuWNWvW8H//9388//zzee1Hjx6dt71mzRrOnDnD4sWLadWqVd7+//znP0WqUaQk05kbkVJm0KBBnD9/nnnz5pGamsq8efPscklq165dhIaGEhoaSv369fn444/p3bs3U6ZMyTvm1ltvxWKx8Prrrxf59QA6dep0RbAB8gWbM2fOkJyczC233EJMTEyB+n388cfz3Y11yy23kJOTw6FDh/L2nT17lrVr1+YLSwMHDsTd3Z0ZM2bk7du+fTt//fUXgwcPztvn5eWVF2xycnI4deoU/v7+1K1bt8A1Xs/GjRs5fvw4Tz31VL4xSL1796ZevXp5Z7B8fHzw9PQkOjqaM2fOXLWv3LNu8+bNIysrq8i1iZQGCjcipUxoaChdu3Zl2rRpzJo1i5ycHO66664i91utWjWWLFnC0qVLWbVqFYmJicybN48KFSrYoeqrq169+lX3z5s3jzZt2uDt7U1ISAihoaFMmjSJ5OTkAvVbpUqVfN+XK1cOIF8AWLRoEQDdunXL21ehQgW6dOmS79LUjBkzcHd3Z+DAgXn7zGYz77//PrVr18bLy4sKFSoQGhrK1q1bC1zj9eSGsLp1617xXL169fKe9/LyYvz48SxYsIDw8HA6duzIO++8k++29k6dOnHnnXfyxhtvUKFCBfr168dXX31FRkZGkesUKakUbkRKofvuu48FCxYwefJkevbsmffXeVH4+fnRtWtXunTpQvv27QkLCyt6oTdw6RmaXH/88Qd9+/bF29ubTz/9lN9++40lS5Zw3333YbFYCtSvm5vbVfdf2v63336jffv2+S5TAdxzzz3s3r2b2NhYAH788Ue6dOmSL+S99dZbjBo1io4dO/Ldd9+xaNEilixZQsOGDW2eF6ioRo4cye7duxk3bhze3t6MGTOG+vXrs3nzZsA6cPvnn39m7dq1jBgxgqNHj/Lwww/TsmVLzp07V6y1ihQXhRuRUmjAgAGYTCb+/PNPu98l5WwzZ87E29ubRYsW8fDDD9OzZ0+6du1q19ewWCwsXLjwquN3+vfvj6enJzNmzCA2Npbdu3dfMVj7559/pnPnznz55Zfcc889dOvWja5du+Yb6FsUVatWBaxjki4XFxeX93yumjVr8txzz7F48WK2b99OZmYm7733Xr5j2rRpw3//+182btzI999/z44dO5g+fbpd6hUpaRRuREohf39/Jk2axOuvv06fPn2K7XXteSv4tbi5uWEYBjk5OXn7Dh48yJw5c+z2Ghs2bOD48eNXDTfBwcF0796dH3/8kenTp+Pp6Un//v2vqPHys0g//fTTFbdpF1arVq0ICwtj8uTJ+S4fLViwgJ07d+bVnZ6ezoULF/K1rVmzJgEBAXntzpw5c0WtzZo1A9ClKXFZultKpJR68MEHC3zsxo0br3p3zK233nrFbMfXs379ejp37sxrr71mt0HFl+vduzcTJkygR48e3HfffRw/fpyJEydSq1Yttm7dapfXmD9/PtWqVbvqYGaw3jV1//338+mnn9K9e/crLvvdcccdjB07loceeoh27dqxbds2vv/+e2rUqGGX+jw8PBg/fjwPPfQQnTp14t577yUpKSnvdvlnn30WsN5V1qVLFwYNGkSDBg1wd3dn9uzZJCUl5Z1t+vrrr/n0008ZMGAANWvWJDU1lS+++ILAwEB69epll3pFShqFG5EyYN26daxbt+6K/W+++aZN4aY43HbbbXz55Ze8/fbbjBw5kurVqzN+/HgOHjxot3Dz22+/XfeDvW/fvvj4+JCamprvLqlc//rXv0hLS2PatGnMmDGDFi1aMH/+fF5++WW71AcwbNgwfH19efvtt3nppZfyJiMcP358XtiqXLky9957L8uWLePbb7/F3d2devXq8eOPP3LnnXcC1gHF69evZ/r06SQlJREUFMRNN93E999/f80B3SKlnWEp6Ag9EREXkJSURGRkJPPmzdOZCxEXpTE3IlKmJCcn8+qrr9K5c2dnlyIiDqIzNyIiIuJSdOZGREREXIrCjYiIiLgUhRsRERFxKQo3IiIi4lLK3Dw3ZrOZY8eOERAQkG/VYBERESm5LBYLqampREVFYTJd/9xMmQs3x44do3Llys4uQ0RERArh8OHDVKpU6brHlLlwExAQAFh/OYGBgU6uRkRERAoiJSWFypUr532OX0+ZCze5l6ICAwMVbkREREqZggwp0YBiERERcSkKNyIiIuJSFG5ERETEpZS5MTciIuKacnJyyMrKcnYZUgSenp43vM27IBRuRESkVLNYLCQmJnL27FlnlyJFZDKZqF69Op6enkXqR+FGRERKtdxgExYWhq+vryZoLaVyJ9lNSEigSpUqRfp3VLgREZFSKycnJy/YlC9f3tnlSBGFhoZy7NgxsrOz8fDwKHQ/GlAsIiKlVu4YG19fXydXIvaQezkqJyenSP0o3IiISKmnS1GuwV7/jgo3IiIi4lIUbkREREq5atWq8cEHH9ilr+joaAzDKNV3n2lAsYiIiBPceuutNGvWzC6hZMOGDfj5+RW9KBehMzd2lJh8gV2JKc4uQ0REXIDFYiE7O7tAx4aGhmpQ9SUUbuxkwbYEbnnnd0bP2obFYnF2OSIiUoINGzaMFStW8OGHH2IYBoZhMHXqVAzDYMGCBbRs2RIvLy9WrVrFvn376NevH+Hh4fj7+9O6dWuWLl2ar7/LL0sZhsH//vc/BgwYgK+vL7Vr12bu3LmFrnfmzJk0bNgQLy8vqlWrxnvvvZfv+U8//ZTatWvj7e1NeHg4d911V95zP//8M40bN8bHx4fy5cvTtWtX0tLSCl1LQSjc2EnLauUwDIPN8Wf5c/9pZ5cjIlJmWSwW0jOznfIo6B+3H374IW3btuWxxx4jISGBhIQEKleuDMDLL7/M22+/zc6dO2nSpAnnzp2jV69eLFu2jM2bN9OjRw/69OlDfHz8dV/jjTfeYNCgQWzdupVevXoxZMgQTp+2/fNp06ZNDBo0iHvuuYdt27bx+uuvM2bMGKZOnQrAxo0befrppxk7dixxcXEsXLiQjh07ApCQkMC9997Lww8/zM6dO4mOjmbgwIEOPwmgMTd2EhbgzaBWlfjuz3g+jd5L25qaTEpExBnOZ+XQ4NVFTnntv8Z2x9fzxh+tQUFBeHp64uvrS0REBAC7du0CYOzYsdx+++15x4aEhNC0adO87998801mz57N3LlzGTFixDVfY9iwYdx7770AvPXWW3z00UesX7+eHj162PQzTZgwgS5dujBmzBgA6tSpw19//cX//d//MWzYMOLj4/Hz8+OOO+4gICCAqlWr0rx5c8AabrKzsxk4cCBVq1YFoHHjxja9fmHozI0dPdGxJm4mgz/2nGTbkWRnlyMiIqVQq1at8n1/7tw5nn/+eerXr09wcDD+/v7s3LnzhmdumjRpkrft5+dHYGAgx48ft7menTt30r59+3z72rdvz549e8jJyeH222+natWq1KhRgwceeIDvv/+e9PR0AJo2bUqXLl1o3Lgxd999N1988QVnzpyxuQZb6cyNHVUO8aVv0yhmbz7Kp9F7mXR/S2eXJCJS5vh4uPHX2O5Oe+2iuvyup+eff54lS5bw7rvvUqtWLXx8fLjrrrvIzMy8bj+XL19gGAZms7nI9V0uICCAmJgYoqOjWbx4Ma+++iqvv/46GzZsIDg4mCVLlrBmzRoWL17Mxx9/zCuvvMK6deuoXr263WvJpTM3dvaPW2sCsHBHInuPn3NyNSIiZY9hGPh6ujvlYcsMu56engVaZmD16tUMGzaMAQMG0LhxYyIiIjh48GARfkO2qV+/PqtXr76ipjp16uDmZg1z7u7udO3alXfeeYetW7dy8OBBfv/9d8D679G+fXveeOMNNm/ejKenJ7Nnz3ZozTpzY2d1wgO4vUE4S/5KYvKKfbx7d9MbNxIRkTKnWrVqrFu3joMHD+Lv73/Nsyq1a9dm1qxZ9OnTB8MwGDNmjEPOwFzLc889R+vWrXnzzTcZPHgwa9eu5ZNPPuHTTz8FYN68eezfv5+OHTtSrlw5fvvtN8xmM3Xr1mXdunUsW7aMbt26ERYWxrp16zhx4gT169d3aM06c+MAT108ezNn81GOnj3v5GpERKQkev7553Fzc6NBgwaEhoZecwzNhAkTKFeuHO3ataNPnz50796dFi1aFFudLVq04Mcff2T69Ok0atSIV199lbFjxzJs2DAAgoODmTVrFrfddhv169dn8uTJ/PDDDzRs2JDAwEBWrlxJr169qFOnDv/+979577336Nmzp0NrNixlbFKWlJQUgoKCSE5OJjAw0GGvc98Xf7Jm3ymGtavG630bOux1RETKsgsXLnDgwAGqV6+Ot7e3s8uRIrrev6ctn986c+MgwzvXAuCH9fGcPJfh5GpERETKDoUbB2lXszxNKwWRkW1myqoDzi5HREQEgCeffBJ/f/+rPp588klnl2cXGlDsIIZh8FTnWjzx7Sa+XXuIJ2+tSaC3x40bioiIONDYsWN5/vnnr/qcI4drFCeFGwe6vX44tcP82XP8HN+uPZR3qUpERMRZwsLCCAsLc3YZDqXLUg5kMhl5895MWXWA85k3ns9AREREikbhxsH6NI2iUjkfTqVl8uPGw84uR0RExOUp3DiYh5uJJzrWAODzlfvJyim+iZdERETKIoWbYnB3q8pU8Pfi6Nnz/BJ7zNnliIiIuDSFm2Lg7eHGIx2sC4RNit6L2Vym5k0UEREpVgo3xeT+NlUI8HZn34k0Fv+V6OxyRESklKtWrRoffPBBgY41DIM5c+Y4tJ6SROGmmAR4e/Bg22oATFy+jzK26oWIiEixUbgpRg+1r4a3h4ltR5NZtfeks8sRERFxSQo3xai8vxf3tK4CwMTle51cjYiIOMvnn39OVFQUZnP+O2j79evHww8/zL59++jXrx/h4eH4+/vTunVrli5darfX37ZtG7fddhs+Pj6UL1+exx9/nHPnzuU9Hx0dzU033YSfnx/BwcG0b9+eQ4cOAbBlyxY6d+5MQEAAgYGBtGzZko0bN9qtNntQuClmj3esgbvJ4M/9p9l06IyzyxERcT0WC2SmOedRwCEHd999N6dOnWL58uV5+06fPs3ChQsZMmQI586do1evXixbtozNmzfTo0cP+vTpQ3x8fJF/PWlpaXTv3p1y5cqxYcMGfvrpJ5YuXcqIESMAyM7Opn///nTq1ImtW7eydu1aHn/8cQzDAGDIkCFUqlSJDRs2sGnTJl5++WU8PErW8kJafqGYRQX7MKB5RX7adIRJ0Xv534OtnV2SiIhryUqHt6Kc89r/Ogaefjc8rFy5cvTs2ZNp06bRpUsXAH7++WcqVKhA586dMZlMNG3aNO/4N998k9mzZzN37ty8EFJY06ZN48KFC3zzzTf4+Vlr/eSTT+jTpw/jx4/Hw8OD5ORk7rjjDmrWtM6yX79+/bz28fHxvPDCC9SrVw+A2rVrF6keR9CZGyd48taaGAYs3XmcXYkpzi5HREScYMiQIcycOZOMjAwAvv/+e+655x5MJhPnzp3j+eefp379+gQHB+Pv78/OnTvtcuZm586dNG3aNC/YALRv3x6z2UxcXBwhISEMGzaM7t2706dPHz788EMSEhLyjh01ahSPPvooXbt25e2332bfvn1FrsnedObGCWqG+tOrUSTztyUwKXofH97T3NkliYi4Dg9f6xkUZ712AfXp0weLxcL8+fNp3bo1f/zxB++//z4Azz//PEuWLOHdd9+lVq1a+Pj4cNddd5GZmemoyvP56quvePrpp1m4cCEzZszg3//+N0uWLKFNmza8/vrr3HfffcyfP58FCxbw2muvMX36dAYMGFAstRWEztw4Se6Cmr9uOcahU2lOrkZExIUYhvXSkDMeF8elFIS3tzcDBw7k+++/54cffqBu3bq0aNECgNWrVzNs2DAGDBhA48aNiYiI4ODBg3b59dSvX58tW7aQlvb3Z8/q1asxmUzUrVs3b1/z5s0ZPXo0a9asoVGjRkybNi3vuTp16vDss8+yePFiBg4cyFdffWWX2uxF4cZJGlUMolOdUMwW+GzlfmeXIyIiTjBkyBDmz5/PlClTGDJkSN7+2rVrM2vWLGJjY9myZQv33XffFXdWFeU1vb29efDBB9m+fTvLly/nn//8Jw888ADh4eEcOHCA0aNHs3btWg4dOsTixYvZs2cP9evX5/z584wYMYLo6GgOHTrE6tWr2bBhQ74xOSWBwo0TPXXx7M3PG4+QlHLBydWIiEhxu+222wgJCSEuLo777rsvb/+ECRMoV64c7dq1o0+fPnTv3j3vrE5R+fr6smjRIk6fPk3r1q2566676NKlC5988kne87t27eLOO++kTp06PP744wwfPpwnnngCNzc3Tp06xdChQ6lTpw6DBg2iZ8+evPHGG3apzV4MSxmbKjclJYWgoCCSk5MJDAx0ai0Wi4W7J69l46EzPN6xBv/qVbKSr4hISXfhwgUOHDhA9erV8fb2dnY5UkTX+/e05fNbZ26cyDAMnupsPXvz3Z+HOJtePAPFREREXJnCjZN1rhtGvYgA0jNz+HrNIWeXIyIipcz333+Pv7//VR8NGzZ0dnlOoVvBncx69qYWT/+wma/WHODRW6rj56V/FhERKZi+ffty8803X/W5kjZzcHHRp2gJ0LtxJBMWx3HwVDo/rI/n0VtqOLskEREpJQICAggICHB2GSWKLkuVAG4mgyc7WcfefPHHfjKyc5xckYiISOmlcFNCDGhRkfBAL5JSMpgVc9TZ5YiIlCr2mgNGnMteN3DrslQJ4eXuxmO31OA/83cyecU+7m5ZCXc3ZU8Rkevx9PTEZDJx7NgxQkND8fT0zFu9WkoXi8XCiRMnMAyjyGOFFG5KkHtvqsIny/dy6FQ6v21PpG9TJ61qKyJSSphMJqpXr05CQgLHjjlpPSmxG8MwqFSpEm5ubkXqR+HGXtJOwYrxEBABt4wqVBd+Xu481K467y/dzafL99KnSaT+AhERuQFPT0+qVKlCdnY2OTkas1iaeXh4FDnYgMKN/RyIhvWfgYcfNBsCAeGF6ubBdlX5fOU+diWmsjzuOLfVK1w/IiJlSe6ljLJ667Pkp0Ed9tJwIFRsBVlpEP1WobsJ9vVkSJuqAExcvs9ug6tERETKCoUbezEM6PYf63bMN3B8Z6G7eqRDdTzdTGw6dIb1B07bqUAREZGywanhZty4cbRu3ZqAgADCwsLo378/cXFxN2z3008/Ua9ePby9vWncuDG//fZbMVRbAFXbQv0+YDHDklcL3U14oDd3taoEwKfR++xVnYiISJng1HCzYsUKhg8fzp9//smSJUvIysqiW7dupKWlXbPNmjVruPfee3nkkUfYvHkz/fv3p3///mzfvr0YK7+Orm+AyR32LIZ9ywvdzRMda2AyYMXuE2w/mmzHAkVERFybYSlBgzpOnDhBWFgYK1asoGPHjlc9ZvDgwaSlpTFv3ry8fW3atKFZs2ZMnjz5hq9hy5LphbbgJVg3GcIbwxMrwFS4kd/PTN/ML7HH6NU4gk+HtLRzkSIiIqWHLZ/fJWrMTXKy9QxFSEjINY9Zu3YtXbt2zbeve/furF279qrHZ2RkkJKSku/hcB1fBK8gSNoGW2cUupt/3GpdkmHB9kT2nThnr+pERERcWokJN2azmZEjR9K+fXsaNWp0zeMSExMJD89/e3R4eDiJiYlXPX7cuHEEBQXlPSpXrmzXuq/Krzx0fM66vexNyEwvVDf1IgLpWj8MiwUma+yNiIhIgZSYcDN8+HC2b9/O9OnT7drv6NGjSU5OznscPnzYrv1f001PQFAVSD0Gf04sdDdPda4FwOzNRzl69ry9qhMREXFZJSLcjBgxgnnz5rF8+XIqVap03WMjIiJISkrKty8pKYmIiIirHu/l5UVgYGC+R7Hw8Iaur1m3V30A544XqpsWVcrRpkYI2WYLX6zcb7/6REREXJRTw43FYmHEiBHMnj2b33//nerVq9+wTdu2bVm2bFm+fUuWLKFt27aOKrPwGg6EqBaQeQ6ixxW6m+EXz95M3xDPqXMZ9qpORETEJTk13AwfPpzvvvuOadOmERAQQGJiIomJiZw///fll6FDhzJ69Oi875955hkWLlzIe++9x65du3j99dfZuHEjI0aMcMaPcH0mE3T/r3V701Q4vqtQ3XSoVYHGFYO4kGVmyuoD9qtPRETEBTk13EyaNInk5GRuvfVWIiMj8x4zZvx9h1F8fDwJCQl537dr145p06bx+eef07RpU37++WfmzJlz3UHITlW1HdS7o0gT+xmGwfDO1junvll7iJQLWfasUERExKWUqHluikOxzHNzuZN74dObwZwNQ3+BGrfa3IXZbOH291ew70QaL/aoy1O31rJ/nSIiIiVUqZ3nxmVVqAWtHrFuL/43mM02d2EyGXmB5ss/DnA+M8eeFYqIiLgMhZvi0ukl8AqExMJP7Ne3WRQVg304lZbJjxuL6ZZ2ERGRUkbhprj4lYdbLk7s93vhJvbzcDPxZKcaAHy2Yh+Z2bafARIREXF1CjfF6eYnIagypByFPz8tVBd3t6pMBX8vjiVf4JfYo3YuUEREpPRTuClOHt7QJXdiv/cLNbGft4cbj95inQ9o0op95JjL1HhwERGRG1K4KW6N7oSo5hcn9nu7UF3c36Yqgd7u7D+RxqIdV19TS0REpKxSuCluJhN0+491e9NUOBFncxf+Xu4Ma289ezNx+V7K2N38IiIi16Vw4wzVOkDd3mDJgSWvFaqLh9pVw9fTjR3HUlix+4SdCxQRESm9FG6c5fY3wHCD3QvgwEqbm5fz8+S+m6oA8OnyffauTkREpNRSuHGWCrWh1cPW7UJO7PfoLTXwdDOx/uBp1h84becCRURESieFG2e69WXwDICELbDtJ5ubRwR5c2fLSgB8Gr3X3tWJiIiUSgo3zuRXAW4ZZd1eNhayzl//+Kt4slMNTAZEx51g+9FkOxcoIiJS+ijcOFubf0BgJUg5An9Osrl51fJ+9GkaBcCkaI29ERERUbhxNg8f6PKqdfuPCZB20uYu/nFrTQB+257AvhPn7FmdiIhIqaNwUxI0vhsim0JmaqEm9qsXEUjX+uFYLDp7IyIionBTElw6sd/GKXByj81dPNXZevZmzuajHDlj+6KcIiIirkLhpqSo3hHq9Cz0xH4tqpSjXc3yZJstfLFyvwMKFBERKR0UbkqS28daJ/aLmw8HV9ncfHjnWgBM33CYE6kZ9q5ORESkVFC4KUlC60Crh6zbhZjYr13N8jStHExGtpkpqw84oEAREZGST+GmpOl0cWK/Y5th+0ybmhqGwfCLd059u/YQyeezHFGhiIhIiaZwU9L4h8Itz1q3l70BWRdsat61fjh1wv05l5HNt2sP2r8+ERGREk7hpiRq8xQEVoTkw7Busk1NTSaDp261jr2Zsvog6ZnZjqhQRESkxFK4KYnyTez3HqSdsqn5HU0iqRLiy+m0TKavP+yAAkVEREouhZuSqvEgiGgCGSmwYrxNTd3dTDzZyTr25vOV+8nMtn3FcRERkdJK4aakyjex35dw0rZVv+9sWZGwAC8SUy4we/MRBxQoIiJSMinclGQ1OkGdHmDOhqW2Tezn5e7G4x1rANYlGXLMFkdUKCIiUuIo3JR0uRP77ZoHh9bY1PTem6oQ7OvBwVPp/LYtwUEFioiIlCwKNyVdaF1o+aB1e9ErNk3s5+flzkPtqgMwcfleLBadvREREdencFMa3DoaPP3hWAzsmGVT0wfbVcXP041diaksjzvuoAJFRERKDoWb0sA/DNo/Y93+81Obmgb7enJ/m6oAfPK7zt6IiIjrU7gpLVo+BCYPOLoJErfb1PSRDtXxdDcRE3+WdQdOO6hAERGRkkHhprTwD4V6vazbMV/b1DQs0JtBrSoB1rE3IiIirkzhpjRpcXFg8dYZkHXepqZPdKyJm8ngjz0n2XrkrP1rExERKSEUbkqTGp0huApcSIa/frGpaeUQX/o1jQLg0+X7HFGdiIhIiaBwU5qYTNB8qHV7k22XpgD+cat1SYaFOxLZk5Rqz8pERERKDIWb0qb5EDBMEL8GTuy2qWnt8AC6NwwHYNIKnb0RERHXpHBT2gRGQe3u1m0bBxYDPHVrLQB+iT3G4dPp9qxMRESkRFC4KY1yZyze8gNkZ9jUtGnlYG6pXYEcs4XPVursjYiIuB6Fm9Ko1u0QEAnpp2DXfJub5569+XHjEY6nXLB3dSIiIk6lcFMaublD8/ut24W4NNWmRggtqgSTmW3my1UH7FyciIiIcynclFbNHwAM2B8Np20LKIZhMOI269mb7/48xNn0TPvXJyIi4iQKN6VVuapQs7N1e/O3NjfvXDeMehEBpGXm8PWaQ3YuTkRExHkUbkqz3BmLN38POdk2NTUMg+GdrWdvvlpzgLQM29qLiIiUVAo3pVndXuBbAc4lwp5FNjfv1TiSauV9OZuexQ/r4x1QoIiISPFTuCnN3D2h2X3W7ULMWOxmMvJmLf585X4ysnPsWZ2IiIhTKNyUdrmXpvYugeSjNjcf0LwSkUHeHE/NYOYm29uLiIiUNAo3pV2FWlC1A1jMsPk7m5t7upt47JYaAExesY/sHLO9KxQRESlWCjeuIHfG4s3fgtn2S0v33FSZED9P4k+ns+SvJDsXJyIiUrwUblxB/b7gHQzJh2Hfcpub+3q6M6hVZQBmxujSlIiIlG4KN67Awxua3mPdjplaqC4GtqgIQHTccU6naVI/EREpvRRuXEXuwOK4BXDuuM3N64QH0DAqkGyzhflbj9m5OBERkeKjcOMqwhtApdZgzobY7wvVxYDm1rM3szbr0pSIiJReCjeuJPfsTcw3YLHY3LxvsyhMBmyOP8uBk2l2Lk5ERKR4KNy4kkYDwTMATu+Hg3/Y3DwswJsOtUMBmK2zNyIiUkop3LgSTz9ofJd1uxAzFgMMvHhpas7mo1gKcfZHRETE2RRuXE3unDc750L6aZubd2sYjq+nG/Gn04mJP2Pn4kRERBxP4cbVRDWHiCaQkwlbptvc3NfTnR6NIgCYpTlvRESkFFK4cUW5Z29ivi7UwOKBzSsBMG9rghbTFBGRUkfhxhU1vhs8fOHELji83ubmbWuWJzzQi+TzWUTHnXBAgSIiIo6jcOOKvIOg4QDrdoztA4vdTAb9mlkHFs/WpSkRESllFG5cVe6cN9tnwYVkm5vnTuj3+67jJKdn2bMyERERh1K4cVWVb4LQepB9Hrb9ZHPz+pGB1IsIIDPHzLxtWo5BRERKD6eGm5UrV9KnTx+ioqIwDIM5c+Zc9/jo6GgMw7jikZiYWDwFlyaG8ffZm8LOedNCl6ZERKT0cWq4SUtLo2nTpkycONGmdnFxcSQkJOQ9wsLCHFRhKdf0HnDzhMStcGyzzc37Nq2IYcDGQ2eIP5XugAJFRETsz92ZL96zZ0969uxpc7uwsDCCg4PtX5Cr8Q2B+n1h+8/WszdRzW1qHhHkTfuaFVi19yRzYo/ydJfaDipURETEfkrlmJtmzZoRGRnJ7bffzurVq697bEZGBikpKfkeZUrunDfbfoaMczY3zx1YPFvLMYiISClRqsJNZGQkkydPZubMmcycOZPKlStz6623EhMTc80248aNIygoKO9RuXLlYqy4BKh2C4TUgMxU2DHb5uY9GkXg4+HGgZNpxB4+a//6RERE7KxUhZu6devyxBNP0LJlS9q1a8eUKVNo164d77///jXbjB49muTk5LzH4cOHi7HiEsAwoPkD1u1CzHnj5+VO94bhgHUxTRERkZKuVIWbq7npppvYu3fvNZ/38vIiMDAw36PMaTYETO5wZAMk/WVz8/4XL039ujWBrByzvasTERGxq1IfbmJjY4mMjHR2GSVbQDjU6WHdLsTZmw61KlDB34vTaZms0HIMIiJSwjk13Jw7d47Y2FhiY2MBOHDgALGxscTHxwPWS0pDhw7NO/6DDz7gl19+Ye/evWzfvp2RI0fy+++/M3z4cGeUX7q0HGb9umU6ZF2wqam7m4l+zaIA68BiERGRksyp4Wbjxo00b96c5s2ttyiPGjWK5s2b8+qrrwKQkJCQF3QAMjMzee6552jcuDGdOnViy5YtLF26lC5dujil/lKl5m0QVBkunIWdc21unnvX1JKdSSSf13IMIiJSchmWMnZ/b0pKCkFBQSQnJ5e98TfRb0P0OKjaAR6ab1NTi8VC9w9WsjvpHOPvbMzg1lUcVKSIiMiVbPn8LvVjbsQGze8HwwSHVsHJaw/CvhrDMPIGFs/ScgwiIlKCKdyUJUGVoFZX63YhBhb3b2ZdjmHdgdMcOaPlGEREpGRSuClrchfTjJ0G2Zk2NY0K9qFN9fIA/BKrlcJFRKRkUrgpa+p0B/9wSD8Jcb/Z3HxAi9xLU0e0HIOIiJRICjdljZuHdVI/KNSlqZ6NIvByN7HvRBrbj5axdbpERKRUULgpi1pcXI5h33I4c8impgHeHtzewLocw6zNR+xdmYiISJEp3JRFITWgeifAApu/tbn5wIuXpn7dcoxsLccgIiIljMJNWdXy4sDizd9BTrZNTW+pHUp5P09Onsvkjz0nHVCciIhI4SnclFX17gCfEEhNgL1LbGrq4WaiT1MtxyAiIiWTwk1Z5e4Fze6zbm+yfWBx7nIMi/9K5FyGbWd+REREHKnQ4Wbv3r0sWrSI8+fPA+i24NKoxcVFSfcsghTb5q1pUimIGqF+XMgys2BbggOKExERKRybw82pU6fo2rUrderUoVevXiQkWD/YHnnkEZ577jm7FygOFFoXqrQFixk2f29TU8MwGHjx7I0uTYmISElic7h59tlncXd3Jz4+Hl9f37z9gwcPZuHChXYtTopB7ozFm78Bs213PvVrZg03a/efIiH5vL0rExERKRSbw83ixYsZP348lSpVyre/du3aHDpk25wpUgI06AdeQXA2HvYvt6lp5RBfbqoegsWi5RhERKTksDncpKWl5Ttjk+v06dN4eXnZpSgpRp6+0GSQdbsQMxbnDiyeHXNU465ERKREsDnc3HLLLXzzzTd53xuGgdls5p133qFz5852LU6KSe6cN7t+g3MnbGraq3Eknu4m4pJS+StByzGIiIjzudva4J133qFLly5s3LiRzMxMXnzxRXbs2MHp06dZvXq1I2oUR4toDFEt4FgMbJ8JbZ4scNMgHw+61g/jt22JzI45SsOoIAcWKiIicmM2n7lp1KgRu3fvpkOHDvTr14+0tDQGDhzI5s2bqVmzpiNqlOLQ+G7r1x2zbW46oLl1/NUvW46RY9alKRERcS6bz9wABAUF8corr9i7FnGmhv1h0Wg4/CckH4WgigVu2qlOKOV8PTiRmsHqvSfpWCfUcXWKiIjcgM3hZuXKldd9vmPHjoUuRpwoMMo65038WvhrDrQdXuCmnu4m7mgSxbd/HmL25qMKNyIi4lQ2h5tbb731in2GYeRt5+TkFKkgcaKGA63hZsdsm8INwIAWFfn2z0Ms3J7If/pn4+dVqJOCIiIiRWbzmJszZ87kexw/fpyFCxfSunVrFi9e7Igapbg06AsYcGSDdd4bGzSvHEz1Cn6cz8ph8V+JjqlPRESkAGwON0FBQfkeFSpU4Pbbb2f8+PG8+OKLjqhRiktABFTrYN22cWCxYRj0vzhj8awYLccgIiLOY7dVwcPDw4mLi7NXd+IsDftbvxbqrilruFm99yTHUy7YsSgREZGCs3lgxNatW/N9b7FYSEhI4O2336ZZs2b2qkucpX4/+O0FOLYZTu+HkBoFblqlvC8tq5Zj06Ez/BJ7jMc6FrytiIiIvdgcbpo1a4ZhGFdMtd+mTRumTJlit8LESfxDodotcGAF7JgDt4yyqfmA5hXZdOgMszYfVbgRERGnsDncHDhwIN/3JpOJ0NBQvL297VaUOFmjgRfDzSybw80dTSIZ++tf7ExIYVdiCvUiAh1UpIiIyNXZPOamatWq+R6VK1dWsHE19fqA4QaJ2+DkXpuaBvt60rmedZ6b2Zs1sFhERIpfgc7cfPTRRwXu8Omnny50MVJC+JWHGrfCvmXWgcWdXrCp+YDmlVi0I4lfNh/jxe71cDMZN24kIiJiJwUKN++//36BOjMMQ+HGVTQaWOhw07leKEE+HiSmXODP/adoX6uCg4oUERG5UoHCzeXjbKQMqNcbfh0Jx3fAiTgIrVvgpl7ubvRuEsm0dfHMijmqcCMiIsXKbvPciIvxKQc1b7NuF2LOm4EX57xZuD2B85lakkNERIpPoRYAOnLkCHPnziU+Pp7MzMx8z02YMMEuhUkJ0HAA7FkE22dBp5fAKPjYmZZVy1E5xIfDp8+z+K9E+jUr+CrjIiIiRWFzuFm2bBl9+/alRo0a7Nq1i0aNGnHw4EEsFgstWrRwRI3iLPV6gZsnnIyD4zshvEGBmxqGwYBmFfno973M3nxU4UZERIqNzZelRo8ezfPPP8+2bdvw9vZm5syZHD58mE6dOnH33Xc7okZxFu8gqNXVur1jls3NB7SoBMAfe05yIjXDnpWJiIhck83hZufOnQwdOhQAd3d3zp8/j7+/P2PHjmX8+PF2L1CcrOFA69cds+GyWalvpHoFP5pVDibHbOHXLcccUJyIiMiVbA43fn5+eeNsIiMj2bdvX95zJ0+etF9lUjLU7QHu3nBqr3VSPxsNbGG9HKUJ/UREpLjYHG7atGnDqlWrAOjVqxfPPfcc//3vf3n44Ydp06aN3QsUJ/MKgNq3W7cLcdfUHU2icDcZbDuazN7jqXYuTkRE5Eo2h5sJEyZw8803A/DGG2/QpUsXZsyYQbVq1fjyyy/tXqCUAA0HWL/umGXzpakQP09urWtdjmFWjM7eiIiI49l8t1SNGn+v9Ozn58fkyZPtWpCUQHV6gLsPnDkIxzZDRdvuihvQvBJLdx7nl9hjPN+tLiYtxyAiIg5k85mbRx99lOjoaAeUIiWWpx/U6W7dLsSlqS71wwjwcufo2fNsij9j5+JERETyszncnDhxgh49elC5cmVeeOEFtmzZ4oi6pKRplHvX1BybL015e7jRvVEEAHNjddeUiIg4ls3h5pdffiEhIYExY8awYcMGWrRoQcOGDXnrrbc4ePCgA0qUEqHW7eDhB8nxcHSTzc37NI0C4LdtCWTnmO1dnYiISJ5CrS1Vrlw5Hn/8caKjozl06BDDhg3j22+/pVatWvauT0oKT1+o29O6vd32Cf3a1yxPeT9PTqVlsnrfKTsXJyIi8rciLZyZlZXFxo0bWbduHQcPHiQ8PNxedUlJlHvX1F9zwGzb2Rd3NxO9GkcCaEI/ERFxqEKFm+XLl/PYY48RHh7OsGHDCAwMZN68eRw5csTe9UlJUqsreAVCylE4st7m5n2bWS9NLdqeyIUsrRQuIiKOYXO4qVixIr169eLkyZN8/vnnJCUlMWXKFLp06YJhw6rRUgp5eEPdXtbtQtw11bJKOSKDvEnNyCY67oSdixMREbGyOdy8/vrrJCQkMHv2bO666y68vLwcUZeUVHkT+s0Bs21nX0wmI29g8a9bdWlKREQcw+Zw89hjjxEcHOyAUqRUqHkbeAXBuUSI/9Pm5n0vhptlO5NIy8i2d3UiIiJFG1AsZZC7J9S/w7q9w/a7phpGBVK9gh8Xssws+SvJzsWJiIgo3EhhNLw4od9fv9h8acowLrk0pbumRETEARRuxHY1OoFPOUg7AQdX2dy8b1PrLeEr95zgbHqmvasTEZEyTuFGbOfmAfX7WLcLcddUrbAA6kcGkpVjYcH2RDsXJyIiZZ3N4ebrr79m/vz5ed+/+OKLBAcH065dOw4dOmTX4qQEy71raudcyLF9YHBfXZoSEREHsTncvPXWW/j4+ACwdu1aJk6cyDvvvEOFChV49tln7V6glFDVOoJveUg/BQdX2tz8jibWS1Nr95/ieMoFe1cnIiJlmM3h5vDhw3lrSM2ZM4c777yTxx9/nHHjxvHHH3/YvUApodzcoX5f63Yh1pqqHOJLiyrBWCwwb2uCnYsTEZGyzOZw4+/vz6lT1oUPFy9ezO233w6At7c358+ft291UrI1unjX1M5fIdv2gcF9NaGfiIg4gM3h5vbbb+fRRx/l0UcfZffu3fTqZZ2Of8eOHVSrVs3e9UlJVrU9+IXBhbNwYIXNzXs1icRkwOb4sxw+nW7/+kREpEyyOdxMnDiRtm3bcuLECWbOnEn58uUB2LRpE/fee6/dC5QSzOQGDfpZtwtxaSoswJu2Na3//czVwGIREbETw2KxWJxdRHFKSUkhKCiI5ORkAgMDnV1O6XdwNUztZV2S4YU94G7bWmMzNsTz0sxt1IsIYOHIjg4qUkRESjtbPr9tPnOzcOFCVq36e+K2iRMn0qxZM+677z7OnDlje7VSulVpC/4RkJEM+363uXmPhpF4uBnsSkxld1KqAwoUEZGyxuZw88ILL5CSkgLAtm3beO655+jVqxcHDhxg1KhRdi9QSjiTCRr2t24XYkK/IF8POtUJBTTnjYiI2IfN4ebAgQM0aNAAgJkzZ3LHHXfw1ltvMXHiRBYsWGBTXytXrqRPnz5ERUVhGAZz5sy5YZvo6GhatGiBl5cXtWrVYurUqbb+CGJvuWtN7foNsmyfsyZ3ram5W45Rxq6SioiIA9gcbjw9PUlPt97ZsnTpUrp16wZASEhI3hmdgkpLS6Np06ZMnDixQMcfOHCA3r1707lzZ2JjYxk5ciSPPvooixYtsu2HEPuq1BoCK0FmKuxdanPz2xuE4+PhxqFT6Ww7muyAAkVEpCxxt7VBhw4dGDVqFO3bt2f9+vXMmDEDgN27d1OpUiWb+urZsyc9e/Ys8PGTJ0+mevXqvPfeewDUr1+fVatW8f7779O9e3ebXlvsKPfS1NpPYMcsqH+HTc19Pd3pUj+MeVsTmBt7jCaVgh1SpoiIlA02n7n55JNPcHd35+eff2bSpElUrFgRgAULFtCjRw+7F3iptWvX0rVr13z7unfvztq1a6/ZJiMjg5SUlHwPcYDcS1NxCyHT9jlrcif0m7c1AbNZl6ZERKTwbD5zU6VKFebNm3fF/vfff98uBV1PYmIi4eHh+faFh4eTkpLC+fPn89a8utS4ceN44403HF5bmVexBQRXgbPxsGfx34OMC6hT3VACvN1JTLnAhoOnublGecfUKSIiLs/mMzcAOTk5zJw5k//85z/85z//Yfbs2eTk5Ni7NrsYPXo0ycnJeY/Dhw87uyTXZBh/rxReiLumvNzd6NEwAtCEfiIiUjQ2h5u9e/dSv359hg4dyqxZs5g1axb3338/DRs2ZN++fY6oMU9ERARJSUn59iUlJREYGHjVszYAXl5eBAYG5nuIg+SGm92LIOOczc37NrNemvptWwJZOWZ7ViYiImWIzeHm6aefpmbNmhw+fJiYmBhiYmKIj4+nevXqPP30046oMU/btm1ZtmxZvn1Lliyhbdu2Dn1dKaDIZlCuOmSfhz2238HWtkZ5Kvh7ciY9i9V7T9q/PhERKRNsDjcrVqzgnXfeISQkJG9f+fLlefvtt1mxwrbFE8+dO0dsbCyxsbGA9Vbv2NhY4uPjAeslpaFDh+Yd/+STT7J//35efPFFdu3axaeffsqPP/7Is88+a+uPIY5w6aWpQqw15e5molfjSECXpkREpPBsDjdeXl6kpl45Tf65c+fw9PS0qa+NGzfSvHlzmjdvDsCoUaNo3rw5r776KgAJCQl5QQegevXqzJ8/nyVLltC0aVPee+89/ve//+k28JKk0cW7pvYsgQzbl1PIvWtq8Y4kLmSVzHFcIiJSstm8cObQoUOJiYnhyy+/5KabbgJg3bp1PPbYY7Rs2bLEzxishTMdzGKBT1rBqb0w8AtoMsim5mazhVveWc7Rs+eZfH8LejSKdFChIiJSmjh04cyPPvqImjVr0rZtW7y9vfH29qZ9+/bUqlWLDz/8sNBFi4swjL/nvCnEpSmTyeCOJro0JSIihWfzPDfBwcH88ssv7Nmzh127dgHWmYJr1apl9+KklGo4AFa+A/uWwfmz4BNsU/M+TaP4bOV+lu08TuqFLAK8PRxSpoiIuCabw02u2rVrU7t2bXvWIq4ivAGE1oMTuyDuN2h2n03NG0YFUiPUj/0n0ljyVxIDW9i2rIeIiJRtBQo3o0aNKnCHEyZMKHQx4kIaDoDocdYJ/WwMN4Zh0KdJFB8u28OvW44p3IiIiE0KFG42b95coM4MwyhSMeJCcsPNvt8h/TT4hty4zSX6NrOGmz/2nORMWibl/Gy7E09ERMquAoWb5cuXO7oOcTWhdSGsIRzfAbvmQ4sHbGpeM9SfhlGB7DiWwm/bExhyc1UHFSoiIq6mUGtLiRRIo9y1pmy/awr+nvPmV901JSIiNlC4EcfJvSV8/wpIO2Vz8zsuhpt1B06TmHzBnpWJiIgLU7gRxylfEyKagCUHds61uXnFYB9aVS2HxQLztursjYiIFIzCjThW7nIMO2YXqnnuSuG/bk2wV0UiIuLiFG7EsRr0t349+AecO25z856NIjEZsOXwWQ6dSrNvbSIi4pIUbsSxQqpDVAuwmAt1aSo0wIv2tSoAGlgsIiIFo3Ajjtfw4l1T2wt3aapP3l1TujQlIiI3pnAjjpcbbg6thrPxNjfv3jACTzcTcUmpxCWm2rk4ERFxNQo34njBlaF6R8ACm762uXmQjwed6oYCMHfLUTsXJyIirkbhRopHq0esX2O+gexMm5v3veTSlMVisWdlIiLiYhRupHjU6w3+EZB2HHbNs7l5l/ph+Hi4EX86nS1Hkh1QoIiIuAqFGykebh7QYqh1e+MUm5v7erpze4NwAObG6q4pERG5NoUbKT4tHwTDZJ3z5kSczc1zL03N23qMHLMuTYmIyNUp3EjxCaoEdXpYtwtx9uaWOhUI9HbneGoG6w+ctnNxIiLiKhRupHjlDiyO/QEy021q6uXuRs9GkQDM1YR+IiJyDQo3Urxq3gblqkFGMmyfaXPz3LWmFmxPIDPbbOfiRETEFSjcSPEymaDlQ9btjV/a3LxNjfJU8PfibHoWq/eetHNxIiLiChRupPg1vx/cPOHYZjgaY1NTN5PBHU10aUpERK5N4UaKn1+Fv1cLL8TZm9y1phbvSOR8Zo4dCxMREVegcCPO0eph69dtM+H8GZuatqgSTMVgH9Iyc1ged9wBxYmISGmmcCPOUaUNhDWA7POwZYZNTQ3DyDt7own9RETkcgo34hyG8ffZm41TwMb1onIn9Ps97jgpF7LsXZ2IiJRiCjfiPE0Gg4cfnIyDg6tsalo/MoCaoX5kZptZsiPJQQWKiEhppHAjzuMdCE0GWbdtHFhsGAZ9m1YEdNeUiIjkp3AjztX64ozFO3+FVNvOwPRpar0lfNXek5w6l2HvykREpJRSuBHnimgMlW4CczZs/sampjVC/WlUMZAcs4UF2xMdVKCIiJQ2CjfifLkDizd9DWbb5q3JHVisS1MiIpJL4Uacr+EA8CkHyYdhzxKbmt7RxBpuNhw8TULyeUdUJyIipYzCjTifhzc0G2LdtnFgcVSwD62rlcNigXlbEhxQnIiIlDYKN1Iy5F6a2rMEzhy0qWnupamZMUew2DhfjoiIuB6FGykZyteEGp0BC2yaalPTvk0r4uvpxq7EVFZppXARkTJP4UZKjtzbwmO+heyC39od5OvBoFaVAfh85X5HVCYiIqWIwo2UHHV6QkAkpJ+0zntjg0c6VMdkwB97TrIzIcVBBYqISGmgcCMlh5s7tHjQur1xik1NK4f40rOxdVK///1xwN6ViYhIKaJwIyVLywfBcINDq+H4TpuaPnZLDQDmbjlKYvIFR1QnIiKlgMKNlCyBUVC3p3XbxrM3zSoHc1O1ELJyLExdc9D+tYmISKmgcCMlT+7A4i3TIeOcTU0f62g9e/P9ukOcy8i2d2UiIlIKKNxIyVP9VgipARkpsP1nm5p2qRdGjQp+pF7IZsaGww4pT0RESjaFGyl5TKa/J/Xb8CXYMDGfyWTw6MWxN1NWHSA7x+yICkVEpARTuJGSqdkQcPOCxK1wNMampgNbVKS8nydHz57XauEiImWQwo2UTL4h1gU1web1prw93BjathpgndRPSzKIiJQtCjdScuUOLN4+E9JP29T0/jZV8HI3se1oMusO2NZWRERKN4UbKbkqtYbwxpB9Abb8YFPT8v5e3NWyEgBfaEkGEZEyReFGSi7DgNYXBxZvnGLTwGKwLslgGLBs13H2Hk91QIEiIlISKdxIydZ4EHgGwKm9cGCFTU1rhPpze/1wQEsyiIiUJQo3UrJ5+UPTwdZtG2csBnj84qR+s2KOciK14CuNi4hI6aVwIyVf7pw3u+ZDqm23dresWo7mVYLJzDHz7dqD9q9NRERKHIUbKfnCG0LlNmDOhphvbGpqGAaPX5zU75s/D3E+M8cRFYqISAmicCOlQ+5t4ZumQo5ta0Z1axhBlRBfzqZn8fMmLckgIuLqFG6kdGjQD3zLQ8pR2LPIpqZuJoNHOlQH4H+rDpBj1qR+IiKuTOFGSgd3L2h+v3V7g20zFgPc3aoSQT4eHDqVzpK/tCSDiIgrU7iR0qPlQ4AB+5bBadtu7fb1dOeBNlUB65IMIiLiuhRupPQIqQ61uli3N31lc/Oh7ari6WYiJv4smw5pSQYREVelcCOlS+5t4Zu/g2zb5q0JC/BmQPOKAHyxUpP6iYi4KoUbKV1qd4fAipB+Cv76xebmj95iHVi86K9EDp5Ms3d1IiJSApSIcDNx4kSqVauGt7c3N998M+vXr7/msVOnTsUwjHwPb2/vYqxWnMrNHVoOs24XYmBx7fAAOtcNxWKBL1fp7I2IiCtyeriZMWMGo0aN4rXXXiMmJoamTZvSvXt3jh8/fs02gYGBJCQk5D0OHTpUjBWL07UYCiZ3OPwnJO2wufljF5dk+GnTYU6nZdq7OhERcTKnh5sJEybw2GOP8dBDD9GgQQMmT56Mr68vU6Zcex0hwzCIiIjIe4SHhxdjxeJ0ARFQr7d1uxDrTbWtUZ5GFQO5kGXmuz8VjEVEXI1Tw01mZiabNm2ia9eueftMJhNdu3Zl7dq112x37tw5qlatSuXKlenXrx87dtj+17uUcq0uzli8ZQZknLOpqWEYPHZxSYav1xzkQpaWZBARcSVODTcnT54kJyfnijMv4eHhJCZefaK1unXrMmXKFH755Re+++47zGYz7dq148iRI1c9PiMjg5SUlHwPcQHVO0L5WpCZCtt+tLl5r8aRVAz24VRaJrM3H3VAgSIi4ixOvyxlq7Zt2zJ06FCaNWtGp06dmDVrFqGhoXz22WdXPX7cuHEEBQXlPSpXrlzMFYtDGMbft4VvmAIW25ZU8HAz8VD7agD874/9mLUkg4iIy3BquKlQoQJubm4kJSXl25+UlERERESB+vDw8KB58+bs3bv3qs+PHj2a5OTkvMfhw1o40WU0vRfcvSFpGxzZYHPze26qQoC3O/tOpLE87toD2EVEpHRxarjx9PSkZcuWLFu2LG+f2Wxm2bJltG3btkB95OTksG3bNiIjI6/6vJeXF4GBgfke4iJ8Q6DRndbtQtwW7u/lzn03VQG0JIOIiCtx+mWpUaNG8cUXX/D111+zc+dO/vGPf5CWlsZDDz0EwNChQxk9enTe8WPHjmXx4sXs37+fmJgY7r//fg4dOsSjjz7qrB9BnCl3YPGO2ZBu+5IKw9pXw91ksO7AabYcPmvf2kRExCmcHm4GDx7Mu+++y6uvvkqzZs2IjY1l4cKFeYOM4+PjSUhIyDv+zJkzPPbYY9SvX59evXqRkpLCmjVraNCggbN+BHGmii0gsinkZEDs9zY3jwzyoW/TKAC++ENnb0REXIFhsdg4ErOUS0lJISgoiOTkZF2ichWbvoZfnwb/cBi+HnyCbWr+17EUen30ByYDVrzQmcohvo6pU0RECs2Wz2+nn7kRKbImg623hZ9LgmVv2Ny8QVQgt9SugNkCU1ZrSQYRkdJO4UZKPw9v6POhdXvjFDh07QkgryV3Ur8ZGw6TnJ5lz+pERKSYKdyIa6jWAZo/YN3+9RnIzrCp+S21K1AvIoD0zBymrY93QIEiIlJcFG7Eddw+FvxC4WQcrPrApqaGYfDoxbM3X60+QGa22QEFiohIcVC4EdfhGwI9x1u3/3gXTuy2qXnfplGEB3pxPDWDuVuOOaBAEREpDgo34loaDoTa3SAn03p5ylzwMzCe7iaGtasOwBcr91PGbiQUEXEZCjfiWgwDer8HHr4QvwY2f2tT8/turoKfpxtxSams3HPSQUWKiIgjKdyI6wmuArf927q9ZAykJl3/+EsE+XgwuLV1SYb/aVI/EZFSSeFGXNNNT0BkM7iQDAtfsqnpQ+2r4WYy+GPPSf46luKY+kRExGEUbsQ1ublD34/AcLOuOxW3sMBNK4f40rORdVV6nb0RESl9FG7EdUU2hbbDrdvzn4OMcwVu+nhH623hc7ccIyH5vCOqExERB1G4Edd268sQXBVSjsDy/xa4WZNKwdxcPYRss4Wpqw86rj4REbE7hRtxbZ5+cMcE6/a6yXB0U4Gb5p69mbYuntQLWpJBRKS0ULgR11erKzQeBBYzzH0GcgoWVDrXDaNmqB+pGdnM2HDYwUWKiIi9KNxI2dBjHPiUg6RtsHZigZqYTEbegppfrT5IVo6WZBARKQ0UbqRs8KsA3d+ybke/DacLdhdU/+YVqeDvydGz55kVc8SBBYqIiL0o3EjZ0fReqN4Rss/DvFFQgOUVvD3ceKSD9ezN63P/0rw3IiKlgMKNlB2GAXd8AO7esH85bP2xQM0eu6U6t9SuwPmsHB77ZiOn0zIdW6eIiBSJwo2ULeVrQqcXrduLRkPaqRs2cXcz8cm9Laha3pejZ88z/PsYjb8RESnBFG6k7Gn3NIQ1hPRTsPjfBWoS5OvBF0Nb4efpxtr9p/jv/J0OLlJERApL4UbKHjcP69IMGLBlGuxbXqBmdcIDmDC4GQBT1xzkx426PVxEpCRSuJGyqVIruOlx6/a8kZCZXqBm3RtGMLJrbQD+PXs7MfFnHFSgiIgUlsKNlF1dxkBgRThzEFa+U+BmT99Wm24NwsnMMfPkt5tISrnguBpFRMRmCjdSdnkFQO/3rNurP4LE7QVqZjIZTBjcjDrh/hxPzeCJbzdxISvHgYWKiIgtFG6kbKvbExr0A0sO/Po0mAsWUvy93PliaCuCfDyIPXyWf8/ZjqUA8+aIiIjjKdyI9HwHvIKsi2pu+F+Bm1Ut78cn9zXHZMDPm44wdc1Bx9UoIiIFpnAjEhABt79u3V42FpILvszCLbVD+Vev+gD8Z/5O1uw96YACRUTEFgo3IgAthkGVtpB5DuY/V6ClGXI90qE6A5pXJMdsYfi0GA6fLtidVyIi4hgKNyIAJhP0+RBMHrB7Ifz1S4GbGobBuIGNaVIpiDPpWTz2zUbSM7MdWKyIiFyPwo1IrtC6cMtz1u0FL8L5swVu6u3hxmcPtKSCvxe7ElN54aetGmAsIuIkCjcil7plFJSvDeeSYOnrNjWNDPJh8v0t8HAzmL8tgU+j9zmmRhERuS6FG5FLuXtZL08BbPoKDq21qXmraiGM7dcIgHcXx7FsZ5K9KxQRkRtQuBG5XLX20OJB6/avT0N2hk3N772pCve3qYLFAs9Mj2Xv8XMOKFJERK5F4Ubkam4fC/7hcHI3rHrf5uav3tGQm6qFcC4jm8e/2Ujy+SwHFCkiIlejcCNyNT7B0HO8dfuP9+BEnE3NPd1NfHp/C6KCvNl/Mo1npm8mx6wBxiIixUHhRuRaGvSHOj0gJxN+HQlms03NK/h78fnQVnh7mIiOO8G7i20LSCIiUjgKNyLXYhjQ613w9If4NbDwJZvH3zSqGMT4O5sAMCl6H3O3HHNEpSIicgmFG5HrCa4M3f5j3V7/OfyvK5zcY1MX/ZpV5IlONQB48ectbD+abO8qRUTkEgo3IjfS6iG4dzr4hEDiVvisI2z62qYlGl7sXo9OdUK5kGXmiW83ceqcbWeARESk4BRuRAqibk/4xxqocStkpVtvEf/pQTh/pkDN3UwGH93TnOoV/Dh69jz/+D6GrBzbxvCIiEjBKNyIFFRgJNw/23qbuMnduv7UpA5waE2Bmgf5evDF0Jb4e7mz/sBp3pz3l4MLFhEpmxRuRGxhMkH7Z+CRJRBSA1KOwNTesPwtyLnxYpm1wgJ4f3AzAL5Ze4jp6+MdXLCISNmjcCNSGBVbwBMrodkQsJhhxXiY2gvOHLph09sbhPPc7XUAGPPLdjYdOu3oakVEyhSFG5HC8gqA/p/CnV+CVyAcXgeTb4HtM2/YdMRttejZKIKsHAtPfhdDYvKFYihYRKRsULgRKarGd8GTq6DSTZCRDD8/DHOegozUazYxDIN3725KvYgATqRm8Pi3Gzl4Mq0YixYRcV0KNyL2UK4qPLQAOr4Ihgliv7feMn405ppN/Lzc+fyBVgT7erD1SDK3vRfNyOmb2ZN07VAkIiI3ZlgsNkzW4QJSUlIICgoiOTmZwMBAZ5cjrujgapj1uHWwsckdbhsD7Z62Dka+irjEVN5esJPlcScA68TIPRtFMLxzLRpGBRVn5SIiJZYtn98KNyKOcP4MzH0ads61fl/jVhjwGQREXLPJ9qPJfPL7XhbuSMzb17V+GCNuq02zysGOrVdEpIRTuLkOhRspNhYLxHwDC1+2TvznWx76TbROCHgdcYmpTFy+l3lbj5G7kPgttSvwz9tqc1P1kGIoXESk5FG4uQ6FGyl2J3bDzIchcZv1+5set04E6OFz3Wb7T5zj0+h9zN58lJyLKeem6iH887ZadKhVAcMwHF25iEiJoXBzHQo34hTZGbD0DfhzovX7sAZw1xQIq3/DpodPpzN5xT5+2niEzItLNjSrHMw/b6vFbfXCFHJEpExQuLkOhRtxqj1LYc6TkHYC3L2h+3+h1SPWUcQ3kJB8ns9X7mfaungysq0hp35kIP+8rRY9GkZgMinkiIjrUri5DoUbcbpzx2HOP2DvUuv3dXtB30/Ar3yBmp9IzeDLVQf4du1B0jJzAKgV5s+IzrW4o0kk7m6a4UFEXI/CzXUo3EiJYDbDusmw9DXIybQONq7XG2p3t95Z5eV/wy7OpGXy1ZqDfLX6AKkXrOtaVSvvy1O31qJ/84p4uivkiIjrULi5DoUbKVEStsDMR+Hk7r/3uXlC1fZQpzvU7gbla163i5QLWXy79hD/+2M/Z9KzAKgY7MOTnWpwd6vKeHu4OfInEBEpFgo316FwIyVOdiYcXAm7F8OeRXDmYP7ny9eyntGp0w2qtAN3z6t2k56ZzbR18Xy2cj8nUjMACAvw4vGONbj3pir4ebk7+AcREXEchZvrULiREs1igZN7rCFn9yKIXwvm7L+f9/S3XrbKPatzlUkBL2Tl8OPGw0yO3sexiwtyergZNIgKokWVYFpWLUeLKuWICr7+regiIiWJws11KNxIqXIhBfYvv3hWZzGkHc//fGTTi2d1ukNUi3xLPGRmm5m9+QiTV+znwFUW5YwM8qZFlXK0qFqOFlWCaRgVpHE6IlJiKdxch8KNlFpmMyTEwp4l1jM7R2OAS96+vhWgVlfr5auaXcAnGACLxcKRM+eJiT9DzKEzxMSf5a+ElLyJAXN5uptoUjGIllXL0bxKOVpUDSYswLvYfjwRketRuLkOhRtxGedOwN4l1stX+36HjJS/nzPcoEob66WrOt0htF6+uXTSM7PZeiSZTYfOsDn+DJsOnckbjHypyiE+tKhSLu9SVr2IAN1qLiJOoXBzHQo34pJysuDwOmvQ2bMYTuzK/7xnAARGQkAkBEZd9jUSS0AkBy/4s+lwSt4ZnrikVC7/v4OPhxtNKwflhZ0WVcpRzu/qA5xFROxJ4eY6FG6kTDhzyBpydi+Cg39A9oUbtzHcwD88LwRl+kZw1BzM7vQAYs76sOa4J/svBJBG/oHINSr4UbGcD8G+npTz9cj7Ws7Xk6CLX3P3B3q7a7kIESmUUhduJk6cyP/93/+RmJhI06ZN+fjjj7npppuuefxPP/3EmDFjOHjwILVr12b8+PH06tWrQK+lcCNlTtYFOBsPqccgJeGSrwmQcsz69VwSWMwF687dj9Om8hzJCeZgZhCJlnKkWnxJx4vzeJFu8b5k2/o1dzvD5I2Xtx/Bfp55oSfI52IY8vMk+GIYCva5GJL8PAjy8cDb3U3LS4iUcaUq3MyYMYOhQ4cyefJkbr75Zj744AN++ukn4uLiCAsLu+L4NWvW0LFjR8aNG8cdd9zBtGnTGD9+PDExMTRq1OiGr6dwI3IVOdnWO7HyhZ9jkJr4dwBKSYDM1CK/lNlicB5PawCyeJGONxfwJN3iddWAdN7iSTbuWEzWB24eGCYPcPMAN3dMbh4Y7h6Y3C4+3D0xuXvg5u6Jm4cnbu4euLt74O7hhbuHJ+4eHnh4eOHh6Wl9eHhicnPHzeSGm5sb7m4m3EwG7qaLX92Mi98b+febDNzc8u83GejMlIiDlKpwc/PNN9O6dWs++eQTAMxmM5UrV+af//wnL7/88hXHDx48mLS0NObNm5e3r02bNjRr1ozJkyff8PUUbkSKICP16gEoIxUy0yDrPGSlX9xOh8x0yLq4vyCXxkqAHIuBGRNmTORgwoxx/W1L/v0WTJgNExYMzIYbZkyAgQUDy8X91q9gwZS3j9znLj4PJiyGgQUTGFx83roP42KfF79aB4sb1uew9mMY1v157THlHYdxadtLnst7Pv+x+b5iYJgufT3j70B3WZ/GJX0beT+DcfGQ3Nfg4lfr8RbDwMDI68e4+Jy1PvK9Tr7+8/r5u1bjkv6NvJ+PfM9d+rsyrujb+tVy2c9t7fKy7dy683bnPme6rM0l9VysJd/v49If0zD9fXzez8dlP0felnXDZMp9tbzaDS7p99LwbVxaY/768g7J7c1k5Ps+fz/52xsYeHj7EFmxKvZky+e3U6cszczMZNOmTYwePTpvn8lkomvXrqxdu/aqbdauXcuoUaPy7evevTtz5sy56vEZGRlkZGTkfZ+SknLV40SkALwCIDQAQuvY3tackz/wZKZbv8/bd2kosn61ZKaTnZFOTnYm5uwszDmZWLKzsORkYc7JghzrtsWcDTlZGLlfLdkY5mwMcxYmczYmSzYmSw4mSzZulmzcyLlmmW6G5eLz1z4mH1tO1Fgu+yriona51yfy33867fWdGm5OnjxJTk4O4eHh+faHh4eza9euq7ZJTEy86vGJiYlXPX7cuHG88cYb9ilYRArP5GYNR14BBW5iAB4XH3ZlsVhnfs7Jsn61mK0Pc87F7ZzLts359lssOZhzzOTkZGE255CTk4M5O5scsxlzTjY5OdmYzWZysq19W8xmzBe/WvK+5mCxWLCYzZC733Lp83/vx5L7vcXaznzJPosFCxYMsxkLFuvPdsn+y783LGasu637wHwxbF38Pt9x1vNLV3y9pB+wYLFYzzld9VgL13zOuPj839u5/V/cd+lzlxybey6Fi69rgbz2uedjcvv5u18ueS7//r9fw8q4bL+F3BMyF7ctl/bPJW0v7Zu/ty977u88bLnk9S6t8Rp9Xnzeku/73D6usf+yJG1c9rNeuv/y7fy5/cr+rnec2c25d1G6/GIzo0ePznemJyUlhcqVKzuxIhFxOsO4OGancLHJANwuPkTkSg2c/PpODTcVKlTAzc2NpKSkfPuTkpKIiLhyzRyAiIgIm4738vLCy8vLPgWLiIhIiefUqUY9PT1p2bIly5Yty9tnNptZtmwZbdu2vWqbtm3b5jseYMmSJdc8XkRERMoWp1+WGjVqFA8++CCtWrXipptu4oMPPiAtLY2HHnoIgKFDh1KxYkXGjRsHwDPPPEOnTp1477336N27N9OnT2fjxo18/vnnzvwxREREpIRwergZPHgwJ06c4NVXXyUxMZFmzZqxcOHCvEHD8fHxmC5Z6bhdu3ZMmzaNf//73/zrX/+idu3azJkzp0Bz3IiIiIjrc/o8N8VN89yIiIiUPrZ8fmt5XxEREXEpCjciIiLiUhRuRERExKUo3IiIiIhLUbgRERERl6JwIyIiIi5F4UZERERcisKNiIiIuBSFGxEREXEpTl9+objlTsickpLi5EpERESkoHI/twuysEKZCzepqakAVK5c2cmViIiIiK1SU1MJCgq67jFlbm0ps9nMsWPHCAgIwDAMu/adkpJC5cqVOXz4sEPWrXJk/6W5dkf3X5prL+39l+baS3v/pbn20t5/aa7dkf1bLBZSU1OJiorKt6D21ZS5Mzcmk4lKlSo59DUCAwMduiinI/svzbU7uv/SXHtp7780117a+y/NtZf2/ktz7Y7q/0ZnbHJpQLGIiIi4FIUbERERcSkKN3bk5eXFa6+9hpeXV6nrvzTX7uj+S3Ptpb3/0lx7ae+/NNde2vsvzbUXR/8FUeYGFIuIiIhr05kbERERcSkKNyIiIuJSFG5ERETEpSjciIiIiEtRuLGDlStX0qdPH6KiojAMgzlz5tit73HjxtG6dWsCAgIICwujf//+xMXF2a3/SZMm0aRJk7zJltq2bcuCBQvs1v+l3n77bQzDYOTIkXbp7/XXX8cwjHyPevXq2aXvXEePHuX++++nfPny+Pj40LhxYzZu3GiXvqtVq3ZF/YZhMHz4cLv0n5OTw5gxY6hevTo+Pj7UrFmTN998s0DrshREamoqI0eOpGrVqvj4+NCuXTs2bNhQqL5u9B6yWCy8+uqrREZG4uPjQ9euXdmzZ4/d+p81axbdunWjfPnyGIZBbGys3erPysripZdeonHjxvj5+REVFcXQoUM5duyYXWp//fXXqVevHn5+fpQrV46uXbuybt06u9R+uSeffBLDMPjggw/s1v+wYcOueA/06NHDbrXv3LmTvn37EhQUhJ+fH61btyY+Pt4u/V/t/WsYBv/3f/9nl/7PnTvHiBEjqFSpEj4+PjRo0IDJkycXqO+C9J+UlMSwYcOIiorC19eXHj16FPh9VZDPpgsXLjB8+HDKly+Pv78/d955J0lJSQWuvygUbuwgLS2Npk2bMnHiRLv3vWLFCoYPH86ff/7JkiVLyMrKolu3bqSlpdml/0qVKvH222+zadMmNm7cyG233Ua/fv3YsWOHXfrPtWHDBj777DOaNGli134bNmxIQkJC3mPVqlV26/vMmTO0b98eDw8PFixYwF9//cV7771HuXLl7NL/hg0b8tW+ZMkSAO6++2679D9+/HgmTZrEJ598ws6dOxk/fjzvvPMOH3/8sV36f/TRR1myZAnffvst27Zto1u3bnTt2pWjR4/a3NeN3kPvvPMOH330EZMnT2bdunX4+fnRvXt3Lly4YJf+09LS6NChA+PHj7e59hv1n56eTkxMDGPGjCEmJoZZs2YRFxdH37597VJ7nTp1+OSTT9i2bRurVq2iWrVqdOvWjRMnTtil/1yzZ8/mzz//JCoqqkD92tJ/jx498r0XfvjhB7v0vW/fPjp06EC9evWIjo5m69atjBkzBm9vb7v0f2nNCQkJTJkyBcMwuPPOO+3S/6hRo1i4cCHfffcdO3fuZOTIkYwYMYK5c+cWuX+LxUL//v3Zv38/v/zyC5s3b6Zq1ap07dq1QJ8vBflsevbZZ/n111/56aefWLFiBceOHWPgwIEFqr3ILGJXgGX27NkO6//48eMWwLJixQqHvUa5cuUs//vf/+zWX2pqqqV27dqWJUuWWDp16mR55pln7NLva6+9ZmnatKld+rqal156ydKhQweH9X+5Z555xlKzZk2L2Wy2S3+9e/e2PPzww/n2DRw40DJkyJAi952enm5xc3OzzJs3L9/+Fi1aWF555ZUi9X35e8hsNlsiIiIs//d//5e37+zZsxYvLy/LDz/8UOT+L3XgwAELYNm8ebPN/Rak/1zr16+3AJZDhw7Zve/k5GQLYFm6dKlNfV+v/yNHjlgqVqxo2b59u6Vq1aqW999/3+a+r9X/gw8+aOnXr1+h+rtR34MHD7bcf//9Re77Wv1frl+/fpbbbrvNbv03bNjQMnbs2Hz7Cvseu7z/uLg4C2DZvn173r6cnBxLaGio5YsvvrC5/8s/m86ePWvx8PCw/PTTT3nH7Ny50wJY1q5da3P/ttKZm1ImOTkZgJCQELv3nZOTw/Tp00lLS6Nt27Z263f48OH07t2brl272q3PXHv27CEqKooaNWowZMiQAp9uLoi5c+fSqlUr7r77bsLCwmjevDlffPGF3fq/VGZmJt999x0PP/yw3RZ0bdeuHcuWLWP37t0AbNmyhVWrVtGzZ88i952dnU1OTs4VfwH7+PjY9ewZwIEDB0hMTMz3309QUBA333wza9eutetrFZfk5GQMwyA4ONiu/WZmZvL5558TFBRE06ZN7dKn2WzmgQce4IUXXqBhw4Z26fNy0dHRhIWFUbduXf7xj39w6tSpIvdpNpuZP38+derUoXv37oSFhXHzzTfbddjApZKSkpg/fz6PPPKI3fps164dc+fO5ejRo1gsFpYvX87u3bvp1q1bkfvOyMgAyPceNplMeHl5Feo9fPln06ZNm8jKysr3vq1Xrx5VqlQplvetwk0pYjabGTlyJO3bt6dRo0Z263fbtm34+/vj5eXFk08+yezZs2nQoIFd+p4+fToxMTGMGzfOLv1d6uabb2bq1KksXLiQSZMmceDAAW655RZSU1Pt0v/+/fuZNGkStWvXZtGiRfzjH//g6aef5uuvv7ZL/5eaM2cOZ8+eZdiwYXbr8+WXX+aee+6hXr16eHh40Lx5c0aOHMmQIUOK3HdAQABt27blzTff5NixY+Tk5PDdd9+xdu1aEhIS7FD93xITEwEIDw/Ptz88PDzvudLkwoULvPTSS9x77712W1Rw3rx5+Pv74+3tzfvvv8+SJUuoUKGCXfoeP3487u7uPP3003bp73I9evTgm2++YdmyZYwfP54VK1bQs2dPcnJyitTv8ePHOXfuHG+//TY9evRg8eLFDBgwgIEDB7JixQo7Vf+3r7/+moCAALtedvn4449p0KABlSpVwtPTkx49ejBx4kQ6duxY5L5zg8bo0aM5c+YMmZmZjB8/niNHjtj8Hr7aZ1NiYiKenp5XBPjiet+WuVXBS7Phw4ezfft2u/9lXLduXWJjY0lOTubnn3/mwQcfZMWKFUUOOIcPH+aZZ55hyZIlBb7GbYtLz0A0adKEm2++mapVq/Ljjz/a5a8ns9lMq1ateOuttwBo3rw527dvZ/LkyTz44INF7v9SX375JT179rR5PMP1/Pjjj3z//fdMmzaNhg0bEhsby8iRI4mKirJL/d9++y0PP/wwFStWxM3NjRYtWnDvvfeyadMmO1TvmrKyshg0aBAWi4VJkybZrd/OnTsTGxvLyZMn+eKLLxg0aBDr1q0jLCysSP1u2rSJDz/8kJiYGLudUbzcPffck7fduHFjmjRpQs2aNYmOjqZLly6F7tdsNgPQr18/nn32WQCaNWvGmjVrmDx5Mp06dSpa4ZeZMmUKQ4YMsev/6z7++GP+/PNP5s6dS9WqVVm5ciXDhw8nKiqqyGfCPTw8mDVrFo888gghISG4ubnRtWtXevbsafNNB476bCoKnbkpJUaMGMG8efNYvnw5lSpVsmvfnp6e1KpVi5YtWzJu3DiaNm3Khx9+WOR+N23axPHjx2nRogXu7u64u7uzYsUKPvroI9zd3Yv8l9nlgoODqVOnDnv37rVLf5GRkVcEvPr169v10hfAoUOHWLp0KY8++qhd+33hhRfyzt40btyYBx54gGeffdZuZ9Fq1qzJihUrOHfuHIcPH2b9+vVkZWVRo0YNu/SfKyIiAuCKuyySkpLynisNcoPNoUOHWLJkid3O2gD4+flRq1Yt2rRpw5dffom7uztffvllkfv9448/OH78OFWqVMl7Dx86dIjnnnuOatWqFb3wq6hRowYVKlQo8vu4QoUKuLu7F8t7+I8//iAuLs6u7+Hz58/zr3/9iwkTJtCnTx+aNGnCiBEjGDx4MO+++65dXqNly5bExsZy9uxZEhISWLhwIadOnbLpPXytz6aIiAgyMzM5e/ZsvuOL632rcFPCWSwWRowYwezZs/n999+pXr26w1/TbDbnXY8tii5durBt2zZiY2PzHq1atWLIkCHExsbi5uZmh2r/du7cOfbt20dkZKRd+mvfvv0Vtzbu3r2bqlWr2qX/XF999RVhYWH07t3brv2mp6djMuV/i7u5ueX9RWsvfn5+REZGcubMGRYtWkS/fv3s2n/16tWJiIhg2bJleftSUlJYt26dXceGOVJusNmzZw9Lly6lfPnyDn09e72HH3jgAbZu3ZrvPRwVFcULL7zAokWL7FDplY4cOcKpU6eK/D729PSkdevWxfIe/vLLL2nZsqXdxjmB9b+ZrKysYnkPBwUFERoayp49e9i4cWOB3sM3+mxq2bIlHh4e+d63cXFxxMfHF8v7Vpel7ODcuXP5/so4cOAAsbGxhISEUKVKlSL1PXz4cKZNm8Yvv/xCQEBA3rXKoKAgfHx8itQ3wOjRo+nZsydVqlQhNTWVadOmER0dbZf/cQUEBFwxNsjPz4/y5cvbZczQ888/T58+fahatSrHjh3jtddew83NjXvvvbfIfYP1NsZ27drx1ltvMWjQINavX8/nn3/O559/bpf+wfoh9NVXX/Hggw/i7m7ft2OfPn3473//S5UqVWjYsCGbN29mwoQJPPzww3bpf9GiRVgsFurWrcvevXt54YUXqFevHg899JDNfd3oPTRy5Ej+85//ULt2bapXr86YMWOIioqif//+dun/9OnTxMfH5809k/uBGBERUaC/Mq/Xf2RkJHfddRcxMTHMmzePnJycvPdxSEgInp6ehe67fPny/Pe//6Vv375ERkZy8uRJJk6cyNGjRws8pcCNfjeXBzEPDw8iIiKoW7dukfsPCQnhjTfe4M477yQiIoJ9+/bx4osvUqtWLbp3717k2l944QUGDx5Mx44d6dy5MwsXLuTXX38lOjq6yLXn/r89JSWFn376iffee69AfdrSf6dOnXjhhRfw8fGhatWqrFixgm+++YYJEybYpf+ffvqJ0NBQqlSpwrZt23jmmWfo379/gQYs3+izKSgoiEceeYRRo0YREhJCYGAg//znP2nbti1t2rSx8TdVCA6/H6sMWL58uQW44vHggw8Wue+r9QtYvvrqqyL3bbFYLA8//LClatWqFk9PT0toaKilS5culsWLF9ul76ux563ggwcPtkRGRlo8PT0tFStWtAwePNiyd+9eu/Sd69dff7U0atTI4uXlZalXr57l888/t2v/ixYtsgCWuLg4u/ZrsVgsKSkplmeeecZSpUoVi7e3t6VGjRqWV155xZKRkWGX/mfMmGGpUaOGxdPT0xIREWEZPny45ezZs4Xq60bvIbPZbBkzZowlPDzc4uXlZenSpYtNv7Mb9f/VV19d9fnXXnutyP3n3l5+tcfy5cuL1Pf58+ctAwYMsERFRVk8PT0tkZGRlr59+1rWr19vt9/N5Wy9Ffx6/aenp1u6detmCQ0NtXh4eFiqVq1qeeyxxyyJiYl2q/3LL7+01KpVy+Lt7W1p2rSpZc6cOXapPddnn31m8fHxKdR/+zfqPyEhwTJs2DBLVFSUxdvb21K3bl3Le++9V+DpIm7U/4cffmipVKmSxcPDw1KlShXLv//97wL//6Egn03nz5+3PPXUU5Zy5cpZfH19LQMGDLAkJCTY8isqNONikSIiIiIuQWNuRERExKUo3IiIiIhLUbgRERERl6JwIyIiIi5F4UZERERcisKNiIiIuBSFGxEREXEpCjciUuZFR0djGMYV6+CISOmkcCMiIiIuReFGREREXIrCjYg4ndlsZty4cVSvXh0fHx+aNm3Kzz//DPx9yWj+/Pk0adIEb29v2rRpw/bt2/P1MXPmTBo2bIiXlxfVqlW7YiHDjIwMXnrpJSpXroyXlxe1atXiyy+/zHfMpk2baNWqFb6+vrRr1+6KFaVFpHRQuBERpxs3bhzffPMNkydPZseOHTz77LPcf//9rFixIu+YF154gffee48NGzYQGhpKnz59yMrKAqyhZNCgQdxzzz1s27aN119/nTFjxjB16tS89kOHDuWHH37go48+YufOnXz22Wf4+/vnq+OVV17hvffeY+PGjbi7u9ttBXURKV5aOFNEnCojI4OQkBCWLl1K27Zt8/Y/+uijpKen8/jjj9O5c2emT5/O4MGDATh9+jSVKlVi6tSpDBo0iCFDhnDixAkWL16c1/7FF19k/vz57Nixg927d1O3bl2WLFlC165dr6ghOjqazp07s3TpUrp06QLAb7/9Ru/evTl//jze3t4O/i2IiD3pzI2IONXevXtJT0/n9ttvx9/fP+/xzTffsG/fvrzjLg0+ISEh1K1bl507dwKwc+dO2rdvn6/f9u3bs2fPHnJycoiNjcXNzY1OnTpdt5YmTZrkbUdGRgJw/PjxIv+MIlK83J1dgIiUbefOnQNg/vz5VKxYMd9zXl5e+QJOYfn4+BToOA8Pj7xtwzAA63ggESlddOZGRJyqQYMGeHl5ER8fT61atfI9KleunHfcn3/+mbd95swZdu/eTf369QGoX78+q1evztfv6tWrqVOnDm5ubjRu3Biz2ZxvDI+IuC6duRERpwoICOD555/n2WefxWw206FDB5KTk1m9ejWBgYFUrVoVgLFjx1K+fHnCw8N55ZVXqFChAv379wfgueeeo3Xr1rz55psMHjyYtWvX8sknn/Dpp58CUK1aNR588EEefvhhPvroI5o2bcqhQ4c4fvw4gwYNctaPLiIOonAjIk735ptvEhoayrhx49i/fz/BwcG0aNGCf/3rX3mXhd5++22eeeYZ9uzZQ7Nmzfj111/x9PQEoEWLFvz444+8+uqrvPnmm0RGRjJ27FiGDRuW9xqTJk3iX//6F0899RSnTp2iSpUq/Otf/3LGjysiDqa7pUSkRMu9k+nMmTMEBwc7uxwRKQU05kZERERcisKNiIiIuBRdlhIRERGXojM3IiIi4lIUbkRERMSlKNyIiIiIS1G4EREREZeicCMiIiIuReFGREREXIrCjYiIiLgUhRsRERFxKQo3IiIi4lL+H4W6RQj9sX6pAAAAAElFTkSuQmCC\n"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["
"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAjsAAAHHCAYAAABZbpmkAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAWwJJREFUeJzt3Xd4VGX6xvHvpIdUQkIKkBB6R9qPqtSliEhbiqKCYAUUUFDRxYJKW0Fp1kXUXVABxVVQEJAiiFRRmnSkhp6EJKSQOb8/YmaJtGTmTGYS7s915XJyZuaZJ4nj3L7ve95jMQzDQERERKSY8nB1AyIiIiLOpLAjIiIixZrCjoiIiBRrCjsiIiJSrCnsiIiISLGmsCMiIiLFmsKOiIiIFGsKOyIiIlKsKeyIiIhIsaawIyIuUb58eQYMGODqNq7r8OHDWCwWPvroI1e3IiIOUtgRcbKPPvoIi8WCxWJh7dq1V91vGAblypXDYrFw11135bnPYrEwdOjQG9Zv1aqVrb7FYiEsLIxGjRrx4YcfYrVaC9TrqlWr8tS60Zc7+Oabb/Dw8CAhIcHVrYiIG/NydQMitwo/Pz/mzp1LixYt8hxfvXo1x44dw9fX1+7aZcuWZfz48QCcOXOGTz75hEGDBrF3714mTJiQ7zrVq1fn3//+d55jo0ePJjAwkBdeeMHu/q5lz549eHg49v9bixcvpkGDBkRFRZnUlYgURwo7IoXkzjvvZP78+UybNg0vr/+99ebOnUuDBg04e/as3bVDQkK47777bN8/+uijVK1alRkzZvDqq6/i7e2drzqRkZF56gBMmDCB8PDwq45fyWq1kpmZiZ+fX757diTc5fr2228ZOHCgw3VuJampqQQEBLi6DZFCpWkskUJyzz33cO7cOZYtW2Y7lpmZyYIFC7j33ntNfa0SJUrQpEkTUlNTOXPmDABpaWn8/vvvDoWqXLnTa3PmzKFmzZr4+vqyZMkSAN544w2aNWtGqVKl8Pf3p0GDBixYsOCqGn9ds5M73bdu3TqeeuopIiIiCAgIoHv37raf4Urbt2/n6NGjdO7cmVOnTuHl5cUrr7xy1eP27NmDxWJhxowZAJw/f56RI0dSu3ZtAgMDCQ4OplOnTvz66692/S4KUi89PZ2XX36ZKlWq4OfnR3R0ND169ODAgQO2x1itVqZOnUrt2rXx8/MjIiKCjh07snnzZuDGa4ksFgsvv/yy7fuXX34Zi8XCrl27uPfeeylZsqRtZPG3335jwIABVKhQAT8/P6Kiohg4cCDnzp27qu7x48cZNGgQMTEx+Pr6Eh8fz+OPP05mZiYHDx7EYrHw5ptvXvW8n376CYvFwqefflrQX6uIqRR2RApJ+fLladq0aZ7/8H/33XckJSXRt29f01/v4MGDeHp6EhoaCsDGjRupXr267UPfUT/88AMjRoygT58+TJ06lfLlywMwdepU6tWrx9ixYxk3bhxeXl706tWLxYsX56vuE088wa+//spLL73E448/zjfffHPNdUvffvstpUuXpmHDhkRGRtKyZUvmzZt31eM+//xzPD096dWrF5Dze/nqq6+46667mDJlCqNGjWL79u20bNmSEydOFPj3kN962dnZ3HXXXbzyyis0aNCAyZMnM2zYMJKSktixY4ftcYMGDWL48OGUK1eOiRMn8txzz+Hn58fPP/9c4N5y9erVi7S0NMaNG8fDDz8MwLJlyzh48CAPPvgg06dPp2/fvnz22WfceeedGIZhe+6JEyf4v//7Pz777DP69OnDtGnTuP/++1m9ejVpaWlUqFCB5s2bM2fOnKted86cOQQFBdG1a1e7excxhSEiTjV79mwDMDZt2mTMmDHDCAoKMtLS0gzDMIxevXoZrVu3NgzDMOLi4ozOnTvneS5gDBky5Ib1W7ZsaVSrVs04c+aMcebMGWP37t3Gk08+aQBGly5dbI9buXKlARgvvfRSgfqvWbOm0bJly6v68vDwMHbu3HnV43N/tlyZmZlGrVq1jDZt2uQ5HhcXZ/Tv39/2fe7vqV27dobVarUdHzFihOHp6WkkJibmef7tt9+e5/nvvfeeARjbt2/P87gaNWrkee309HQjOzs7z2MOHTpk+Pr6GmPHjs1zDDBmz5591c94pfzW+/DDDw3AmDJlylU1cn/eH374wQCMJ5988rqPuVFff/37vvTSSwZg3HPPPVc99q9/J8MwjE8//dQAjDVr1tiOPfDAA4aHh4exadOm6/aU+7vfvXu37b7MzEwjPDw8z99IxFU0siNSiHr37s2lS5dYtGgRFy9eZNGiRaZMYf3+++9EREQQERFB9erVmT59Op07d+bDDz+0PaZVq1YYhpFnmsMRLVu2pEaNGlcd9/f3t92+cOECSUlJ3H777WzdujVfdR955JE8Z3vdfvvtZGdn88cff9iOJSYmsn79ejp37mw71qNHD7y8vPj8889tx3bs2MGuXbvo06eP7Zivr69tYXR2djbnzp0jMDCQqlWr5rvHK+W33hdffEF4eDhPPPHEVTVyf94vvvgCi8XCSy+9dN3H2OOxxx676tiVf6f09HTOnj1LkyZNAGx9W61WvvrqK7p06ULDhg2v21Pv3r3x8/PLM7qzdOlSzp49e8O1XiKFRWFHpBBFRETQrl075s6dy5dffkl2djZ///vfHa5bvnx5li1bxvLly1m7di0JCQksWrSI8PBwE7q+tvj4+GseX7RoEU2aNMHPz4+wsDAiIiJ45513SEpKylfd2NjYPN+XLFkSyAlOuZYuXQpA+/btbcfCw8Np27Ztnqmszz//HC8vL3r06GE7ZrVaefPNN6lcuTK+vr6Eh4cTERHBb7/9lu8er5TfegcOHKBq1ap5Fqf/1YEDB4iJiSEsLKzAfdzItf5W58+fZ9iwYURGRuLv709ERITtcbl9nzlzhuTkZGrVqnXD+qGhoXTp0oW5c+fajs2ZM4cyZcrQpk0bE38SEfvobCyRQnbvvffy8MMPk5CQQKdOnWxrahwREBBAu3btHG+uAK4cGcj1448/cvfdd3PHHXfw9ttvEx0djbe3N7Nnz87zQXgjnp6e1zxuXLGO5Ntvv6V58+aEhITkeUzfvn158MEH2bZtG7fddhvz5s2jbdu2eULfuHHjGDNmDAMHDuTVV18lLCwMDw8Phg8fXuB9iZxR72auN8KTnZ193edc62/Vu3dvfvrpJ0aNGsVtt91GYGAgVquVjh072tX3Aw88wPz58/npp5+oXbs2X3/9NYMHD3Z4ewERMyjsiBSy7t278+ijj/Lzzz/nmXIpDr744gv8/PxYunRpnlPLZ8+ebdprGIbBkiVLGDly5FX3devWjUcffdT2e927dy+jR4/O85gFCxbQunVrZs2aled4YmKiXSNh+a1XsWJFNmzYQFZW1nW3AqhYsSJLly7l/Pnz1x3dyR3pSkxMzHP8ymm+m7lw4QIrVqzglVde4cUXX7Qd37dvX57HRUREEBwcnGcB9fV07NiRiIgI5syZQ+PGjUlLS+P+++/Pd08izqTILVLIAgMDeeedd3j55Zfp0qVLob2umaeeX4+npycWiyXPKMPhw4f56quvTHuNTZs2cfr06TzrdXKFhobSoUMH5s2bx2effYaPjw/dunW7qscrR4kA5s+fz/Hjx+3qJ7/1evbsydmzZ695Nlzu83v27IlhGNc8hT73McHBwYSHh7NmzZo897/99tsF6vnKmrneeuutPN97eHjQrVs3vvnmG9up79fqCcDLy4t77rmHefPm8dFHH1G7dm3q1KmT755EnEkjOyIu0L9//3w/dvPmzbz22mtXHW/VqtVVuzHfyMaNG2ndujUvvfSSaYuU/6pz585MmTKFjh07cu+993L69GlmzpxJpUqV+O2330x5jcWLF1O+fPlrLo4G6NOnD/fddx9vv/02HTp0uGqa8K677mLs2LE8+OCDNGvWjO3btzNnzhwqVKhgVz/5rffAAw/wySef8NRTT7Fx40Zuv/12UlNTWb58OYMHD6Zr1660bt2a+++/n2nTprFv3z7blNKPP/5I69atbafgP/TQQ0yYMIGHHnqIhg0bsmbNGvbu3ZvvnoODg7njjjuYNGkSWVlZlClThu+//55Dhw5d9dhx48bx/fff07JlSx555BGqV6/OyZMnmT9/PmvXrs3z+33ggQeYNm0aK1euZOLEiXb9PkWcQWFHxM1t2LCBDRs2XHX81VdfLVDYKQxt2rRh1qxZTJgwgeHDhxMfH8/EiRM5fPiwaWHn22+/5c4777zu/XfffTf+/v5cvHgxz1lYuZ5//nlSU1OZO3cun3/+OfXr12fx4sU899xzdvWT33qenp58++23vP7668ydO5cvvviCUqVK0aJFC2rXrm173OzZs6lTpw6zZs1i1KhRhISE0LBhQ5o1a2Z7zIsvvsiZM2dYsGAB8+bNo1OnTnz33XeULl06333PnTuXJ554gpkzZ2IYBu3bt+e7774jJiYmz+PKlCnDhg0bGDNmDHPmzCE5OZkyZcrQqVMnSpQokeexDRo0oGbNmuzevZt+/foV5Nco4lQW46/jmCIiburUqVNER0ezaNGiGwYecZ169eoRFhbGihUrXN2KiI3W7IhIkZGUlMSLL75I69atXd2KXMPmzZvZtm0bDzzwgKtbEclDIzsiIuKQHTt2sGXLFiZPnszZs2c5ePBggS4KK+JsGtkRERGHLFiwgAcffJCsrCw+/fRTBR1xOxrZERERkWJNIzsiIiJSrCnsiIiISLGmfXbIuZDfiRMnCAoKcujKwiIiIlJ4DMPg4sWLxMTE3PA6bAo7wIkTJyhXrpyr2xARERE7HD16lLJly173foUdICgoCMj5ZQUHB7u4GxEREcmP5ORkypUrZ/scvx6FHbBNXQUHByvsiIiIFDE3W4KiBcoiIiJSrCnsiIiISLGmsCMiIiLFmsKOiIiIFGsKOyIiIlKsKeyIiIhIsaawIyIiIsWawo6IiIgUawo7IiIiUqwp7IiIiEix5tKws2bNGrp06UJMTAwWi4Wvvvoqz/2GYfDiiy8SHR2Nv78/7dq1Y9++fXkec/78efr160dwcDChoaEMGjSIlJSUQvwpRERExJ25NOykpqZSt25dZs6cec37J02axLRp03j33XfZsGEDAQEBdOjQgfT0dNtj+vXrx86dO1m2bBmLFi1izZo1PPLII4X1I4iIiIibsxiGYbi6Cci5iNfChQvp1q0bkDOqExMTw9NPP83IkSMBSEpKIjIyko8++oi+ffuye/duatSowaZNm2jYsCEAS5Ys4c477+TYsWPExMTk67WTk5MJCQkhKSlJFwKVfEk8m8Cl1CRXtyEiUmSER5fH28fX1Jr5/fx226ueHzp0iISEBNq1a2c7FhISQuPGjVm/fj19+/Zl/fr1hIaG2oIOQLt27fDw8GDDhg107979mrUzMjLIyMiwfZ+cnOy8H0SKrOzLl0k4so+zh3/l0onf8Ti3j6CUQ0RlHaEkFwl1dYMiIkXI0X5rKFe5rkte223DTkJCAgCRkZF5jkdGRtruS0hIoHTp0nnu9/LyIiwszPaYaxk/fjyvvPKKyR1LUZWWmsyJ/dtJPLqTrFN78L2wj5JpfxCTfZwylizKXOd56YZ3ofYpIlKkWVy3csZtw44zjR49mqeeesr2fXJyMuXKlXNhR+JshtXKudPHOXVwOynHdmGc3UOJ5INEpP9BNGeodK0nWSDD8OaEZxnOlyhPZslKeEdWJbRcTWIq1qJEYEhh/xgiIkWWKz9l3TbsREVFAXDq1Cmio6Ntx0+dOsVtt91me8zp06fzPO/y5cucP3/e9vxr8fX1xdfX3HlDKaCMi7DnO7icfvPH2uFkUjp/HD2Kx/n9BKccIubyEcJJJfw6j79AEAnesVwMjMdaqjL+MdUJL1+HqNjKxHt5Ee+ULkVEpDC4bdiJj48nKiqKFStW2MJNcnIyGzZs4PHHHwegadOmJCYmsmXLFho0aADADz/8gNVqpXHjxq5qXW4mPQlmd4ZT2532EtF/fl3Jalg46VGas35xpAVXxLN0VYLK1iCqQh1KRkRT0mndiIiIK7k07KSkpLB//37b94cOHWLbtm2EhYURGxvL8OHDee2116hcuTLx8fGMGTOGmJgY2xlb1atXp2PHjjz88MO8++67ZGVlMXToUPr27ZvvM7GkkGWlw6f35gSdEuFQtpFppc+kZLAn4SKXsrIBCAwKgfAq+ERVpWRsLWIq1qKMf8B11+CIiEjx5NKws3nzZlq3bm37PncdTf/+/fnoo4945plnSE1N5ZFHHiExMZEWLVqwZMkS/Pz8bM+ZM2cOQ4cOpW3btnh4eNCzZ0+mTZtW6D+L5IM1G758CP5YC77B8MBXEFXb4bKnktN55ZudfHswZ1F62ZL+vNatFk2rlr7JM0VE5FbgNvvsuJL22SkEhgGLhsOWj8DTF+7/Esq3cKik1WowZ+MRJn33OxczLuPpYeGhFvEMa1eZEj5uO0MrIiImKfL77Egxs2p8TtCxeEDPfzkcdPYkXGT0l7+x9UgiAHXLhjCuR21qxugMKRERyUthR5xv4wewemLO7c6TocbddpdKz8pm2op9vL/mIJetBgE+nozqUJX7m5bH08NiUsMiIlKcKOyIc+1cCN+Oyrnd6nloONDuUmv3neWFr7bzx7k0ANrXiOSVrjWJDvE3o1MRESmmFHbEeQ6uhi8fAQxo9BC0fMauMudSMnht8W4W/nIcgKhgP16+uyYda11/LyUREZFcCjviHCe2wWf9IDsTanSFTpPAUrBpJsMwmL/lGOO+3U1iWhYWCzzQJI6RHaoS5KdLNYiISP4o7Ij5zh2AOX+HzIsQfwf0+AA8PAtU4uCZFJ5fuJ2fD54HoFpUEON71KZerLb+ExGRglHYEXNdPAX/6QGpZ3L20OkzB7zyf2mOzMtW3l19gBkr95N52YqftwfD21VhUIt4vD1ddxE5EREpuhR2xDzpSTCnJ1w4DCXLQ78vwC//+xZtOnye0V9uZ//pFADuqBLBa11rEVuqhHP6FRGRW4LCjpgjKz1njU7CdgiIgPsXQlBkvp6alJbFhCW7+XTjUQDCA30Yc1cN7q4bg6WA63xERET+SmFHHGfNhi8fhsM/gk8Q3PcFhFW46dMMw+Cb304y9ptdnE3JAKBvo3I816kaoSV8nN21iIjcIhR2xDGGAd+OhN1fg6cP9J0D0XVv+rTTyek888VvrNpzBoCKEQGM616bxhVKObtjERG5xSjsiGNWT4TNHwKWnLOuKrS86VMMw2D459v46cA5fDw9GNK6Eo+1qoCvV8HO2BIREckPhR2x36ZZOde8Auj8BtTslq+nrd1/1hZ0vn6iOdWidPFVERFxHp3LK/bZ9V9Y/HTO7ZbP5uyQnA9Wq8GkJXsAuK9JnIKOiIg4ncKOFNyhH+GLhwADGjwIrUbn+6nf7Uhg+/EkAnw8GdK6ovN6FBER+ZPCjhTMyd/gs3tzLgNRvUvOVczzeXr45Wwrk7/PGdV56PYKlArM/2aDIiIi9lLYkfw7fwj+0xMykiGuBfT4V4EuA7FgyzEOnk0lLMCHh26Pd2KjIiIi/6OwI/mTchr+3R1ST0NkbbhnLnj75fvp6VnZTF2xD4DBrSrqQp4iIlJoFHbk5tKTc0Z0LhyC0Di4bwH4hRSoxL/X/8HJpHRiQvy4r0mckxoVERG5msKO3NjlDPi8HyT8BiXC/7wMRFSBSiSnZzFz1X4Ahrergp+39tMREZHCo7Aj12fNhi8fgUNrwCcwZ0SnVMHPoPrXmoMkpmVRMSKAHvXLOKFRERGR61PYkWszDPjuWdj1FXh4Q5//QEy9Apc5czGDf609BMCoDlXx8tS/ciIiUrj0ySPXtnkWbPqAnMtAvAcVW9tVZubK/aRlZlO3bAgdahZs+ktERMQMCjtytTN7Yek/cm7/7RWo1dOuMkfPpzFnwx8APNOxGpZ87scjIiJiJoUdyetyJnz5EFy+BBXbQNMn7C715vK9ZGUbtKgUTvNK4SY2KSIikn8KO5LXytfh5K/gHwZd3wYP+/4V2ZNwkYW/HAdy1uqIiIi4isKO/M+hH2Hd1Jzbd0+D4Gi7S73x/R4MAzrViqJuuVBz+hMREbGDwo7kuHQBFj4GGFDv/pzrXtlp65ELLNt1Cg8LPN2+ink9ioiI2EFhR3IsHgnJxyCsAnScYHcZwzCY+N3vAPy9QVkqlQ4yq0MRERG7KOwI/DYPdiwAiyf0+AB8A+0utWbfWTYcOo+PlwfD2mlUR0REXE9h51Z34Q9Y/HTO7VbPQdmGdpeyWg3+uTRnVOf+JnGUCfU3o0MRERGHKOzcyqzZOet0MpKhXGNo8ZRD5b7dcZIdx5MJ9PVicKuCX1ZCRETEGRR2bmVr34QjP4FPEHR/Dzy97C6VlW1l8vd7AXjo9nhKBfqa1aWIiIhDFHZuVce3wqrxObfvnARh8Q6VW7DlGIfOphIW4MNDt1cwoUERERFzKOzcijJT4cuHwXoZanSDuvc4VC49K5u3lueM6gxtXYlAX/tHiERERMymsHMrWvoCnNsPQTFw15vg4DWrPll/mFPJGZQJ9adfk1iTmhQRETGHws6t5vdvYcvsnNvd34USYQ6VS07P4u1VBwAY3q4yvl6ejnYoIiJiKoWdW8nFU/D10JzbTYdChZYOl/xgzUES07KoVDqQHvXLOlxPRETEbAo7twrDgP8OhrRzEFkb2r7ocMkzFzP414+HABjZviqeHo5Nh4mIiDiDws6tYtO/YP9y8PKDnh+Al+Onhs/4YR+XsrKpWy6UDjUjTWhSRETEfAo7t4LTv8P3/8i5/bexULq6wyWPnk9j7sYjADzboSoWBxc5i4iIOIvCTnF3OQO+fAgup0OldvB/j5hS9s1le8nKNri9cjjNKoWbUlNERMQZFHaKux9eg4TtUKIUdH3b4dPMAfYkXGThtuMAjOpQ1eF6IiIizqSwU5wdWgM/Tc+5ffd0CDJnXc0/l+7BMODO2lHUKRtqSk0RERFnUdgpri5dyLnIJwbU7w/VOptSdssf51m++xSeHhaebq9RHRERcX8KO8WRYcCiEZB8HMIqQsfxJpU1mLhkDwB/r1+WihGBptQVERFxJoWd4ui3z2HnQvDwyjnN3CfAlLKr955h46Hz+Hh5MKxdZVNqioiIOJvCTnFz4TAsHplzu9VzUKaBKWWtVoN/Ls0Z1enfNI6YUH9T6oqIiDibwk5xkn0ZvnwUMi9CuSbQ4inTSi/efpKdJ5IJ9PXi8VaVTKsrIiLibAo7xcnaN+Hoz+ATBD3eAw9zLsqZlW1l8vc5ozqP3FGBsAAfU+qKiIgUBoWd4uLYFlj150Lkzm9AyfKmlZ6/+RiHz6VRKsCHgS3iTasrIiJSGBR2ioOMlJxdko1sqNkD6vQxrXR6VjZTV+wFYGibSgT6eplWW0REpDAo7BQHS5+H8wchuCzcNcWUXZJzffzTYU4lZ1Am1J97G8eaVldERKSwKOwUdbsXwdaPAQt0fwf8S5pWOulSFm+vOgDAiL9VwdfLnDVAIiIihUlhpyi7mABfP5Fzu9kTEH+HqeXfX3OApEtZVC4dSPd6ZUytLSIiUlgUdooqw4D/DoFL5yGqNrT5h6nlT19M58O1h4Gci316epg3NSYiIlKYFHaKqjN7YP9y8PSBHv8CL19Tyy/cepxLWdnULRfK32qYcwFRERERV1DYKaqObsj5Z7nGULqa6eVX7jkNQI96ZbCYuOBZRESksCnsFFVXhh2TJadnsfnwBQBaVy1ten0REZHCpLBTVDkx7Kzbd5bLVoMKEQHEliphen0REZHCpLBTFKWeg3P7c26XbWh6+dwpLI3qiIhIcaCwUxQd25jzz/CqUCLM1NKGYbByzxlAYUdERIoHhZ2iKHcKK9b8KaydJ5I5czGDEj6eNIo3b4NCERERV3HrsJOdnc2YMWOIj4/H39+fihUr8uqrr2IYhu0xhmHw4osvEh0djb+/P+3atWPfvn0u7LoQHP1zZMcJ63VW/TmF1axiuHZMFhGRYsGtw87EiRN55513mDFjBrt372bixIlMmjSJ6dOn2x4zadIkpk2bxrvvvsuGDRsICAigQ4cOpKenu7BzJ8rOguNbcm47IezYprCqRZheW0RExBXc+hLWP/30E127dqVz584AlC9fnk8//ZSNG3NGNgzD4K233uIf//gHXbt2BeCTTz4hMjKSr776ir59+7qsd6dJ+A0up+dcA6tUJVNLX0jN5JcjOaect9J6HRERKSbcemSnWbNmrFixgr179wLw66+/snbtWjp16gTAoUOHSEhIoF27drbnhISE0LhxY9avX++Snp3uyikskzf7W7PvDFYDqkYGUSbU39TaIiIiruLWIzvPPfccycnJVKtWDU9PT7Kzs3n99dfp168fAAkJCQBERua9nEFkZKTtvmvJyMggIyPD9n1ycrITuneSIz/n/LPc/5leetWfU1itNIUlIiLFiFuP7MybN485c+Ywd+5ctm7dyscff8wbb7zBxx9/7FDd8ePHExISYvsqV66cSR07mWE4bTPBbKvB6r065VxERIoftw47o0aN4rnnnqNv377Url2b+++/nxEjRjB+/HgAoqKiADh16lSe5506dcp237WMHj2apKQk29fRo0ed90OYKekYXDwJFk+IqW9q6d+OJXI+NZMgXy8axOmUcxERKT7cOuykpaXh4ZG3RU9PT6xWKwDx8fFERUWxYsUK2/3Jycls2LCBpk2bXreur68vwcHBeb6KhNxRneg64GPuZRxyz8K6vUo43p5u/a+FiIhIgbj1mp0uXbrw+uuvExsbS82aNfnll1+YMmUKAwcOBMBisTB8+HBee+01KleuTHx8PGPGjCEmJoZu3bq5tnlnsC1ObmJ66dV/7q+js7BERKS4ceuwM336dMaMGcPgwYM5ffo0MTExPProo7z44ou2xzzzzDOkpqbyyCOPkJiYSIsWLViyZAl+fn4u7NxJbOt1zF2cfOZiBr8eSwKgVRUtThYRkeLFYly5HfEtKjk5mZCQEJKSktx3SiszFcaXAyMbRuyCkDKmlf5iyzGenv8rtcoEs+iJ202rKyIi4kz5/fzW4oyi4vjWnKATXNbUoAO6yrmIiBRvCjtFxVHn7K9zOdvKmj9POdd6HRERKY4UdooKJ13885ejiSSnXya0hDe3lQs1tbaIiIg7UNgpCqzWK8KOuSM7K3/PmcJqWSUCTw9zLz8hIiLiDhR2ioJz+yA9Ebz8Iaq2qaVtVznXFJaIiBRTCjtFQe4p52UagKe3aWUTktLZfTIZiwXu0CnnIiJSTCnsFAW5YSfW3PU6q/48C+u2cqGEBfiYWltERMRdKOwUBU5anKxTzkVE5FagsOPu0s7D2b05t8s2Mq1s5mUra/edBRR2RESkeFPYcXfHNuX8M7wKlAgzrezmw+dJzcwmPNCXmjFuumu0iIiICRR23N0R52wmuNJ24c8IPHTKuYiIFGMKO+7Oaet1dMq5iIjcGhR23Fl2FhzfknPbxLBz9Hwa+0+n4OlhoUXlcNPqioiIuCOFHXeWsB0uXwK/UChV2bSyuaecN4grSYi/efv2iIiIuCOFHXd25SUiPMz7U2kKS0REbiUKO+4sdzNBE6ew0rOy+enAn6ecV9OuySIiUvwp7LgzJyxO/vngOdKzrESH+FE1Msi0uiIiIu5KYcddJR2D5GNg8YQy9U0ru+rPKaxWVUtjseiUcxERKf4UdtxV7qhOVG3wCTClpGEY/PB77iUiNIUlIiK3BoUdd+WE9TqHzqZy5Hwa3p4WmlfSKeciInJrUNhxV7awY97OyblnYTWOL0WAr5dpdUVERNyZwo47ykyFk7/l3DZxZGfVFZeIEBERuVUo7LijE7+AkQ1BMRBS1pSSqRmX2XDwPJCzOFlERORWobDjjq6cwjLpjKmfDpwjM9tKuTB/KkaYs+BZRESkKFDYcUe5Z2LFNjGtZO5VzlvrlHMREbnFKOy4G8MwfXGyYRis+v1/YUdERORWorDjbs7th0sXwMsfouqYUnLf6RROJKXj6+VBkwqlTKkpIiJSVCjsuJsjP+f8s0x98DTniuQr/xzVaVqxFP4+nqbUFBERKSoUdtyNU/bX0RSWiIjcuhR23I3JF/9MTs9i8+ELgMKOiIjcmhR23EnaeTi7J+d2WXNGdtbtO8tlq0GFiABiS5UwpaaIiEhRorDjTo5tzvlnqUoQYM5CYk1hiYjIrU5hx52YfPFPwzBs18NS2BERkVuVwo47MTns7DyRzJmLGZTw8aRRfElTaoqIiBQ1CjvuIvsyHN+Sc9uksJN74c/mlcLx9dIp5yIicmtS2HEXp3ZAVhr4hUB4FVNKagpLREREYcd95E5hlf0/8HD8z3IhNZNfjuScct6qaoTD9URERIoqhR13YfJ6nTX7zmA1oFpUEDGh/qbUFBERKYoUdtyFbTNBc/bXWfXnFFYrTWGJiMgtTmHHHSQdh6SjYPGAMg0cLpdtNVi9N3e9jqawRETk1qaw4w6O/TmqE1kLfAMdLvfbsUTOp2YS5OdF/Tidci4iIrc2hR13YPL1sHLPwrqjcgTenvoTi4jIrU2fhO4gd3FybBNTyuXur6OzsERERBR2XC/rEpz8Nee2CYuTz1zM4LdjSQC0VNgRERFR2HG541vBehmCoiGknMPlchcm1y4TQukgP4friYiIFHUKO65m21/n/8Bicbjc/65yrlEdERERUNhxPRMXJ1/OtrLmz5GdVtW0v46IiAgo7LiWYZi6c/LWI4lcTL9MyRLe1C0b6nA9ERGR4kBhx5XOHYBL58HTF6LqOFwudwqrZZUIPD0cnxITEREpDhR2XCl3VKdMffDycbjcyt//XK+jKSwREREbhR1XunJxsoNOJl3i94SLWCw5mwmKiIhIDoUdV7ItTnZ8M8HcC3/WKxdKyQDHR4lERESKC4UdV7mUCGd259w2YWTHNoWlq5yLiIjkobDjKsc25fwzrCIEhDtUKuNyNuv2nwWglcKOiIhIHgo7rmLiKeebD18gNTOb8EBfasYEO1xPRESkOFHYcRUTFydfeeFPD51yLiIikofCjitkX4ZjW3JumzCys/LPxclaryMiInI1hR1XOL0TslLBNxgiqjlU6uj5NPafTsHTw0KLyo6t/RERESmO7Ao7K1euNLuPW0vuKedlG4GHY3kzdwqrQVxJQvy9He1MRESk2LHrk7Zjx45UrFiR1157jaNHj5rdU/Fn4uJkTWGJiIjcmF1h5/jx4wwdOpQFCxZQoUIFOnTowLx588jMzDS7v+IpN+zEOhZ20rOy+elAzinnratp12QREZFrsSvshIeHM2LECLZt28aGDRuoUqUKgwcPJiYmhieffJJff/3V7D6Lj+STkHgELB5QpoFDpX4+eI70LCvRIX5UjQwyqUEREZHixeEFyvXr12f06NEMHTqUlJQUPvzwQxo0aMDtt9/Ozp07zeixeMkd1YmsCb6OBZTcS0S0qloai0WnnIuIiFyL3WEnKyuLBQsWcOeddxIXF8fSpUuZMWMGp06dYv/+/cTFxdGrVy8zey0ebNfDcmwKyzAMfrBdIkJTWCIiItdjV9h54okniI6O5tFHH6VKlSr88ssvrF+/noceeoiAgADKly/PG2+8we+//+5wg8ePH+e+++6jVKlS+Pv7U7t2bTZv3my73zAMXnzxRaKjo/H396ddu3bs27fP4dd1GpMWJyckp3PkfBpeHhaaV9Ip5yIiItfjZc+Tdu3axfTp0+nRowe+vr7XfEx4eLjDp6hfuHCB5s2b07p1a7777jsiIiLYt28fJUuWtD1m0qRJTJs2jY8//pj4+HjGjBlDhw4d2LVrF35+fg69vumyLsHJP9czObhz8pFzaQCULelPgK9df0YREZFbgl2fkitWrLh5YS8vWrZsaU95m4kTJ1KuXDlmz55tOxYfH2+7bRgGb731Fv/4xz/o2rUrAJ988gmRkZF89dVX9O3b16HXN92JbWDNgsBICI1zqNSxC5cAKFuyhAmNiYiIFF92TWONHz+eDz/88KrjH374IRMnTnS4qVxff/01DRs2pFevXpQuXZp69erxwQcf2O4/dOgQCQkJtGvXznYsJCSExo0bs379+uvWzcjIIDk5Oc9XobjyelgOLij+X9jxd7QrERGRYs2usPPee+9RrdrVlzmoWbMm7777rsNN5Tp48CDvvPMOlStXZunSpTz++OM8+eSTfPzxxwAkJCQAEBkZmed5kZGRtvuuZfz48YSEhNi+ypUrZ1rPN2TS4mSAYxf+N40lIiIi12dX2ElISCA6Ovqq4xEREZw8edLhpnJZrVbq16/PuHHjqFevHo888ggPP/yww4Fq9OjRJCUl2b4KZRdow7hiZKeJw+WOJ2oaS0REJD/sCjvlypVj3bp1Vx1ft24dMTExDjeVKzo6mho1auQ5Vr16dY4cOQJAVFQUAKdOncrzmFOnTtnuuxZfX1+Cg4PzfDnd+YOQdhY8fSG6jsPlcqexymhkR0RE5IbsWqD88MMPM3z4cLKysmjTpg2Qs2j5mWee4emnnzatuebNm7Nnz548x/bu3UtcXM7i3vj4eKKiolixYgW33XYbAMnJyWzYsIHHH3/ctD5MkTuqE1MPvK59Blt+ZVsNTiRqzY6IiEh+2BV2Ro0axblz5xg8eLDtelh+fn48++yzjB492rTmRowYQbNmzRg3bhy9e/dm48aNvP/++7z//vsAWCwWhg8fzmuvvUblypVtp57HxMTQrVs30/owxZWLkx10Kjmdy1YDb08LpYPc7PR6ERERN2NX2LFYLEycOJExY8awe/du/P39qVy58nX33LFXo0aNWLhwIaNHj2bs2LHEx8fz1ltv0a9fP9tjnnnmGVJTU3nkkUdITEykRYsWLFmyxP322DF1cXLOqE5MqD+eHrpMhIiIyI1YDMMwXN2EqyUnJxMSEkJSUpJz1u9cSoSJ5QEDRu6DwNIOlfty6zGemvcrzSuVYs5Dji92FhERKYry+/lt99a7mzdvZt68eRw5csQ2lZXryy+/tLds8XR8M2BAyXiHgw5cscdOqM7EEhERuRm7zsb67LPPaNasGbt372bhwoVkZWWxc+dOfvjhB0JCQszusegzcQoLtMeOiIhIQdgVdsaNG8ebb77JN998g4+PD1OnTuX333+nd+/exMbGmt1j0Ze7ODnWrLDz58hOmMKOiIjIzdgVdg4cOEDnzp0B8PHxITU1FYvFwogRI2xnSsmfrNlw7M+rtJs2sqMNBUVERPLLrrBTsmRJLl68CECZMmXYsWMHAImJiaSlpZnXXXFwaidkpoBvMERcfYmNgtIeOyIiIgVj1wLlO+64g2XLllG7dm169erFsGHD+OGHH1i2bBlt27Y1u8eiLXcKq2xD8PB0uNzpi9pjR0REpCDsCjszZswgPT0dgBdeeAFvb29++uknevbsyT/+8Q9TGyzyTF+cnDOqEx2iPXZERETyo8Bh5/LlyyxatIgOHToA4OHhwXPPPWd6Y8WGiTsng87EEhERKagCr9nx8vLiscces43syA1cTIDEPwALlGloSslj57VeR0REpCDsWqD8f//3f2zbts3kVoqh3CmsyJrgZ87OzDoTS0REpGDsWrMzePBgnnrqKY4ePUqDBg0ICAjIc3+dOnVMaa7IM3kKC+BYoqaxRERECsKusNO3b18AnnzySdsxi8WCYRhYLBays7PN6a6osy1ONu/6VRrZERERKRi7ws6hQ4fM7qP4sVohOyPntkkjO9pjR0REpODsCjtxcXFm91H8eHjAo2vg0gXwCzWl5OmL6WRlG3h5WIgM1h47IiIi+WFX2Pnkk09ueP8DDzxgVzPFkn9J00rlTmHFhGqPHRERkfyyK+wMGzYsz/dZWVmkpaXh4+NDiRIlFHacRHvsiIiIFJxdp55fuHAhz1dKSgp79uyhRYsWfPrpp2b3KH/SHjsiIiIFZ1fYuZbKlSszYcKEq0Z9xDzH/1ycXCZUZ2KJiIjkl2lhB3J2Vz5x4oSZJeUK/zvtXCM7IiIi+WXXmp2vv/46z/eGYXDy5ElmzJhB8+bNTWlMrqY1OyIiIgVnV9jp1q1bnu8tFgsRERG0adOGyZMnm9GX/IXVatimscqGaRpLREQkv+wKO1ar1ew+5CZOX8z43x47Qb6ubkdERKTIMHXNjjhP7hRWdKgfXp76s4mIiOSXXZ+aPXv2ZOLEiVcdnzRpEr169XK4KbmabXGyzsQSEREpELvCzpo1a7jzzjuvOt6pUyfWrFnjcFNyNS1OFhERsY9dYSclJQUfH5+rjnt7e5OcnOxwU3I1Xe1cRETEPnaFndq1a/P5559fdfyzzz6jRo0aDjclV9MeOyIiIvax62ysMWPG0KNHDw4cOECbNm0AWLFiBZ9++inz5883tUHJoWksERER+9gVdrp06cJXX33FuHHjWLBgAf7+/tSpU4fly5fTsmVLs3u85VmtBicS0wEoo7AjIiJSIHaFHYDOnTvTuXNnM3uR6ziTkkFmthVPDwtRwX6ubkdERKRIsWvNzqZNm9iwYcNVxzds2MDmzZsdbkrysu2xE6I9dkRERArKrk/OIUOGcPTo0auOHz9+nCFDhjjclOSlxckiIiL2syvs7Nq1i/r16191vF69euzatcvhpiQvnXYuIiJiP7vCjq+vL6dOnbrq+MmTJ/HysnsZkFyHzsQSERGxn11hp3379owePZqkpCTbscTERJ5//nn+9re/mdac5NDIjoiIiP3sGoZ54403uOOOO4iLi6NevXoAbNu2jcjISP7973+b2qBozY6IiIgj7Ao7ZcqU4bfffmPOnDn8+uuv+Pv78+CDD3LPPffg7e1tdo+3NKvV4LjCjoiIiN3sXmATEBBAixYtiI2NJTMzE4DvvvsOgLvvvtuc7kR77IiIiDjIrrBz8OBBunfvzvbt27FYLBiGgcVisd2fnZ1tWoO3Ou2xIyIi4hi7Pj2HDRtGfHw8p0+fpkSJEuzYsYPVq1fTsGFDVq1aZXKLt7bc9TplQjWFJSIiYg+7RnbWr1/PDz/8QHh4OB4eHnh6etKiRQvGjx/Pk08+yS+//GJ2n7csnYklIiLiGLtGdrKzswkKCgIgPDycEydOABAXF8eePXvM6050JpaIiIiD7BrZqVWrFr/++ivx8fE0btyYSZMm4ePjw/vvv0+FChXM7vGWpg0FRUREHGNX2PnHP/5BamoqAGPHjuWuu+7i9ttvp1SpUnz++eemNnirO65pLBEREYfYFXY6dOhgu12pUiV+//13zp8/T8mSJfOclSWOsVoNjiVqGktERMQRpl3IKiwszKxS8qezKRlkXs7ZYyc6RHvsiIiI2EMbt7ixo39OYUUFa48dERERe+kT1I1pcbKIiIjjFHbcmPbYERERcZzCjhvTHjsiIiKOU9hxY7nTWGUUdkREROymsOPGjuu0cxEREYcp7LgpwzBsGwqW05odERERuynsuKkzKRlkXLbiYYEo7bEjIiJiN4UdN5W7ODk6xB9v7bEjIiJiN32KuqncsKPFySIiIo5R2HFT2lBQRETEHAo7bkobCoqIiJhDYcdNaUNBERERcyjsuClNY4mIiJhDYccNaY8dERER8yjsuCHtsSMiImIehR03lDuqExXspz12REREHKRPUjekM7FERETMU6TCzoQJE7BYLAwfPtx2LD09nSFDhlCqVCkCAwPp2bMnp06dcl2TJtCZWCIiIuYpMmFn06ZNvPfee9SpUyfP8REjRvDNN98wf/58Vq9ezYkTJ+jRo4eLujSHzsQSERExT5EIOykpKfTr148PPviAkiVL2o4nJSUxa9YspkyZQps2bWjQoAGzZ8/mp59+4ueff3Zhx47RNJaIiIh5ikTYGTJkCJ07d6Zdu3Z5jm/ZsoWsrKw8x6tVq0ZsbCzr16+/br2MjAySk5PzfLkTjeyIiIiYx8vVDdzMZ599xtatW9m0adNV9yUkJODj40NoaGie45GRkSQkJFy35vjx43nllVfMbtUUhmFoZEdERMREbj2yc/ToUYYNG8acOXPw8zNvv5nRo0eTlJRk+zp69KhptR11NiVTe+yIiIiYyK3DzpYtWzh9+jT169fHy8sLLy8vVq9ezbRp0/Dy8iIyMpLMzEwSExPzPO/UqVNERUVdt66vry/BwcF5vtxF7hRWVLAfPl5u/ecREREpEtx6Gqtt27Zs3749z7EHH3yQatWq8eyzz1KuXDm8vb1ZsWIFPXv2BGDPnj0cOXKEpk2buqJlh2kKS0RExFxuHXaCgoKoVatWnmMBAQGUKlXKdnzQoEE89dRThIWFERwczBNPPEHTpk1p0qSJK1p2WG7YKaPFySIiIqZw67CTH2+++SYeHh707NmTjIwMOnTowNtvv+3qtuymM7FERETMVeTCzqpVq/J87+fnx8yZM5k5c6ZrGjLZ8UTtniwiImImrYB1M1qzIyIiYi6FHTeSs8eOprFERETMpLDjRs6lZpKeZcVigegQhR0REREzKOy4kdwpLO2xIyIiYh59oroRTWGJiIiYT2HHjWhxsoiIiPkUdtyIRnZERETMp7DjRv43sqOwIyIiYhaFHTeiaSwRERHzKey4iSv32CkTqpEdERERsyjsuInzV+6xE+rn6nZERESKDYUdN5E7hRUZ5Ievl6eLuxERESk+FHbchBYni4iIOIfCjpvQaeciIiLOobDjJnQmloiIiHMo7LgJjeyIiIg4h8KOm9DIjoiIiHMo7LiBnD12tEBZRETEGRR23MD51EwuZWVrjx0REREnUNhxA9pjR0RExHkUdtxAbtgpoyksERER0ynsuIHjiToTS0RExFkUdtyAFieLiIg4j8KOG9Bp5yIiIs6jsOMGtKGgiIiI8yjsuFjePXY0siMiImI2hR0Xu5CWRVpmNgAx2mNHRETEdAo7LpY7hRUZ7Ks9dkRERJxAYcfFNIUlIiLiXAo7LqbFySIiIs6lsONi2mNHRETEuRR2XMx2qYhQTWOJiIg4g8KOi2kaS0RExLkUdlzIMAyOaxpLRETEqRR2XCgxLYtU2x47CjsiIiLOoLDjQrnrdUoH+eLnrT12REREnEFhx4W0XkdERMT5FHZcSBsKioiIOJ/CjgtpZEdERMT5FHZcSCM7IiIizqew40LaPVlERMT5FHZcxDAMTWOJiIgUAoUdF9EeOyIiIoVDYcdFcqewIrTHjoiIiFMp7LjI8URNYYmIiBQGhR0X0ZlYIiIihUNhx0V0JpaIiEjhUNhxEZ2JJSIiUjgUdlxE01giIiKFQ2HHBXL22NE0loiISGFQ2HGBpEtZpGRcBqCM9tgRERFxKoUdF9AeOyIiIoVHYccFtDhZRESk8CjsuEDuyI6msERERJxPYccFdCaWiIhI4VHYcQGdiSUiIlJ4FHZcQGt2RERECo/CTiEzDIPjmsYSEREpNAo7hSz50mUu/rnHjkZ2REREnE9hp5Ad/XMKKzxQe+yIiIgUBoWdQqbFySIiIoVLYaeQaXGyiIhI4VLYKWTaY0dERKRwuXXYGT9+PI0aNSIoKIjSpUvTrVs39uzZk+cx6enpDBkyhFKlShEYGEjPnj05deqUizq+OU1jiYiIFC63DjurV69myJAh/PzzzyxbtoysrCzat29Pamqq7TEjRozgm2++Yf78+axevZoTJ07Qo0cPF3Z9Y7nTWGUUdkRERAqFl6sbuJElS5bk+f6jjz6idOnSbNmyhTvuuIOkpCRmzZrF3LlzadOmDQCzZ8+mevXq/PzzzzRp0sQVbV/XlXvslFPYERERKRRuPbLzV0lJSQCEhYUBsGXLFrKysmjXrp3tMdWqVSM2Npb169e7pMcbuXKPnTKhWrMjIiJSGNx6ZOdKVquV4cOH07x5c2rVqgVAQkICPj4+hIaG5nlsZGQkCQkJ162VkZFBRkaG7fvk5GSn9PxXxxJz99jxwd9He+yIiBSW7OxssrKyXN2GFJC3tzeeno5/XhaZsDNkyBB27NjB2rVrHa41fvx4XnnlFRO6KpjcxclldCaWiEihMAyDhIQEEhMTXd2K2Ck0NJSoqCgsFovdNYpE2Bk6dCiLFi1izZo1lC1b1nY8KiqKzMxMEhMT84zunDp1iqioqOvWGz16NE899ZTt++TkZMqVK+eU3q+kM7FERApXbtApXbo0JUqUcOgDUwqXYRikpaVx+vRpAKKjo+2u5dZhxzAMnnjiCRYuXMiqVauIj4/Pc3+DBg3w9vZmxYoV9OzZE4A9e/Zw5MgRmjZtet26vr6++Pr6OrX3a9GGgiIihSc7O9sWdEqVKuXqdsQO/v45n5enT5+mdOnSdk9puXXYGTJkCHPnzuW///0vQUFBtnU4ISEh+Pv7ExISwqBBg3jqqacICwsjODiYJ554gqZNm7rdmVigDQVFRApT7hqdEiX039yiLPfvl5WVVTzDzjvvvANAq1at8hyfPXs2AwYMAODNN9/Ew8ODnj17kpGRQYcOHXj77bcLudP80TSWiEjh09RV0WbG38+tw45hGDd9jJ+fHzNnzmTmzJmF0JFjcqextMeOiIhI4SlS++wUZUmXsriYrj12RESkcJUvX5633nrL1W24lFuP7BQnuaM6pQK0x46IiNxYq1atuO2220wJKZs2bSIgIMDxpoowhZ1CovU6IiJiFsMwyM7Oxsvr5h/jERERhdCRe9M0ViHRmVgiIpIfAwYMYPXq1UydOhWLxYLFYuGjjz7CYrHw3Xff0aBBA3x9fVm7di0HDhyga9euREZGEhgYSKNGjVi+fHmeen+dxrJYLPzrX/+ie/fulChRgsqVK/P111/nq7fs7GwGDRpEfHw8/v7+VK1alalTp171uA8//JCaNWvi6+tLdHQ0Q4cOtd2XmJjIo48+SmRkJH5+ftSqVYtFixbZ98vKJ43sFJLjGtkREXE5wzC4lJXtktf29/bM15lFU6dOZe/evdSqVYuxY8cCsHPnTgCee+453njjDSpUqEDJkiU5evQod955J6+//jq+vr588skndOnShT179hAbG3vd13jllVeYNGkS//znP5k+fTr9+vXjjz/+sF178nqsVitly5Zl/vz5lCpVip9++olHHnmE6OhoevfuDeScSf3UU08xYcIEOnXqRFJSEuvWrbM9v1OnTly8eJH//Oc/VKxYkV27dplySYgbUdgpJNpQUETE9S5lZVPjxaUuee1dYztQwufmH7shISH4+PhQokQJ29UAfv/9dwDGjh3L3/72N9tjw8LCqFu3ru37V199lYULF/L111/nGU35qwEDBnDPPfcAMG7cOKZNm8bGjRvp2LHjDXvz9vbOc7ml+Ph41q9fz7x582xh57XXXuPpp59m2LBhtsc1atQIgOXLl7Nx40Z2795NlSpVAKhQocJNfyeOUtgpJJrGEhERRzVs2DDP9ykpKbz88sssXryYkydPcvnyZS5dusSRI0duWKdOnTq22wEBAQQHB9suy3AzM2fO5MMPP+TIkSNcunSJzMxMbrvtNiBnp+MTJ07Qtm3baz5327ZtlC1b1hZ0CovCTiHRyI6IiOv5e3uya2wHl722o/56VtXIkSNZtmwZb7zxBpUqVcLf35+///3vZGZm3rCOt7d3nu8tFgtWq/Wmr//ZZ58xcuRIJk+eTNOmTQkKCuKf//wnGzZsAP53eYfrudn9zqKwUwiSLmWRnLvHjsKOiIjLWCyWfE0luZqPjw/Z2TdfW7Ru3ToGDBhA9+7dgZyRnsOHDzutr3Xr1tGsWTMGDx5sO3bgwAHb7aCgIMqXL8+KFSto3br1Vc+vU6cOx44dY+/evYU6uqOzsQpB7uLkUgE+ReJNJiIirlW+fHk2bNjA4cOHOXv27HVHXSpXrsyXX37Jtm3b+PXXX7n33nvzNUJjr8qVK7N582aWLl3K3r17GTNmDJs2bcrzmJdffpnJkyczbdo09u3bx9atW5k+fToALVu25I477qBnz54sW7aMQ4cO8d1337FkyRKn9QwKO4VCU1giIlIQI0eOxNPTkxo1ahAREXHdNThTpkyhZMmSNGvWjC5dutChQwfq16/vtL4effRRevToQZ8+fWjcuDHnzp3LM8oD0L9/f9566y3efvttatasyV133cW+ffts93/xxRc0atSIe+65hxo1avDMM8/kaxTLERYjPxegKuaSk5MJCQkhKSmJ4OBg0+t/uPYQYxftonPtaGb2c96/hCIi8j/p6ekcOnSI+Ph4/Pz8XN2O2OlGf8f8fn5rZKcQ5J6JpfU6IiIihU9hpxBoGktERIqCxx57jMDAwGt+PfbYY65uz25aLVsIdF0sEREpCsaOHcvIkSOveZ8zlnkUFoWdQnA8URsKioiI+ytdujSlS5d2dRum0zSWkyWnZ5F0KQuAMqEa2RERESlsCjtOlrvHTliADwG+GkgTEREpbAo7Tqb1OiIiIq6lsONkOhNLRETEtRR2nExXOxcREXEthR0n08iOiIgUtvLly/PWW2+5ug23obDjZFqzIyIi4loKO05mu1REqKaxREREXEFhx4ny7LGjkR0REcmH999/n5iYGKxWa57jXbt2ZeDAgRw4cICuXbsSGRlJYGAgjRo1Yvny5Xa/3pQpU6hduzYBAQGUK1eOwYMHk5KSkucx69ato1WrVpQoUYKSJUvSoUMHLly4AIDVamXSpElUqlQJX19fYmNjef311+3uxxkUdpwod4+dkiW8CdQeOyIirmcYkJnqmi/DyFeLvXr14ty5c6xcudJ27Pz58yxZsoR+/fqRkpLCnXfeyYoVK/jll1/o2LEjXbp04ciRI3b9Sjw8PJg2bRo7d+7k448/5ocffuCZZ56x3b9t2zbatm1LjRo1WL9+PWvXrqVLly5kZ2cDMHr0aCZMmMCYMWPYtWsXc+fOJTIy0q5enEWfwE6kM7FERNxMVhqMi3HNaz9/AnwCbvqwkiVL0qlTJ+bOnUvbtm0BWLBgAeHh4bRu3RoPDw/q1q1re/yrr77KwoUL+frrrxk6dGiB2xo+fLjtdvny5Xnttdd47LHHePvttwGYNGkSDRs2tH0PULNmTQAuXrzI1KlTmTFjBv379wegYsWKtGjRosB9OJNGdpzouM7EEhERO/Tr148vvviCjIwMAObMmUPfvn3x8PAgJSWFkSNHUr16dUJDQwkMDGT37t12j+wsX76ctm3bUqZMGYKCgrj//vs5d+4caWk5n2G5IzvXsnv3bjIyMq57v7vQyI4T6UwsERE3410iZ4TFVa+dT126dMEwDBYvXkyjRo348ccfefPNNwEYOXIky5Yt44033qBSpUr4+/vz97//nczMzAK3dPjwYe666y4ef/xxXn/9dcLCwli7di2DBg0iMzOTEiVK4O9//c+wG93nThR2nEjTWCIibsZiyddUkqv5+fnRo0cP5syZw/79+6latSr169cHchYLDxgwgO7duwOQkpLC4cOH7XqdLVu2YLVamTx5Mh4eOZM98+bNy/OYOnXqsGLFCl555ZWrnl+5cmX8/f1ZsWIFDz30kF09FAaFHSc6lqhpLBERsU+/fv2466672LlzJ/fdd5/teOXKlfnyyy/p0qULFouFMWPGXHXmVn5VqlSJrKwspk+fTpcuXVi3bh3vvvtunseMHj2a2rVrM3jwYB577DF8fHxYuXIlvXr1Ijw8nGeffZZnnnkGHx8fmjdvzpkzZ9i5cyeDBg1y6Oc3k9bsONHlbAOLRSM7IiJScG3atCEsLIw9e/Zw77332o5PmTKFkiVL0qxZM7p06UKHDh1soz4FVbduXaZMmcLEiROpVasWc+bMYfz48XkeU6VKFb7//nt+/fVX/u///o+mTZvy3//+Fy+vnPGSMWPG8PTTT/Piiy9SvXp1+vTpw+nTp+3/wZ3AYhj5PBeuGEtOTiYkJISkpCSCg4NNrZ152YqnhwVPD4updUVE5MbS09M5dOgQ8fHx+Pn5ubodsdON/o75/fzWNJaT+Xhp8ExERMSV9EksIiJSTM2ZM4fAwMBrfuXulXMr0MiOiIhIMXX33XfTuHHja97n7e1dyN24jsKOiIhIMRUUFERQUJCr23A5TWOJiIhIsaawIyIixZpOOi7azPj7KeyIiEixlLsmJfcaT1I05f79HFljpDU7IiJSLHl6ehIaGmrb4K5EiRJYLNrzrKgwDIO0tDROnz5NaGgonp6edtdS2BERkWIrKioKwO129JX8Cw0Ntf0d7aWwIyIixZbFYiE6OprSpUuTlZXl6nakgLy9vR0a0cmlsCMiIsWep6enKR+aUjRpgbKIiIgUawo7IiIiUqwp7IiIiEixpjU7/G/DouTkZBd3IiIiIvmV+7l9s40HFXaAixcvAlCuXDkXdyIiIiIFdfHiRUJCQq57v8XQPtpYrVZOnDhBUFCQqRtOJScnU65cOY4ePUpwcLBpdYtD/aLce1GvX5R7L+r1i3Lvzq5flHsv6vWLcu+GYXDx4kViYmLw8Lj+yhyN7AAeHh6ULVvWafWDg4Od8i9QcahflHsv6vWLcu9FvX5R7t3Z9Yty70W9flHt/UYjOrm0QFlERESKNYUdERERKdYUdpzI19eXl156CV9fX9UvxNqq77raqu+62kW9flHuvajXL8q955cWKIuIiEixppEdERERKdYUdkRERKRYU9gRERGRYk1hR0RERIo1hR0nWLNmDV26dCEmJgaLxcJXX31lav3x48fTqFEjgoKCKF26NN26dWPPnj2m1H7nnXeoU6eObfOnpk2b8t1335lS+1omTJiAxWJh+PDhptR7+eWXsVgseb6qVatmSm2A48ePc99991GqVCn8/f2pXbs2mzdvNqV2+fLlr+rdYrEwZMgQU+pnZ2czZswY4uPj8ff3p2LFirz66qs3vaZMfl28eJHhw4cTFxeHv78/zZo1Y9OmTXbXu9n7yDAMXnzxRaKjo/H396ddu3bs27fPlNpffvkl7du3p1SpUlgsFrZt22Za71lZWTz77LPUrl2bgIAAYmJieOCBBzhx4oQp9SHnfVCtWjUCAgIoWbIk7dq1Y8OGDabUvtJjjz2GxWLhrbfeMq33AQMGXPUe6Nixo2n1AXbv3s3dd99NSEgIAQEBNGrUiCNHjphS/1rvYYvFwj//+U+Ha6ekpDB06FDKli2Lv78/NWrU4N13381X3/mpf+rUKQYMGEBMTAwlSpSgY8eO+X5P5edzKT09nSFDhlCqVCkCAwPp2bMnp06dynf/jlDYcYLU1FTq1q3LzJkznVJ/9erVDBkyhJ9//plly5aRlZVF+/btSU1Ndbh22bJlmTBhAlu2bGHz5s20adOGrl27snPnThM6z2vTpk2899571KlTx9S6NWvW5OTJk7avtWvXmlL3woULNG/eHG9vb7777jt27drF5MmTKVmypCn1N23alKfvZcuWAdCrVy9T6k+cOJF33nmHGTNmsHv3biZOnMikSZOYPn26KfUfeughli1bxr///W+2b99O+/btadeuHcePH7er3s3eR5MmTWLatGm8++67bNiwgYCAADp06EB6errDtVNTU2nRogUTJ040vfe0tDS2bt3KmDFj2Lp1K19++SV79uzh7rvvNqU+QJUqVZgxYwbbt29n7dq1lC9fnvbt23PmzBmHa+dauHAhP//8MzExMfnuO7/1O3bsmOe98Omnn5pW/8CBA7Ro0YJq1aqxatUqfvvtN8aMGYOfn58p9a/s++TJk3z44YdYLBZ69uzpcO2nnnqKJUuW8J///Ifdu3czfPhwhg4dytdff+1w74Zh0K1bNw4ePMh///tffvnlF+Li4mjXrl2+Plvy87k0YsQIvvnmG+bPn8/q1as5ceIEPXr0yFfvDjPEqQBj4cKFTn2N06dPG4CxevVqp9QvWbKk8a9//cvUmhcvXjQqV65sLFu2zGjZsqUxbNgwU+q+9NJLRt26dU2p9VfPPvus0aJFC6fUvpZhw4YZFStWNKxWqyn1OnfubAwcODDPsR49ehj9+vVzuHZaWprh6elpLFq0KM/x+vXrGy+88ILD9f/6PrJarUZUVJTxz3/+03YsMTHR8PX1NT799FOHal/p0KFDBmD88ssvdnR98/q5Nm7caADGH3/84ZT6SUlJBmAsX77clNrHjh0zypQpY+zYscOIi4sz3nzzzQLVvVH9/v37G127drWrXn7q9+nTx7jvvvucVv+vunbtarRp08aU2jVr1jTGjh2b55i977G/1t+zZ48BGDt27LAdy87ONiIiIowPPvigwPX/+rmUmJhoeHt7G/Pnz7c9Zvfu3QZgrF+/vsD1C0ojO8VAUlISAGFhYabWzc7O5rPPPiM1NZWmTZuaWnvIkCF07tyZdu3amVoXYN++fcTExFChQgX69euX7+Hpm/n6669p2LAhvXr1onTp0tSrV48PPvjAlNp/lZmZyX/+8x8GDhxo2sVpmzVrxooVK9i7dy8Av/76K2vXrqVTp04O1758+TLZ2dlX/d+xv7+/aSNrVzp06BAJCQl5/v0JCQmhcePGrF+/3vTXc7akpCQsFguhoaGm187MzOT9998nJCSEunXrOlzParVy//33M2rUKGrWrGlCh1dbtWoVpUuXpmrVqjz++OOcO3fOlLpWq5XFixdTpUoVOnToQOnSpWncuLHpSw1ynTp1isWLFzNo0CBT6jVr1oyvv/6a48ePYxgGK1euZO/evbRv397h2hkZGQB53sMeHh74+vra9R7+6+fSli1byMrKyvOerVatGrGxsYXynlXYKeKsVivDhw+nefPm1KpVy5Sa27dvJzAwEF9fXx577DEWLlxIjRo1TKkN8Nlnn7F161bGjx9vWs1cjRs35qOPPmLJkiW88847HDp0iNtvv52LFy86XPvgwYO88847VK5cmaVLl/L444/z5JNP8vHHH5vQeV5fffUViYmJDBgwwLSazz33HH379qVatWp4e3tTr149hg8fTr9+/RyuHRQURNOmTXn11Vc5ceIE2dnZ/Oc//2H9+vWcPHnShO7zSkhIACAyMjLP8cjISNt9RUV6ejrPPvss99xzj6kXSVy0aBGBgYH4+fnx5ptvsmzZMsLDwx2uO3HiRLy8vHjyySdN6PJqHTt25JNPPmHFihVMnDiR1atX06lTJ7Kzsx2uffr0aVJSUpgwYQIdO3bk+++/p3v37vTo0YPVq1eb0H1eH3/8MUFBQaZN1UyfPp0aNWpQtmxZfHx86NixIzNnzuSOO+5wuHZu8Bg9ejQXLlwgMzOTiRMncuzYsQK/h6/1uZSQkICPj89Vgb6w3rO66nkRN2TIEHbs2GHq/z1XrVqVbdu2kZSUxIIFC+jfvz+rV682JfAcPXqUYcOGsWzZsnzPkRfElaMUderUoXHjxsTFxTFv3jyH/+/KarXSsGFDxo0bB0C9evXYsWMH7777Lv3793eo9l/NmjWLTp06FXg9xI3MmzePOXPmMHfuXGrWrMm2bdsYPnw4MTExpvT/73//m4EDB1KmTBk8PT2pX78+99xzD1u2bDGh++IpKyuL3r17YxgG77zzjqm1W7duzbZt2zh79iwffPABvXv3ZsOGDZQuXdrumlu2bGHq1Kls3brVtBHHv+rbt6/tdu3atalTpw4VK1Zk1apVtG3b1qHaVqsVgK5duzJixAgAbrvtNn766SfeffddWrZs6VD9v/rwww/p16+faf+tmz59Oj///DNff/01cXFxrFmzhiFDhhATE+PwKLm3tzdffvklgwYNIiwsDE9PT9q1a0enTp0KfBKDMz6XHKWRnSJs6NChLFq0iJUrV1K2bFnT6vr4+FCpUiUaNGjA+PHjqVu3LlOnTjWl9pYtWzh9+jT169fHy8sLLy8vVq9ezbRp0/Dy8jLl/96uFBoaSpUqVdi/f7/DtaKjo68KfNWrVzdtmizXH3/8wfLly3nooYdMrTtq1Cjb6E7t2rW5//77GTFihGkjbBUrVmT16tWkpKRw9OhRNm7cSFZWFhUqVDCl/pWioqIArjqT49SpU7b73F1u0Pnjjz9YtmyZqaM6AAEBAVSqVIkmTZowa9YsvLy8mDVrlkM1f/zxR06fPk1sbKzt/fvHH3/w9NNPU758eXMa/4sKFSoQHh5uyns4PDwcLy+vQnkf//jjj+zZs8e09/GlS5d4/vnnmTJlCl26dKFOnToMHTqUPn368MYbb5jyGg0aNGDbtm0kJiZy8uRJlixZwrlz5wr0Hr7e51JUVBSZmZkkJibmeXxhvWcVdoogwzAYOnQoCxcu5IcffiA+Pt6pr2e1Wm3zuY5q27Yt27dvZ9u2bbavhg0b0q9fP7Zt24anp6cpr5MrJSWFAwcOEB0d7XCt5s2bX3Uq5d69e4mLi3O49pVmz55N6dKl6dy5s6l109LS8PDI+5b39PS0/d+uWQICAoiOjubChQssXbqUrl27mlofID4+nqioKFasWGE7lpyczIYNG0xfX+YMuUFn3759LF++nFKlSjn9Nc14H99///389ttved6/MTExjBo1iqVLl5rUaV7Hjh3j3LlzpryHfXx8aNSoUaG8j2fNmkWDBg1MWScFOf/OZGVlFcp7OCQkhIiICPbt28fmzZvz9R6+2edSgwYN8Pb2zvOe3bNnD0eOHCmU96ymsZwgJSUlz/+FHDp0iG3bthEWFkZsbKzD9YcMGcLcuXP573//S1BQkG2+MyQkBH9/f4dqjx49mk6dOhEbG8vFixeZO3cuq1atMu0/ZEFBQVetLQoICKBUqVKmrDkaOXIkXbp0IS4ujhMnTvDSSy/h6enJPffc43DtESNG0KxZM8aNG0fv3r3ZuHEj77//Pu+//77DtXNZrVZmz55N//798fIy9+3ZpUsXXn/9dWJjY6lZsya//PILU6ZMYeDAgabUX7p0KYZhULVqVfbv38+oUaOoVq0aDz74oF31bvY+Gj58OK+99hqVK1cmPj6eMWPGEBMTQ7du3Ryuff78eY4cOWLb+yb3wzEqKipf/xd6o/rR0dH8/e9/Z+vWrSxatIjs7GzbezgsLAwfHx+H6pcqVYrXX3+du+++m+joaM6ePcvMmTM5fvx4vrYxuNnv5q/BzNvbm6ioKKpWrXrT2jerHxYWxiuvvELPnj2JioriwIEDPPPMM1SqVIkOHTo4XD82NpZRo0bRp08f7rjjDlq3bs2SJUv45ptvWLVqlSn1ISd4z58/n8mTJ+erZn5rt2zZklGjRuHv709cXByrV6/mk08+YcqUKabUnz9/PhEREcTGxrJ9+3aGDRtGt27d8rUA+mafSyEhIQwaNIinnnqKsLAwgoODeeKJJ2jatClNmjQp0O/JLk4/3+sWtHLlSgO46qt///6m1L9WbcCYPXu2w7UHDhxoxMXFGT4+PkZERITRtm1b4/vvv3e86Rsw89TzPn36GNHR0YaPj49RpkwZo0+fPsb+/ftNqW0YhvHNN98YtWrVMnx9fY1q1aoZ77//vmm1DcMwli5dagDGnj17TK1rGIaRnJxsDBs2zIiNjTX8/PyMChUqGC+88IKRkZFhSv3PP//cqFChguHj42NERUUZQ4YMMRITE+2ud7P3kdVqNcaMGWNERkYavr6+Rtu2bfP9e7tZ7dmzZ1/z/pdeesnh+rmns1/ra+XKlQ7Xv3TpktG9e3cjJibG8PHxMaKjo427777b2Lhxoym/m78q6KnnN6qflpZmtG/f3oiIiDC8vb2NuLg44+GHHzYSEhJMqZ9r1qxZRqVKlQw/Pz+jbt26xldffWVq/ffee8/w9/cv8L//N6t98uRJY8CAAUZMTIzh5+dnVK1a1Zg8eXK+t6e4Wf2pU6caZcuWNby9vY3Y2FjjH//4R77/+5Cfz6VLly4ZgwcPNkqWLGmUKFHC6N69u3Hy5MmC/IrsZvmzSREREZFiSWt2REREpFhT2BEREZFiTWFHREREijWFHRERESnWFHZERESkWFPYERERkWJNYUdERESKNYUdEZG/WLVqFRaL5arr+IhI0aSwIyIiIsWawo6IiIgUawo7IuJ2rFYr48ePJz4+Hn9/f+rWrcuCBQuA/00xLV68mDp16uDn50eTJk3YsWNHnhpffPEFNWvWxNfXl/Lly191UcaMjAyeffZZypUrh6+vL5UqVWLWrFl5HrNlyxYaNmxIiRIlaNas2VVXyxaRokFhR0Tczvjx4/nkk09499132blzJyNGjOC+++5j9erVtseMGjWKyZMns2nTJiIiIujSpQtZWVlATkjp3bs3ffv2Zfv27bz88suMGTOGjz76yPb8Bx54gE8//ZRp06axe/du3nvvPQIDA/P08cILLzB58mQ2b96Ml5eXaVeIF5HCpQuBiohbycjIICwsjOXLl9O0aVPb8Yceeoi0tDQeeeQRWrduzWeffUafPn0AOH/+PGXLluWjjz6id+/e9OvXjzNnzvD999/bnv/MM8+wePFidu7cyd69e6latSrLli2jXbt2V/WwatUqWrduzfLly2nbti0A3377LZ07d+bSpUv4+fk5+bcgImbSyI6IuJX9+/eTlpbG3/72NwIDA21fn3zyCQcOHLA97sogFBYWRtWqVdm9ezcAu3fvpnnz5nnqNm/enH379pGdnc22bdvw9PSkZcuWN+ylTp06ttvR0dEAnD592uGfUUQKl5erGxARuVJKSgoAixcvpkyZMnnu8/X1zRN47OXv75+vx3l7e9tuWywWIGc9kYgULRrZERG3UqNGDXx9fTly5AiVKlXK81WuXDnb437++Wfb7QsXLrB3716qV68OQPXq1Vm3bl2euuvWraNKlSp4enpSu3ZtrFZrnjVAIlJ8aWRHRNxKUFAQI0eOZMSIEVitVlq0aEFSUhLr1q0jODiYuLg4AMaOHUupUqWIjIzkhRdeIDw8nG7dugHw9NNP06hRI1599VX69OnD+vXrmTFjBm+//TYA5cuXp3///gwcOJBp06ZRt25d/vjjD06fPk3v3r1d9aOLiJMo7IiI23n11VeJiIhg/PjxHDx4kNDQUOrXr8/zzz9vm0aaMGECw4YNY9++fdx222188803+Pj4AFC/fn3mzZvHiy++yKuvvkp0dDRjx45lwIABttd45513eP755xk8eDDnzp0jNjaW559/3hU/rog4mc7GEpEiJfdMqQsXLhAaGurqdkSkCNCaHRERESnWFHZERESkWNM0loiIiBRrGtkRERGRYk1hR0RERIo1hR0REREp1hR2REREpFhT2BEREZFiTWFHREREijWFHRERESnWFHZERESkWFPYERERkWLt/wHSD7jskUcV5wAAAABJRU5ErkJggg==\n"},"metadata":{}}]},{"cell_type":"code","source":["# evaluate on last model\n","val_loss, val_accuracy = evaluate(model, val_loader, criterion)\n","print('MLP. Validation loss: {:.2f}, validation accuracy: {:.2f}'.format(val_loss, val_accuracy))\n"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"GREFIdABqxX0","executionInfo":{"status":"ok","timestamp":1774458152577,"user_tz":-60,"elapsed":36,"user":{"displayName":"Olivia Lecomte","userId":"07830699269148802057"}},"outputId":"c2df92bc-a82a-44aa-ac9f-72dc12613b46"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["MLP. Validation loss: 0.00, validation accuracy: 100.00\n"]}]},{"cell_type":"markdown","source":["## 3.3 **(6pts)** Interpret, Diagnose, and Fix the above results\n","Keep your answers as short as possible (one-word answers are acceptable; maximum: one sentence per question).\n","\n","If you were unable to complete the previous question, use the provided loss and accuracy curves below for your interpretation instead.\n","\n","![loss.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAbBdJREFUeJzt3Xd0VNXexvHvmfQeAqn03jsoTRBBqlQVVBSxe4WriJXrxYL3ivgqVgT1itgQVIoI0iUgRVoITQidUJLQk5BA2sz7x5BIqJlkJpNMns9as3Jy5uw9vwTHeXLOPnsbFovFgoiIiIiLMDm7ABERERF7UrgRERERl6JwIyIiIi5F4UZERERcisKNiIiIuBSFGxEREXEpCjciIiLiUhRuRERExKUo3IiIiIhLUbgRkVKrWrVqDBs2zNllXNPBgwcxDIOpU6de97jo6GgMwyA6OrpY6hJxdQo3IqXA1KlTMQwDwzBYtWrVFc9bLBYqV66MYRjccccd+Z4zDIMRI0Zct/9bb701r3/DMAgJCaF169ZMmTIFs9lsU625H9QFeZQEv/76KyaTicTERGeXIiJ24u7sAkSk4Ly9vZk2bRodOnTIt3/FihUcOXIELy+vQvddqVIlxo0bB8CJEyf45ptveOSRR9i9ezdvv/12gfupX78+3377bb59o0ePxt/fn1deeaXQ9V1NXFwcJlPR/kabP38+LVu2JCIiwk5ViYizKdyIlCK9evXip59+4qOPPsLd/e+377Rp02jZsiUnT54sdN9BQUHcf//9ed8/8cQT1K1bl08++YQ333wTDw+PAvUTHh6erx+At99+mwoVKlyx/1Jms5nMzEy8vb0LXHNRwlyu3377jYcffrjI/YhIyaHLUiKlyL333supU6dYsmRJ3r7MzEx+/vln7rvvPru+lq+vL23atCEtLY0TJ04AkJ6ezq5du4oUonLlXi77/vvvadiwIV5eXixcuBCAd999l3bt2lG+fHl8fHxo2bIlP//88xV9XD7mJvfy3erVqxk1ahShoaH4+fkxYMCAvJ/hUtu2bePw4cP07t2bpKQk3N3deeONN644Li4uDsMw+OSTTwA4ffo0zz//PI0bN8bf35/AwEB69uzJli1bivx7udRPP/1Ey5Yt8fHxyQuHR48ezXdMYmIiDz30EJUqVcLLy4vIyEj69evHwYMH847ZuHEj3bt3p0KFCvj4+FC9enUFOnFpCjcipUi1atVo27YtP/zwQ96+BQsWkJyczD333GP319u/fz9ubm4EBwcDsH79eurXr5/3IV9Uv//+O88++yyDBw/mww8/pFq1agB8+OGHNG/enLFjx/LWW2/h7u7O3Xffzfz58wvU7z//+U+2bNnCa6+9xj/+8Q9+/fXXq447+u233wgLC6NVq1aEh4fTqVMnfvzxxyuOmzFjBm5ubtx9992A9fcyZ84c7rjjDiZMmMALL7zAtm3b6NSpE8eOHSv8L+QSU6dOZdCgQbi5uTFu3Dgee+wxZs2aRYcOHTh79mzecXfeeSezZ8/moYce4tNPP+Xpp58mNTWV+Ph4AI4fP063bt04ePAgL7/8Mh9//DFDhgzhzz//tEudIiWRLkuJlDL33Xcfo0eP5vz58/j4+PD999/TqVMnoqKiitRvTk5O3hmZkydPMmnSJGJiYujTpw++vr72KP0KcXFxbNu2jQYNGuTbv3v3bnx8fPK+HzFiBC1atGDChAn07t37hv2WL1+exYsX5w1aNpvNfPTRRyQnJxMUFJR33Pz58+nZs2fecYMHD+aJJ55g+/btNGrUKO+4GTNm0KlTJ8LDwwFo3Lgxu3fvzjfe54EHHqBevXp8+eWXjBkzphC/jb9lZWXx0ksv0ahRI1auXJl3qa5Dhw7ccccdvP/++7zxxhucPXuWNWvW8H//9388//zzee1Hjx6dt71mzRrOnDnD4sWLadWqVd7+//znP0WqUaQk05kbkVJm0KBBnD9/nnnz5pGamsq8efPscklq165dhIaGEhoaSv369fn444/p3bs3U6ZMyTvm1ltvxWKx8Prrrxf59QA6dep0RbAB8gWbM2fOkJyczC233EJMTEyB+n388cfz3Y11yy23kJOTw6FDh/L2nT17lrVr1+YLSwMHDsTd3Z0ZM2bk7du+fTt//fUXgwcPztvn5eWVF2xycnI4deoU/v7+1K1bt8A1Xs/GjRs5fvw4Tz31VL4xSL1796ZevXp5Z7B8fHzw9PQkOjqaM2fOXLWv3LNu8+bNIysrq8i1iZQGCjcipUxoaChdu3Zl2rRpzJo1i5ycHO66664i91utWjWWLFnC0qVLWbVqFYmJicybN48KFSrYoeqrq169+lX3z5s3jzZt2uDt7U1ISAihoaFMmjSJ5OTkAvVbpUqVfN+XK1cOIF8AWLRoEQDdunXL21ehQgW6dOmS79LUjBkzcHd3Z+DAgXn7zGYz77//PrVr18bLy4sKFSoQGhrK1q1bC1zj9eSGsLp1617xXL169fKe9/LyYvz48SxYsIDw8HA6duzIO++8k++29k6dOnHnnXfyxhtvUKFCBfr168dXX31FRkZGkesUKakUbkRKofvuu48FCxYwefJkevbsmffXeVH4+fnRtWtXunTpQvv27QkLCyt6oTdw6RmaXH/88Qd9+/bF29ubTz/9lN9++40lS5Zw3333YbFYCtSvm5vbVfdf2v63336jffv2+S5TAdxzzz3s3r2b2NhYAH788Ue6dOmSL+S99dZbjBo1io4dO/Ldd9+xaNEilixZQsOGDW2eF6ioRo4cye7duxk3bhze3t6MGTOG+vXrs3nzZsA6cPvnn39m7dq1jBgxgqNHj/Lwww/TsmVLzp07V6y1ihQXhRuRUmjAgAGYTCb+/PNPu98l5WwzZ87E29ubRYsW8fDDD9OzZ0+6du1q19ewWCwsXLjwquN3+vfvj6enJzNmzCA2Npbdu3dfMVj7559/pnPnznz55Zfcc889dOvWja5du+Yb6FsUVatWBaxjki4XFxeX93yumjVr8txzz7F48WK2b99OZmYm7733Xr5j2rRpw3//+182btzI999/z44dO5g+fbpd6hUpaRRuREohf39/Jk2axOuvv06fPn2K7XXteSv4tbi5uWEYBjk5OXn7Dh48yJw5c+z2Ghs2bOD48eNXDTfBwcF0796dH3/8kenTp+Pp6Un//v2vqPHys0g//fTTFbdpF1arVq0ICwtj8uTJ+S4fLViwgJ07d+bVnZ6ezoULF/K1rVmzJgEBAXntzpw5c0WtzZo1A9ClKXFZultKpJR68MEHC3zsxo0br3p3zK233nrFbMfXs379ejp37sxrr71mt0HFl+vduzcTJkygR48e3HfffRw/fpyJEydSq1Yttm7dapfXmD9/PtWqVbvqYGaw3jV1//338+mnn9K9e/crLvvdcccdjB07loceeoh27dqxbds2vv/+e2rUqGGX+jw8PBg/fjwPPfQQnTp14t577yUpKSnvdvlnn30WsN5V1qVLFwYNGkSDBg1wd3dn9uzZJCUl5Z1t+vrrr/n0008ZMGAANWvWJDU1lS+++ILAwEB69epll3pFShqFG5EyYN26daxbt+6K/W+++aZN4aY43HbbbXz55Ze8/fbbjBw5kurVqzN+/HgOHjxot3Dz22+/XfeDvW/fvvj4+JCamprvLqlc//rXv0hLS2PatGnMmDGDFi1aMH/+fF5++WW71AcwbNgwfH19efvtt3nppZfyJiMcP358XtiqXLky9957L8uWLePbb7/F3d2devXq8eOPP3LnnXcC1gHF69evZ/r06SQlJREUFMRNN93E999/f80B3SKlnWEp6Ag9EREXkJSURGRkJPPmzdOZCxEXpTE3IlKmJCcn8+qrr9K5c2dnlyIiDqIzNyIiIuJSdOZGREREXIrCjYiIiLgUhRsRERFxKQo3IiIi4lLK3Dw3ZrOZY8eOERAQkG/VYBERESm5LBYLqampREVFYTJd/9xMmQs3x44do3Llys4uQ0RERArh8OHDVKpU6brHlLlwExAQAFh/OYGBgU6uRkRERAoiJSWFypUr532OX0+ZCze5l6ICAwMVbkREREqZggwp0YBiERERcSkKNyIiIuJSFG5ERETEpZS5MTciIuKacnJyyMrKcnYZUgSenp43vM27IBRuRESkVLNYLCQmJnL27FlnlyJFZDKZqF69Op6enkXqR+FGRERKtdxgExYWhq+vryZoLaVyJ9lNSEigSpUqRfp3VLgREZFSKycnJy/YlC9f3tnlSBGFhoZy7NgxsrOz8fDwKHQ/GlAsIiKlVu4YG19fXydXIvaQezkqJyenSP0o3IiISKmnS1GuwV7/jgo3IiIi4lIUbkREREq5atWq8cEHH9ilr+joaAzDKNV3n2lAsYiIiBPceuutNGvWzC6hZMOGDfj5+RW9KBehMzd2lJh8gV2JKc4uQ0REXIDFYiE7O7tAx4aGhmpQ9SUUbuxkwbYEbnnnd0bP2obFYnF2OSIiUoINGzaMFStW8OGHH2IYBoZhMHXqVAzDYMGCBbRs2RIvLy9WrVrFvn376NevH+Hh4fj7+9O6dWuWLl2ar7/LL0sZhsH//vc/BgwYgK+vL7Vr12bu3LmFrnfmzJk0bNgQLy8vqlWrxnvvvZfv+U8//ZTatWvj7e1NeHg4d911V95zP//8M40bN8bHx4fy5cvTtWtX0tLSCl1LQSjc2EnLauUwDIPN8Wf5c/9pZ5cjIlJmWSwW0jOznfIo6B+3H374IW3btuWxxx4jISGBhIQEKleuDMDLL7/M22+/zc6dO2nSpAnnzp2jV69eLFu2jM2bN9OjRw/69OlDfHz8dV/jjTfeYNCgQWzdupVevXoxZMgQTp+2/fNp06ZNDBo0iHvuuYdt27bx+uuvM2bMGKZOnQrAxo0befrppxk7dixxcXEsXLiQjh07ApCQkMC9997Lww8/zM6dO4mOjmbgwIEOPwmgMTd2EhbgzaBWlfjuz3g+jd5L25qaTEpExBnOZ+XQ4NVFTnntv8Z2x9fzxh+tQUFBeHp64uvrS0REBAC7du0CYOzYsdx+++15x4aEhNC0adO87998801mz57N3LlzGTFixDVfY9iwYdx7770AvPXWW3z00UesX7+eHj162PQzTZgwgS5dujBmzBgA6tSpw19//cX//d//MWzYMOLj4/Hz8+OOO+4gICCAqlWr0rx5c8AabrKzsxk4cCBVq1YFoHHjxja9fmHozI0dPdGxJm4mgz/2nGTbkWRnlyMiIqVQq1at8n1/7tw5nn/+eerXr09wcDD+/v7s3LnzhmdumjRpkrft5+dHYGAgx48ft7menTt30r59+3z72rdvz549e8jJyeH222+natWq1KhRgwceeIDvv/+e9PR0AJo2bUqXLl1o3Lgxd999N1988QVnzpyxuQZb6cyNHVUO8aVv0yhmbz7Kp9F7mXR/S2eXJCJS5vh4uPHX2O5Oe+2iuvyup+eff54lS5bw7rvvUqtWLXx8fLjrrrvIzMy8bj+XL19gGAZms7nI9V0uICCAmJgYoqOjWbx4Ma+++iqvv/46GzZsIDg4mCVLlrBmzRoWL17Mxx9/zCuvvMK6deuoXr263WvJpTM3dvaPW2sCsHBHInuPn3NyNSIiZY9hGPh6ujvlYcsMu56engVaZmD16tUMGzaMAQMG0LhxYyIiIjh48GARfkO2qV+/PqtXr76ipjp16uDmZg1z7u7udO3alXfeeYetW7dy8OBBfv/9d8D679G+fXveeOMNNm/ejKenJ7Nnz3ZozTpzY2d1wgO4vUE4S/5KYvKKfbx7d9MbNxIRkTKnWrVqrFu3joMHD+Lv73/Nsyq1a9dm1qxZ9OnTB8MwGDNmjEPOwFzLc889R+vWrXnzzTcZPHgwa9eu5ZNPPuHTTz8FYN68eezfv5+OHTtSrlw5fvvtN8xmM3Xr1mXdunUsW7aMbt26ERYWxrp16zhx4gT169d3aM06c+MAT108ezNn81GOnj3v5GpERKQkev7553Fzc6NBgwaEhoZecwzNhAkTKFeuHO3ataNPnz50796dFi1aFFudLVq04Mcff2T69Ok0atSIV199lbFjxzJs2DAAgoODmTVrFrfddhv169dn8uTJ/PDDDzRs2JDAwEBWrlxJr169qFOnDv/+979577336Nmzp0NrNixlbFKWlJQUgoKCSE5OJjAw0GGvc98Xf7Jm3ymGtavG630bOux1RETKsgsXLnDgwAGqV6+Ot7e3s8uRIrrev6ctn986c+MgwzvXAuCH9fGcPJfh5GpERETKDoUbB2lXszxNKwWRkW1myqoDzi5HREQEgCeffBJ/f/+rPp588klnl2cXGlDsIIZh8FTnWjzx7Sa+XXuIJ2+tSaC3x40bioiIONDYsWN5/vnnr/qcI4drFCeFGwe6vX44tcP82XP8HN+uPZR3qUpERMRZwsLCCAsLc3YZDqXLUg5kMhl5895MWXWA85k3ns9AREREikbhxsH6NI2iUjkfTqVl8uPGw84uR0RExOUp3DiYh5uJJzrWAODzlfvJyim+iZdERETKIoWbYnB3q8pU8Pfi6Nnz/BJ7zNnliIiIuDSFm2Lg7eHGIx2sC4RNit6L2Vym5k0UEREpVgo3xeT+NlUI8HZn34k0Fv+V6OxyRESklKtWrRoffPBBgY41DIM5c+Y4tJ6SROGmmAR4e/Bg22oATFy+jzK26oWIiEixUbgpRg+1r4a3h4ltR5NZtfeks8sRERFxSQo3xai8vxf3tK4CwMTle51cjYiIOMvnn39OVFQUZnP+O2j79evHww8/zL59++jXrx/h4eH4+/vTunVrli5darfX37ZtG7fddhs+Pj6UL1+exx9/nHPnzuU9Hx0dzU033YSfnx/BwcG0b9+eQ4cOAbBlyxY6d+5MQEAAgYGBtGzZko0bN9qtNntQuClmj3esgbvJ4M/9p9l06IyzyxERcT0WC2SmOedRwCEHd999N6dOnWL58uV5+06fPs3ChQsZMmQI586do1evXixbtozNmzfTo0cP+vTpQ3x8fJF/PWlpaXTv3p1y5cqxYcMGfvrpJ5YuXcqIESMAyM7Opn///nTq1ImtW7eydu1aHn/8cQzDAGDIkCFUqlSJDRs2sGnTJl5++WU8PErW8kJafqGYRQX7MKB5RX7adIRJ0Xv534OtnV2SiIhryUqHt6Kc89r/Ogaefjc8rFy5cvTs2ZNp06bRpUsXAH7++WcqVKhA586dMZlMNG3aNO/4N998k9mzZzN37ty8EFJY06ZN48KFC3zzzTf4+Vlr/eSTT+jTpw/jx4/Hw8OD5ORk7rjjDmrWtM6yX79+/bz28fHxvPDCC9SrVw+A2rVrF6keR9CZGyd48taaGAYs3XmcXYkpzi5HREScYMiQIcycOZOMjAwAvv/+e+655x5MJhPnzp3j+eefp379+gQHB+Pv78/OnTvtcuZm586dNG3aNC/YALRv3x6z2UxcXBwhISEMGzaM7t2706dPHz788EMSEhLyjh01ahSPPvooXbt25e2332bfvn1FrsnedObGCWqG+tOrUSTztyUwKXofH97T3NkliYi4Dg9f6xkUZ712AfXp0weLxcL8+fNp3bo1f/zxB++//z4Azz//PEuWLOHdd9+lVq1a+Pj4cNddd5GZmemoyvP56quvePrpp1m4cCEzZszg3//+N0uWLKFNmza8/vrr3HfffcyfP58FCxbw2muvMX36dAYMGFAstRWEztw4Se6Cmr9uOcahU2lOrkZExIUYhvXSkDMeF8elFIS3tzcDBw7k+++/54cffqBu3bq0aNECgNWrVzNs2DAGDBhA48aNiYiI4ODBg3b59dSvX58tW7aQlvb3Z8/q1asxmUzUrVs3b1/z5s0ZPXo0a9asoVGjRkybNi3vuTp16vDss8+yePFiBg4cyFdffWWX2uxF4cZJGlUMolOdUMwW+GzlfmeXIyIiTjBkyBDmz5/PlClTGDJkSN7+2rVrM2vWLGJjY9myZQv33XffFXdWFeU1vb29efDBB9m+fTvLly/nn//8Jw888ADh4eEcOHCA0aNHs3btWg4dOsTixYvZs2cP9evX5/z584wYMYLo6GgOHTrE6tWr2bBhQ74xOSWBwo0TPXXx7M3PG4+QlHLBydWIiEhxu+222wgJCSEuLo777rsvb/+ECRMoV64c7dq1o0+fPnTv3j3vrE5R+fr6smjRIk6fPk3r1q2566676NKlC5988kne87t27eLOO++kTp06PP744wwfPpwnnngCNzc3Tp06xdChQ6lTpw6DBg2iZ8+evPHGG3apzV4MSxmbKjclJYWgoCCSk5MJDAx0ai0Wi4W7J69l46EzPN6xBv/qVbKSr4hISXfhwgUOHDhA9erV8fb2dnY5UkTX+/e05fNbZ26cyDAMnupsPXvz3Z+HOJtePAPFREREXJnCjZN1rhtGvYgA0jNz+HrNIWeXIyIipcz333+Pv7//VR8NGzZ0dnlOoVvBncx69qYWT/+wma/WHODRW6rj56V/FhERKZi+ffty8803X/W5kjZzcHHRp2gJ0LtxJBMWx3HwVDo/rI/n0VtqOLskEREpJQICAggICHB2GSWKLkuVAG4mgyc7WcfefPHHfjKyc5xckYiISOmlcFNCDGhRkfBAL5JSMpgVc9TZ5YiIlCr2mgNGnMteN3DrslQJ4eXuxmO31OA/83cyecU+7m5ZCXc3ZU8Rkevx9PTEZDJx7NgxQkND8fT0zFu9WkoXi8XCiRMnMAyjyGOFFG5KkHtvqsIny/dy6FQ6v21PpG9TJ61qKyJSSphMJqpXr05CQgLHjjlpPSmxG8MwqFSpEm5ubkXqR+HGXtJOwYrxEBABt4wqVBd+Xu481K467y/dzafL99KnSaT+AhERuQFPT0+qVKlCdnY2OTkas1iaeXh4FDnYgMKN/RyIhvWfgYcfNBsCAeGF6ubBdlX5fOU+diWmsjzuOLfVK1w/IiJlSe6ljLJ667Pkp0Ed9tJwIFRsBVlpEP1WobsJ9vVkSJuqAExcvs9ug6tERETKCoUbezEM6PYf63bMN3B8Z6G7eqRDdTzdTGw6dIb1B07bqUAREZGywanhZty4cbRu3ZqAgADCwsLo378/cXFxN2z3008/Ua9ePby9vWncuDG//fZbMVRbAFXbQv0+YDHDklcL3U14oDd3taoEwKfR++xVnYiISJng1HCzYsUKhg8fzp9//smSJUvIysqiW7dupKWlXbPNmjVruPfee3nkkUfYvHkz/fv3p3///mzfvr0YK7+Orm+AyR32LIZ9ywvdzRMda2AyYMXuE2w/mmzHAkVERFybYSlBgzpOnDhBWFgYK1asoGPHjlc9ZvDgwaSlpTFv3ry8fW3atKFZs2ZMnjz5hq9hy5LphbbgJVg3GcIbwxMrwFS4kd/PTN/ML7HH6NU4gk+HtLRzkSIiIqWHLZ/fJWrMTXKy9QxFSEjINY9Zu3YtXbt2zbeve/furF279qrHZ2RkkJKSku/hcB1fBK8gSNoGW2cUupt/3GpdkmHB9kT2nThnr+pERERcWokJN2azmZEjR9K+fXsaNWp0zeMSExMJD89/e3R4eDiJiYlXPX7cuHEEBQXlPSpXrmzXuq/Krzx0fM66vexNyEwvVDf1IgLpWj8MiwUma+yNiIhIgZSYcDN8+HC2b9/O9OnT7drv6NGjSU5OznscPnzYrv1f001PQFAVSD0Gf04sdDdPda4FwOzNRzl69ry9qhMREXFZJSLcjBgxgnnz5rF8+XIqVap03WMjIiJISkrKty8pKYmIiIirHu/l5UVgYGC+R7Hw8Iaur1m3V30A544XqpsWVcrRpkYI2WYLX6zcb7/6REREXJRTw43FYmHEiBHMnj2b33//nerVq9+wTdu2bVm2bFm+fUuWLKFt27aOKrPwGg6EqBaQeQ6ixxW6m+EXz95M3xDPqXMZ9qpORETEJTk13AwfPpzvvvuOadOmERAQQGJiIomJiZw///fll6FDhzJ69Oi875955hkWLlzIe++9x65du3j99dfZuHEjI0aMcMaPcH0mE3T/r3V701Q4vqtQ3XSoVYHGFYO4kGVmyuoD9qtPRETEBTk13EyaNInk5GRuvfVWIiMj8x4zZvx9h1F8fDwJCQl537dr145p06bx+eef07RpU37++WfmzJlz3UHITlW1HdS7o0gT+xmGwfDO1junvll7iJQLWfasUERExKWUqHluikOxzHNzuZN74dObwZwNQ3+BGrfa3IXZbOH291ew70QaL/aoy1O31rJ/nSIiIiVUqZ3nxmVVqAWtHrFuL/43mM02d2EyGXmB5ss/DnA+M8eeFYqIiLgMhZvi0ukl8AqExMJP7Ne3WRQVg304lZbJjxuL6ZZ2ERGRUkbhprj4lYdbLk7s93vhJvbzcDPxZKcaAHy2Yh+Z2bafARIREXF1CjfF6eYnIagypByFPz8tVBd3t6pMBX8vjiVf4JfYo3YuUEREpPRTuClOHt7QJXdiv/cLNbGft4cbj95inQ9o0op95JjL1HhwERGRG1K4KW6N7oSo5hcn9nu7UF3c36Yqgd7u7D+RxqIdV19TS0REpKxSuCluJhN0+491e9NUOBFncxf+Xu4Ma289ezNx+V7K2N38IiIi16Vw4wzVOkDd3mDJgSWvFaqLh9pVw9fTjR3HUlix+4SdCxQRESm9FG6c5fY3wHCD3QvgwEqbm5fz8+S+m6oA8OnyffauTkREpNRSuHGWCrWh1cPW7UJO7PfoLTXwdDOx/uBp1h84becCRURESieFG2e69WXwDICELbDtJ5ubRwR5c2fLSgB8Gr3X3tWJiIiUSgo3zuRXAW4ZZd1eNhayzl//+Kt4slMNTAZEx51g+9FkOxcoIiJS+ijcOFubf0BgJUg5An9Osrl51fJ+9GkaBcCkaI29ERERUbhxNg8f6PKqdfuPCZB20uYu/nFrTQB+257AvhPn7FmdiIhIqaNwUxI0vhsim0JmaqEm9qsXEUjX+uFYLDp7IyIionBTElw6sd/GKXByj81dPNXZevZmzuajHDlj+6KcIiIirkLhpqSo3hHq9Cz0xH4tqpSjXc3yZJstfLFyvwMKFBERKR0UbkqS28daJ/aLmw8HV9ncfHjnWgBM33CYE6kZ9q5ORESkVFC4KUlC60Crh6zbhZjYr13N8jStHExGtpkpqw84oEAREZGST+GmpOl0cWK/Y5th+0ybmhqGwfCLd059u/YQyeezHFGhiIhIiaZwU9L4h8Itz1q3l70BWRdsat61fjh1wv05l5HNt2sP2r8+ERGREk7hpiRq8xQEVoTkw7Busk1NTSaDp261jr2Zsvog6ZnZjqhQRESkxFK4KYnyTez3HqSdsqn5HU0iqRLiy+m0TKavP+yAAkVEREouhZuSqvEgiGgCGSmwYrxNTd3dTDzZyTr25vOV+8nMtn3FcRERkdJK4aakyjex35dw0rZVv+9sWZGwAC8SUy4we/MRBxQoIiJSMinclGQ1OkGdHmDOhqW2Tezn5e7G4x1rANYlGXLMFkdUKCIiUuIo3JR0uRP77ZoHh9bY1PTem6oQ7OvBwVPp/LYtwUEFioiIlCwKNyVdaF1o+aB1e9ErNk3s5+flzkPtqgMwcfleLBadvREREdencFMa3DoaPP3hWAzsmGVT0wfbVcXP041diaksjzvuoAJFRERKDoWb0sA/DNo/Y93+81Obmgb7enJ/m6oAfPK7zt6IiIjrU7gpLVo+BCYPOLoJErfb1PSRDtXxdDcRE3+WdQdOO6hAERGRkkHhprTwD4V6vazbMV/b1DQs0JtBrSoB1rE3IiIirkzhpjRpcXFg8dYZkHXepqZPdKyJm8ngjz0n2XrkrP1rExERKSEUbkqTGp0huApcSIa/frGpaeUQX/o1jQLg0+X7HFGdiIhIiaBwU5qYTNB8qHV7k22XpgD+cat1SYaFOxLZk5Rqz8pERERKDIWb0qb5EDBMEL8GTuy2qWnt8AC6NwwHYNIKnb0RERHXpHBT2gRGQe3u1m0bBxYDPHVrLQB+iT3G4dPp9qxMRESkRFC4KY1yZyze8gNkZ9jUtGnlYG6pXYEcs4XPVursjYiIuB6Fm9Ko1u0QEAnpp2DXfJub5569+XHjEY6nXLB3dSIiIk6lcFMaublD8/ut24W4NNWmRggtqgSTmW3my1UH7FyciIiIcynclFbNHwAM2B8Np20LKIZhMOI269mb7/48xNn0TPvXJyIi4iQKN6VVuapQs7N1e/O3NjfvXDeMehEBpGXm8PWaQ3YuTkRExHkUbkqz3BmLN38POdk2NTUMg+GdrWdvvlpzgLQM29qLiIiUVAo3pVndXuBbAc4lwp5FNjfv1TiSauV9OZuexQ/r4x1QoIiISPFTuCnN3D2h2X3W7ULMWOxmMvJmLf585X4ysnPsWZ2IiIhTKNyUdrmXpvYugeSjNjcf0LwSkUHeHE/NYOYm29uLiIiUNAo3pV2FWlC1A1jMsPk7m5t7upt47JYaAExesY/sHLO9KxQRESlWCjeuIHfG4s3fgtn2S0v33FSZED9P4k+ns+SvJDsXJyIiUrwUblxB/b7gHQzJh2Hfcpub+3q6M6hVZQBmxujSlIiIlG4KN67Awxua3mPdjplaqC4GtqgIQHTccU6naVI/EREpvRRuXEXuwOK4BXDuuM3N64QH0DAqkGyzhflbj9m5OBERkeKjcOMqwhtApdZgzobY7wvVxYDm1rM3szbr0pSIiJReCjeuJPfsTcw3YLHY3LxvsyhMBmyOP8uBk2l2Lk5ERKR4KNy4kkYDwTMATu+Hg3/Y3DwswJsOtUMBmK2zNyIiUkop3LgSTz9ofJd1uxAzFgMMvHhpas7mo1gKcfZHRETE2RRuXE3unDc750L6aZubd2sYjq+nG/Gn04mJP2Pn4kRERBxP4cbVRDWHiCaQkwlbptvc3NfTnR6NIgCYpTlvRESkFFK4cUW5Z29ivi7UwOKBzSsBMG9rghbTFBGRUkfhxhU1vhs8fOHELji83ubmbWuWJzzQi+TzWUTHnXBAgSIiIo6jcOOKvIOg4QDrdoztA4vdTAb9mlkHFs/WpSkRESllFG5cVe6cN9tnwYVkm5vnTuj3+67jJKdn2bMyERERh1K4cVWVb4LQepB9Hrb9ZHPz+pGB1IsIIDPHzLxtWo5BRERKD6eGm5UrV9KnTx+ioqIwDIM5c+Zc9/jo6GgMw7jikZiYWDwFlyaG8ffZm8LOedNCl6ZERKT0cWq4SUtLo2nTpkycONGmdnFxcSQkJOQ9wsLCHFRhKdf0HnDzhMStcGyzzc37Nq2IYcDGQ2eIP5XugAJFRETsz92ZL96zZ0969uxpc7uwsDCCg4PtX5Cr8Q2B+n1h+8/WszdRzW1qHhHkTfuaFVi19yRzYo/ydJfaDipURETEfkrlmJtmzZoRGRnJ7bffzurVq697bEZGBikpKfkeZUrunDfbfoaMczY3zx1YPFvLMYiISClRqsJNZGQkkydPZubMmcycOZPKlStz6623EhMTc80248aNIygoKO9RuXLlYqy4BKh2C4TUgMxU2DHb5uY9GkXg4+HGgZNpxB4+a//6RERE7KxUhZu6devyxBNP0LJlS9q1a8eUKVNo164d77///jXbjB49muTk5LzH4cOHi7HiEsAwoPkD1u1CzHnj5+VO94bhgHUxTRERkZKuVIWbq7npppvYu3fvNZ/38vIiMDAw36PMaTYETO5wZAMk/WVz8/4XL039ujWBrByzvasTERGxq1IfbmJjY4mMjHR2GSVbQDjU6WHdLsTZmw61KlDB34vTaZms0HIMIiJSwjk13Jw7d47Y2FhiY2MBOHDgALGxscTHxwPWS0pDhw7NO/6DDz7gl19+Ye/evWzfvp2RI0fy+++/M3z4cGeUX7q0HGb9umU6ZF2wqam7m4l+zaIA68BiERGRksyp4Wbjxo00b96c5s2ttyiPGjWK5s2b8+qrrwKQkJCQF3QAMjMzee6552jcuDGdOnViy5YtLF26lC5dujil/lKl5m0QVBkunIWdc21unnvX1JKdSSSf13IMIiJSchmWMnZ/b0pKCkFBQSQnJ5e98TfRb0P0OKjaAR6ab1NTi8VC9w9WsjvpHOPvbMzg1lUcVKSIiMiVbPn8LvVjbsQGze8HwwSHVsHJaw/CvhrDMPIGFs/ScgwiIlKCKdyUJUGVoFZX63YhBhb3b2ZdjmHdgdMcOaPlGEREpGRSuClrchfTjJ0G2Zk2NY0K9qFN9fIA/BKrlcJFRKRkUrgpa+p0B/9wSD8Jcb/Z3HxAi9xLU0e0HIOIiJRICjdljZuHdVI/KNSlqZ6NIvByN7HvRBrbj5axdbpERKRUULgpi1pcXI5h33I4c8impgHeHtzewLocw6zNR+xdmYiISJEp3JRFITWgeifAApu/tbn5wIuXpn7dcoxsLccgIiIljMJNWdXy4sDizd9BTrZNTW+pHUp5P09Onsvkjz0nHVCciIhI4SnclFX17gCfEEhNgL1LbGrq4WaiT1MtxyAiIiWTwk1Z5e4Fze6zbm+yfWBx7nIMi/9K5FyGbWd+REREHKnQ4Wbv3r0sWrSI8+fPA+i24NKoxcVFSfcsghTb5q1pUimIGqF+XMgys2BbggOKExERKRybw82pU6fo2rUrderUoVevXiQkWD/YHnnkEZ577jm7FygOFFoXqrQFixk2f29TU8MwGHjx7I0uTYmISElic7h59tlncXd3Jz4+Hl9f37z9gwcPZuHChXYtTopB7ozFm78Bs213PvVrZg03a/efIiH5vL0rExERKRSbw83ixYsZP348lSpVyre/du3aHDpk25wpUgI06AdeQXA2HvYvt6lp5RBfbqoegsWi5RhERKTksDncpKWl5Ttjk+v06dN4eXnZpSgpRp6+0GSQdbsQMxbnDiyeHXNU465ERKREsDnc3HLLLXzzzTd53xuGgdls5p133qFz5852LU6KSe6cN7t+g3MnbGraq3Eknu4m4pJS+StByzGIiIjzudva4J133qFLly5s3LiRzMxMXnzxRXbs2MHp06dZvXq1I2oUR4toDFEt4FgMbJ8JbZ4scNMgHw+61g/jt22JzI45SsOoIAcWKiIicmM2n7lp1KgRu3fvpkOHDvTr14+0tDQGDhzI5s2bqVmzpiNqlOLQ+G7r1x2zbW46oLl1/NUvW46RY9alKRERcS6bz9wABAUF8corr9i7FnGmhv1h0Wg4/CckH4WgigVu2qlOKOV8PTiRmsHqvSfpWCfUcXWKiIjcgM3hZuXKldd9vmPHjoUuRpwoMMo65038WvhrDrQdXuCmnu4m7mgSxbd/HmL25qMKNyIi4lQ2h5tbb731in2GYeRt5+TkFKkgcaKGA63hZsdsm8INwIAWFfn2z0Ms3J7If/pn4+dVqJOCIiIiRWbzmJszZ87kexw/fpyFCxfSunVrFi9e7Igapbg06AsYcGSDdd4bGzSvHEz1Cn6cz8ph8V+JjqlPRESkAGwON0FBQfkeFSpU4Pbbb2f8+PG8+OKLjqhRiktABFTrYN22cWCxYRj0vzhj8awYLccgIiLOY7dVwcPDw4mLi7NXd+IsDftbvxbqrilruFm99yTHUy7YsSgREZGCs3lgxNatW/N9b7FYSEhI4O2336ZZs2b2qkucpX4/+O0FOLYZTu+HkBoFblqlvC8tq5Zj06Ez/BJ7jMc6FrytiIiIvdgcbpo1a4ZhGFdMtd+mTRumTJlit8LESfxDodotcGAF7JgDt4yyqfmA5hXZdOgMszYfVbgRERGnsDncHDhwIN/3JpOJ0NBQvL297VaUOFmjgRfDzSybw80dTSIZ++tf7ExIYVdiCvUiAh1UpIiIyNXZPOamatWq+R6VK1dWsHE19fqA4QaJ2+DkXpuaBvt60rmedZ6b2Zs1sFhERIpfgc7cfPTRRwXu8Omnny50MVJC+JWHGrfCvmXWgcWdXrCp+YDmlVi0I4lfNh/jxe71cDMZN24kIiJiJwUKN++//36BOjMMQ+HGVTQaWOhw07leKEE+HiSmXODP/adoX6uCg4oUERG5UoHCzeXjbKQMqNcbfh0Jx3fAiTgIrVvgpl7ubvRuEsm0dfHMijmqcCMiIsXKbvPciIvxKQc1b7NuF2LOm4EX57xZuD2B85lakkNERIpPoRYAOnLkCHPnziU+Pp7MzMx8z02YMMEuhUkJ0HAA7FkE22dBp5fAKPjYmZZVy1E5xIfDp8+z+K9E+jUr+CrjIiIiRWFzuFm2bBl9+/alRo0a7Nq1i0aNGnHw4EEsFgstWrRwRI3iLPV6gZsnnIyD4zshvEGBmxqGwYBmFfno973M3nxU4UZERIqNzZelRo8ezfPPP8+2bdvw9vZm5syZHD58mE6dOnH33Xc7okZxFu8gqNXVur1jls3NB7SoBMAfe05yIjXDnpWJiIhck83hZufOnQwdOhQAd3d3zp8/j7+/P2PHjmX8+PF2L1CcrOFA69cds+GyWalvpHoFP5pVDibHbOHXLcccUJyIiMiVbA43fn5+eeNsIiMj2bdvX95zJ0+etF9lUjLU7QHu3nBqr3VSPxsNbGG9HKUJ/UREpLjYHG7atGnDqlWrAOjVqxfPPfcc//3vf3n44Ydp06aN3QsUJ/MKgNq3W7cLcdfUHU2icDcZbDuazN7jqXYuTkRE5Eo2h5sJEyZw8803A/DGG2/QpUsXZsyYQbVq1fjyyy/tXqCUAA0HWL/umGXzpakQP09urWtdjmFWjM7eiIiI49l8t1SNGn+v9Ozn58fkyZPtWpCUQHV6gLsPnDkIxzZDRdvuihvQvBJLdx7nl9hjPN+tLiYtxyAiIg5k85mbRx99lOjoaAeUIiWWpx/U6W7dLsSlqS71wwjwcufo2fNsij9j5+JERETyszncnDhxgh49elC5cmVeeOEFtmzZ4oi6pKRplHvX1BybL015e7jRvVEEAHNjddeUiIg4ls3h5pdffiEhIYExY8awYcMGWrRoQcOGDXnrrbc4ePCgA0qUEqHW7eDhB8nxcHSTzc37NI0C4LdtCWTnmO1dnYiISJ5CrS1Vrlw5Hn/8caKjozl06BDDhg3j22+/pVatWvauT0oKT1+o29O6vd32Cf3a1yxPeT9PTqVlsnrfKTsXJyIi8rciLZyZlZXFxo0bWbduHQcPHiQ8PNxedUlJlHvX1F9zwGzb2Rd3NxO9GkcCaEI/ERFxqEKFm+XLl/PYY48RHh7OsGHDCAwMZN68eRw5csTe9UlJUqsreAVCylE4st7m5n2bWS9NLdqeyIUsrRQuIiKOYXO4qVixIr169eLkyZN8/vnnJCUlMWXKFLp06YJhw6rRUgp5eEPdXtbtQtw11bJKOSKDvEnNyCY67oSdixMREbGyOdy8/vrrJCQkMHv2bO666y68vLwcUZeUVHkT+s0Bs21nX0wmI29g8a9bdWlKREQcw+Zw89hjjxEcHOyAUqRUqHkbeAXBuUSI/9Pm5n0vhptlO5NIy8i2d3UiIiJFG1AsZZC7J9S/w7q9w/a7phpGBVK9gh8Xssws+SvJzsWJiIgo3EhhNLw4od9fv9h8acowLrk0pbumRETEARRuxHY1OoFPOUg7AQdX2dy8b1PrLeEr95zgbHqmvasTEZEyTuFGbOfmAfX7WLcLcddUrbAA6kcGkpVjYcH2RDsXJyIiZZ3N4ebrr79m/vz5ed+/+OKLBAcH065dOw4dOmTX4qQEy71raudcyLF9YHBfXZoSEREHsTncvPXWW/j4+ACwdu1aJk6cyDvvvEOFChV49tln7V6glFDVOoJveUg/BQdX2tz8jibWS1Nr95/ieMoFe1cnIiJlmM3h5vDhw3lrSM2ZM4c777yTxx9/nHHjxvHHH3/YvUApodzcoX5f63Yh1pqqHOJLiyrBWCwwb2uCnYsTEZGyzOZw4+/vz6lT1oUPFy9ezO233w6At7c358+ft291UrI1unjX1M5fIdv2gcF9NaGfiIg4gM3h5vbbb+fRRx/l0UcfZffu3fTqZZ2Of8eOHVSrVs3e9UlJVrU9+IXBhbNwYIXNzXs1icRkwOb4sxw+nW7/+kREpEyyOdxMnDiRtm3bcuLECWbOnEn58uUB2LRpE/fee6/dC5QSzOQGDfpZtwtxaSoswJu2Na3//czVwGIREbETw2KxWJxdRHFKSUkhKCiI5ORkAgMDnV1O6XdwNUztZV2S4YU94G7bWmMzNsTz0sxt1IsIYOHIjg4qUkRESjtbPr9tPnOzcOFCVq36e+K2iRMn0qxZM+677z7OnDlje7VSulVpC/4RkJEM+363uXmPhpF4uBnsSkxld1KqAwoUEZGyxuZw88ILL5CSkgLAtm3beO655+jVqxcHDhxg1KhRdi9QSjiTCRr2t24XYkK/IF8POtUJBTTnjYiI2IfN4ebAgQM0aNAAgJkzZ3LHHXfw1ltvMXHiRBYsWGBTXytXrqRPnz5ERUVhGAZz5sy5YZvo6GhatGiBl5cXtWrVYurUqbb+CGJvuWtN7foNsmyfsyZ3ram5W45Rxq6SioiIA9gcbjw9PUlPt97ZsnTpUrp16wZASEhI3hmdgkpLS6Np06ZMnDixQMcfOHCA3r1707lzZ2JjYxk5ciSPPvooixYtsu2HEPuq1BoCK0FmKuxdanPz2xuE4+PhxqFT6Ww7muyAAkVEpCxxt7VBhw4dGDVqFO3bt2f9+vXMmDEDgN27d1OpUiWb+urZsyc9e/Ys8PGTJ0+mevXqvPfeewDUr1+fVatW8f7779O9e3ebXlvsKPfS1NpPYMcsqH+HTc19Pd3pUj+MeVsTmBt7jCaVgh1SpoiIlA02n7n55JNPcHd35+eff2bSpElUrFgRgAULFtCjRw+7F3iptWvX0rVr13z7unfvztq1a6/ZJiMjg5SUlHwPcYDcS1NxCyHT9jlrcif0m7c1AbNZl6ZERKTwbD5zU6VKFebNm3fF/vfff98uBV1PYmIi4eHh+faFh4eTkpLC+fPn89a8utS4ceN44403HF5bmVexBQRXgbPxsGfx34OMC6hT3VACvN1JTLnAhoOnublGecfUKSIiLs/mMzcAOTk5zJw5k//85z/85z//Yfbs2eTk5Ni7NrsYPXo0ycnJeY/Dhw87uyTXZBh/rxReiLumvNzd6NEwAtCEfiIiUjQ2h5u9e/dSv359hg4dyqxZs5g1axb3338/DRs2ZN++fY6oMU9ERARJSUn59iUlJREYGHjVszYAXl5eBAYG5nuIg+SGm92LIOOczc37NrNemvptWwJZOWZ7ViYiImWIzeHm6aefpmbNmhw+fJiYmBhiYmKIj4+nevXqPP30046oMU/btm1ZtmxZvn1Lliyhbdu2Dn1dKaDIZlCuOmSfhz2238HWtkZ5Kvh7ciY9i9V7T9q/PhERKRNsDjcrVqzgnXfeISQkJG9f+fLlefvtt1mxwrbFE8+dO0dsbCyxsbGA9Vbv2NhY4uPjAeslpaFDh+Yd/+STT7J//35efPFFdu3axaeffsqPP/7Is88+a+uPIY5w6aWpQqw15e5molfjSECXpkREpPBsDjdeXl6kpl45Tf65c+fw9PS0qa+NGzfSvHlzmjdvDsCoUaNo3rw5r776KgAJCQl5QQegevXqzJ8/nyVLltC0aVPee+89/ve//+k28JKk0cW7pvYsgQzbl1PIvWtq8Y4kLmSVzHFcIiJSstm8cObQoUOJiYnhyy+/5KabbgJg3bp1PPbYY7Rs2bLEzxishTMdzGKBT1rBqb0w8AtoMsim5mazhVveWc7Rs+eZfH8LejSKdFChIiJSmjh04cyPPvqImjVr0rZtW7y9vfH29qZ9+/bUqlWLDz/8sNBFi4swjL/nvCnEpSmTyeCOJro0JSIihWfzPDfBwcH88ssv7Nmzh127dgHWmYJr1apl9+KklGo4AFa+A/uWwfmz4BNsU/M+TaP4bOV+lu08TuqFLAK8PRxSpoiIuCabw02u2rVrU7t2bXvWIq4ivAGE1oMTuyDuN2h2n03NG0YFUiPUj/0n0ljyVxIDW9i2rIeIiJRtBQo3o0aNKnCHEyZMKHQx4kIaDoDocdYJ/WwMN4Zh0KdJFB8u28OvW44p3IiIiE0KFG42b95coM4MwyhSMeJCcsPNvt8h/TT4hty4zSX6NrOGmz/2nORMWibl/Gy7E09ERMquAoWb5cuXO7oOcTWhdSGsIRzfAbvmQ4sHbGpeM9SfhlGB7DiWwm/bExhyc1UHFSoiIq6mUGtLiRRIo9y1pmy/awr+nvPmV901JSIiNlC4EcfJvSV8/wpIO2Vz8zsuhpt1B06TmHzBnpWJiIgLU7gRxylfEyKagCUHds61uXnFYB9aVS2HxQLztursjYiIFIzCjThW7nIMO2YXqnnuSuG/bk2wV0UiIuLiFG7EsRr0t349+AecO25z856NIjEZsOXwWQ6dSrNvbSIi4pIUbsSxQqpDVAuwmAt1aSo0wIv2tSoAGlgsIiIFo3Ajjtfw4l1T2wt3aapP3l1TujQlIiI3pnAjjpcbbg6thrPxNjfv3jACTzcTcUmpxCWm2rk4ERFxNQo34njBlaF6R8ACm762uXmQjwed6oYCMHfLUTsXJyIirkbhRopHq0esX2O+gexMm5v3veTSlMVisWdlIiLiYhRupHjU6w3+EZB2HHbNs7l5l/ph+Hi4EX86nS1Hkh1QoIiIuAqFGykebh7QYqh1e+MUm5v7erpze4NwAObG6q4pERG5NoUbKT4tHwTDZJ3z5kSczc1zL03N23qMHLMuTYmIyNUp3EjxCaoEdXpYtwtx9uaWOhUI9HbneGoG6w+ctnNxIiLiKhRupHjlDiyO/QEy021q6uXuRs9GkQDM1YR+IiJyDQo3Urxq3gblqkFGMmyfaXPz3LWmFmxPIDPbbOfiRETEFSjcSPEymaDlQ9btjV/a3LxNjfJU8PfibHoWq/eetHNxIiLiChRupPg1vx/cPOHYZjgaY1NTN5PBHU10aUpERK5N4UaKn1+Fv1cLL8TZm9y1phbvSOR8Zo4dCxMREVegcCPO0eph69dtM+H8GZuatqgSTMVgH9Iyc1ged9wBxYmISGmmcCPOUaUNhDWA7POwZYZNTQ3DyDt7own9RETkcgo34hyG8ffZm41TwMb1onIn9Ps97jgpF7LsXZ2IiJRiCjfiPE0Gg4cfnIyDg6tsalo/MoCaoX5kZptZsiPJQQWKiEhppHAjzuMdCE0GWbdtHFhsGAZ9m1YEdNeUiIjkp3AjztX64ozFO3+FVNvOwPRpar0lfNXek5w6l2HvykREpJRSuBHnimgMlW4CczZs/sampjVC/WlUMZAcs4UF2xMdVKCIiJQ2CjfifLkDizd9DWbb5q3JHVisS1MiIpJL4Uacr+EA8CkHyYdhzxKbmt7RxBpuNhw8TULyeUdUJyIipYzCjTifhzc0G2LdtnFgcVSwD62rlcNigXlbEhxQnIiIlDYKN1Iy5F6a2rMEzhy0qWnupamZMUew2DhfjoiIuB6FGykZyteEGp0BC2yaalPTvk0r4uvpxq7EVFZppXARkTJP4UZKjtzbwmO+heyC39od5OvBoFaVAfh85X5HVCYiIqWIwo2UHHV6QkAkpJ+0zntjg0c6VMdkwB97TrIzIcVBBYqISGmgcCMlh5s7tHjQur1xik1NK4f40rOxdVK///1xwN6ViYhIKaJwIyVLywfBcINDq+H4TpuaPnZLDQDmbjlKYvIFR1QnIiKlgMKNlCyBUVC3p3XbxrM3zSoHc1O1ELJyLExdc9D+tYmISKmgcCMlT+7A4i3TIeOcTU0f62g9e/P9ukOcy8i2d2UiIlIKKNxIyVP9VgipARkpsP1nm5p2qRdGjQp+pF7IZsaGww4pT0RESjaFGyl5TKa/J/Xb8CXYMDGfyWTw6MWxN1NWHSA7x+yICkVEpARTuJGSqdkQcPOCxK1wNMampgNbVKS8nydHz57XauEiImWQwo2UTL4h1gU1web1prw93BjathpgndRPSzKIiJQtCjdScuUOLN4+E9JP29T0/jZV8HI3se1oMusO2NZWRERKN4UbKbkqtYbwxpB9Abb8YFPT8v5e3NWyEgBfaEkGEZEyReFGSi7DgNYXBxZvnGLTwGKwLslgGLBs13H2Hk91QIEiIlISKdxIydZ4EHgGwKm9cGCFTU1rhPpze/1wQEsyiIiUJQo3UrJ5+UPTwdZtG2csBnj84qR+s2KOciK14CuNi4hI6aVwIyVf7pw3u+ZDqm23dresWo7mVYLJzDHz7dqD9q9NRERKHIUbKfnCG0LlNmDOhphvbGpqGAaPX5zU75s/D3E+M8cRFYqISAmicCOlQ+5t4ZumQo5ta0Z1axhBlRBfzqZn8fMmLckgIuLqFG6kdGjQD3zLQ8pR2LPIpqZuJoNHOlQH4H+rDpBj1qR+IiKuTOFGSgd3L2h+v3V7g20zFgPc3aoSQT4eHDqVzpK/tCSDiIgrU7iR0qPlQ4AB+5bBadtu7fb1dOeBNlUB65IMIiLiuhRupPQIqQ61uli3N31lc/Oh7ari6WYiJv4smw5pSQYREVelcCOlS+5t4Zu/g2zb5q0JC/BmQPOKAHyxUpP6iYi4KoUbKV1qd4fAipB+Cv76xebmj95iHVi86K9EDp5Ms3d1IiJSApSIcDNx4kSqVauGt7c3N998M+vXr7/msVOnTsUwjHwPb2/vYqxWnMrNHVoOs24XYmBx7fAAOtcNxWKBL1fp7I2IiCtyeriZMWMGo0aN4rXXXiMmJoamTZvSvXt3jh8/fs02gYGBJCQk5D0OHTpUjBWL07UYCiZ3OPwnJO2wufljF5dk+GnTYU6nZdq7OhERcTKnh5sJEybw2GOP8dBDD9GgQQMmT56Mr68vU6Zcex0hwzCIiIjIe4SHhxdjxeJ0ARFQr7d1uxDrTbWtUZ5GFQO5kGXmuz8VjEVEXI1Tw01mZiabNm2ia9eueftMJhNdu3Zl7dq112x37tw5qlatSuXKlenXrx87dtj+17uUcq0uzli8ZQZknLOpqWEYPHZxSYav1xzkQpaWZBARcSVODTcnT54kJyfnijMv4eHhJCZefaK1unXrMmXKFH755Re+++47zGYz7dq148iRI1c9PiMjg5SUlHwPcQHVO0L5WpCZCtt+tLl5r8aRVAz24VRaJrM3H3VAgSIi4ixOvyxlq7Zt2zJ06FCaNWtGp06dmDVrFqGhoXz22WdXPX7cuHEEBQXlPSpXrlzMFYtDGMbft4VvmAIW25ZU8HAz8VD7agD874/9mLUkg4iIy3BquKlQoQJubm4kJSXl25+UlERERESB+vDw8KB58+bs3bv3qs+PHj2a5OTkvMfhw1o40WU0vRfcvSFpGxzZYHPze26qQoC3O/tOpLE87toD2EVEpHRxarjx9PSkZcuWLFu2LG+f2Wxm2bJltG3btkB95OTksG3bNiIjI6/6vJeXF4GBgfke4iJ8Q6DRndbtQtwW7u/lzn03VQG0JIOIiCtx+mWpUaNG8cUXX/D111+zc+dO/vGPf5CWlsZDDz0EwNChQxk9enTe8WPHjmXx4sXs37+fmJgY7r//fg4dOsSjjz7qrB9BnCl3YPGO2ZBu+5IKw9pXw91ksO7AabYcPmvf2kRExCmcHm4GDx7Mu+++y6uvvkqzZs2IjY1l4cKFeYOM4+PjSUhIyDv+zJkzPPbYY9SvX59evXqRkpLCmjVraNCggbN+BHGmii0gsinkZEDs9zY3jwzyoW/TKAC++ENnb0REXIFhsdg4ErOUS0lJISgoiOTkZF2ichWbvoZfnwb/cBi+HnyCbWr+17EUen30ByYDVrzQmcohvo6pU0RECs2Wz2+nn7kRKbImg623hZ9LgmVv2Ny8QVQgt9SugNkCU1ZrSQYRkdJO4UZKPw9v6POhdXvjFDh07QkgryV3Ur8ZGw6TnJ5lz+pERKSYKdyIa6jWAZo/YN3+9RnIzrCp+S21K1AvIoD0zBymrY93QIEiIlJcFG7Eddw+FvxC4WQcrPrApqaGYfDoxbM3X60+QGa22QEFiohIcVC4EdfhGwI9x1u3/3gXTuy2qXnfplGEB3pxPDWDuVuOOaBAEREpDgo34loaDoTa3SAn03p5ylzwMzCe7iaGtasOwBcr91PGbiQUEXEZCjfiWgwDer8HHr4QvwY2f2tT8/turoKfpxtxSams3HPSQUWKiIgjKdyI6wmuArf927q9ZAykJl3/+EsE+XgwuLV1SYb/aVI/EZFSSeFGXNNNT0BkM7iQDAtfsqnpQ+2r4WYy+GPPSf46luKY+kRExGEUbsQ1ublD34/AcLOuOxW3sMBNK4f40rORdVV6nb0RESl9FG7EdUU2hbbDrdvzn4OMcwVu+nhH623hc7ccIyH5vCOqExERB1G4Edd268sQXBVSjsDy/xa4WZNKwdxcPYRss4Wpqw86rj4REbE7hRtxbZ5+cMcE6/a6yXB0U4Gb5p69mbYuntQLWpJBRKS0ULgR11erKzQeBBYzzH0GcgoWVDrXDaNmqB+pGdnM2HDYwUWKiIi9KNxI2dBjHPiUg6RtsHZigZqYTEbegppfrT5IVo6WZBARKQ0UbqRs8KsA3d+ybke/DacLdhdU/+YVqeDvydGz55kVc8SBBYqIiL0o3EjZ0fReqN4Rss/DvFFQgOUVvD3ceKSD9ezN63P/0rw3IiKlgMKNlB2GAXd8AO7esH85bP2xQM0eu6U6t9SuwPmsHB77ZiOn0zIdW6eIiBSJwo2ULeVrQqcXrduLRkPaqRs2cXcz8cm9Laha3pejZ88z/PsYjb8RESnBFG6k7Gn3NIQ1hPRTsPjfBWoS5OvBF0Nb4efpxtr9p/jv/J0OLlJERApL4UbKHjcP69IMGLBlGuxbXqBmdcIDmDC4GQBT1xzkx426PVxEpCRSuJGyqVIruOlx6/a8kZCZXqBm3RtGMLJrbQD+PXs7MfFnHFSgiIgUlsKNlF1dxkBgRThzEFa+U+BmT99Wm24NwsnMMfPkt5tISrnguBpFRMRmCjdSdnkFQO/3rNurP4LE7QVqZjIZTBjcjDrh/hxPzeCJbzdxISvHgYWKiIgtFG6kbKvbExr0A0sO/Po0mAsWUvy93PliaCuCfDyIPXyWf8/ZjqUA8+aIiIjjKdyI9HwHvIKsi2pu+F+Bm1Ut78cn9zXHZMDPm44wdc1Bx9UoIiIFpnAjEhABt79u3V42FpILvszCLbVD+Vev+gD8Z/5O1uw96YACRUTEFgo3IgAthkGVtpB5DuY/V6ClGXI90qE6A5pXJMdsYfi0GA6fLtidVyIi4hgKNyIAJhP0+RBMHrB7Ifz1S4GbGobBuIGNaVIpiDPpWTz2zUbSM7MdWKyIiFyPwo1IrtC6cMtz1u0FL8L5swVu6u3hxmcPtKSCvxe7ElN54aetGmAsIuIkCjcil7plFJSvDeeSYOnrNjWNDPJh8v0t8HAzmL8tgU+j9zmmRhERuS6FG5FLuXtZL08BbPoKDq21qXmraiGM7dcIgHcXx7FsZ5K9KxQRkRtQuBG5XLX20OJB6/avT0N2hk3N772pCve3qYLFAs9Mj2Xv8XMOKFJERK5F4Ubkam4fC/7hcHI3rHrf5uav3tGQm6qFcC4jm8e/2Ujy+SwHFCkiIlejcCNyNT7B0HO8dfuP9+BEnE3NPd1NfHp/C6KCvNl/Mo1npm8mx6wBxiIixUHhRuRaGvSHOj0gJxN+HQlms03NK/h78fnQVnh7mIiOO8G7i20LSCIiUjgKNyLXYhjQ613w9If4NbDwJZvH3zSqGMT4O5sAMCl6H3O3HHNEpSIicgmFG5HrCa4M3f5j3V7/OfyvK5zcY1MX/ZpV5IlONQB48ectbD+abO8qRUTkEgo3IjfS6iG4dzr4hEDiVvisI2z62qYlGl7sXo9OdUK5kGXmiW83ceqcbWeARESk4BRuRAqibk/4xxqocStkpVtvEf/pQTh/pkDN3UwGH93TnOoV/Dh69jz/+D6GrBzbxvCIiEjBKNyIFFRgJNw/23qbuMnduv7UpA5waE2Bmgf5evDF0Jb4e7mz/sBp3pz3l4MLFhEpmxRuRGxhMkH7Z+CRJRBSA1KOwNTesPwtyLnxYpm1wgJ4f3AzAL5Ze4jp6+MdXLCISNmjcCNSGBVbwBMrodkQsJhhxXiY2gvOHLph09sbhPPc7XUAGPPLdjYdOu3oakVEyhSFG5HC8gqA/p/CnV+CVyAcXgeTb4HtM2/YdMRttejZKIKsHAtPfhdDYvKFYihYRKRsULgRKarGd8GTq6DSTZCRDD8/DHOegozUazYxDIN3725KvYgATqRm8Pi3Gzl4Mq0YixYRcV0KNyL2UK4qPLQAOr4Ihgliv7feMn405ppN/Lzc+fyBVgT7erD1SDK3vRfNyOmb2ZN07VAkIiI3ZlgsNkzW4QJSUlIICgoiOTmZwMBAZ5cjrujgapj1uHWwsckdbhsD7Z62Dka+irjEVN5esJPlcScA68TIPRtFMLxzLRpGBRVn5SIiJZYtn98KNyKOcP4MzH0ads61fl/jVhjwGQREXLPJ9qPJfPL7XhbuSMzb17V+GCNuq02zysGOrVdEpIRTuLkOhRspNhYLxHwDC1+2TvznWx76TbROCHgdcYmpTFy+l3lbj5G7kPgttSvwz9tqc1P1kGIoXESk5FG4uQ6FGyl2J3bDzIchcZv1+5set04E6OFz3Wb7T5zj0+h9zN58lJyLKeem6iH887ZadKhVAcMwHF25iEiJoXBzHQo34hTZGbD0DfhzovX7sAZw1xQIq3/DpodPpzN5xT5+2niEzItLNjSrHMw/b6vFbfXCFHJEpExQuLkOhRtxqj1LYc6TkHYC3L2h+3+h1SPWUcQ3kJB8ns9X7mfaungysq0hp35kIP+8rRY9GkZgMinkiIjrUri5DoUbcbpzx2HOP2DvUuv3dXtB30/Ar3yBmp9IzeDLVQf4du1B0jJzAKgV5s+IzrW4o0kk7m6a4UFEXI/CzXUo3EiJYDbDusmw9DXIybQONq7XG2p3t95Z5eV/wy7OpGXy1ZqDfLX6AKkXrOtaVSvvy1O31qJ/84p4uivkiIjrULi5DoUbKVEStsDMR+Hk7r/3uXlC1fZQpzvU7gbla163i5QLWXy79hD/+2M/Z9KzAKgY7MOTnWpwd6vKeHu4OfInEBEpFgo316FwIyVOdiYcXAm7F8OeRXDmYP7ny9eyntGp0w2qtAN3z6t2k56ZzbR18Xy2cj8nUjMACAvw4vGONbj3pir4ebk7+AcREXEchZvrULiREs1igZN7rCFn9yKIXwvm7L+f9/S3XrbKPatzlUkBL2Tl8OPGw0yO3sexiwtyergZNIgKokWVYFpWLUeLKuWICr7+regiIiWJws11KNxIqXIhBfYvv3hWZzGkHc//fGTTi2d1ukNUi3xLPGRmm5m9+QiTV+znwFUW5YwM8qZFlXK0qFqOFlWCaRgVpHE6IlJiKdxch8KNlFpmMyTEwp4l1jM7R2OAS96+vhWgVlfr5auaXcAnGACLxcKRM+eJiT9DzKEzxMSf5a+ElLyJAXN5uptoUjGIllXL0bxKOVpUDSYswLvYfjwRketRuLkOhRtxGedOwN4l1stX+36HjJS/nzPcoEob66WrOt0htF6+uXTSM7PZeiSZTYfOsDn+DJsOnckbjHypyiE+tKhSLu9SVr2IAN1qLiJOoXBzHQo34pJysuDwOmvQ2bMYTuzK/7xnAARGQkAkBEZd9jUSS0AkBy/4s+lwSt4ZnrikVC7/v4OPhxtNKwflhZ0WVcpRzu/qA5xFROxJ4eY6FG6kTDhzyBpydi+Cg39A9oUbtzHcwD88LwRl+kZw1BzM7vQAYs76sOa4J/svBJBG/oHINSr4UbGcD8G+npTz9cj7Ws7Xk6CLX3P3B3q7a7kIESmUUhduJk6cyP/93/+RmJhI06ZN+fjjj7npppuuefxPP/3EmDFjOHjwILVr12b8+PH06tWrQK+lcCNlTtYFOBsPqccgJeGSrwmQcsz69VwSWMwF687dj9Om8hzJCeZgZhCJlnKkWnxJx4vzeJFu8b5k2/o1dzvD5I2Xtx/Bfp55oSfI52IY8vMk+GIYCva5GJL8PAjy8cDb3U3LS4iUcaUq3MyYMYOhQ4cyefJkbr75Zj744AN++ukn4uLiCAsLu+L4NWvW0LFjR8aNG8cdd9zBtGnTGD9+PDExMTRq1OiGr6dwI3IVOdnWO7HyhZ9jkJr4dwBKSYDM1CK/lNlicB5PawCyeJGONxfwJN3iddWAdN7iSTbuWEzWB24eGCYPcPMAN3dMbh4Y7h6Y3C4+3D0xuXvg5u6Jm4cnbu4euLt74O7hhbuHJ+4eHnh4eOHh6Wl9eHhicnPHzeSGm5sb7m4m3EwG7qaLX92Mi98b+febDNzc8u83GejMlIiDlKpwc/PNN9O6dWs++eQTAMxmM5UrV+af//wnL7/88hXHDx48mLS0NObNm5e3r02bNjRr1ozJkyff8PUUbkSKICP16gEoIxUy0yDrPGSlX9xOh8x0yLq4vyCXxkqAHIuBGRNmTORgwoxx/W1L/v0WTJgNExYMzIYbZkyAgQUDy8X91q9gwZS3j9znLj4PJiyGgQUTGFx83roP42KfF79aB4sb1uew9mMY1v157THlHYdxadtLnst7Pv+x+b5iYJgufT3j70B3WZ/GJX0beT+DcfGQ3Nfg4lfr8RbDwMDI68e4+Jy1PvK9Tr7+8/r5u1bjkv6NvJ+PfM9d+rsyrujb+tVy2c9t7fKy7dy683bnPme6rM0l9VysJd/v49If0zD9fXzez8dlP0felnXDZMp9tbzaDS7p99LwbVxaY/768g7J7c1k5Ps+fz/52xsYeHj7EFmxKvZky+e3U6cszczMZNOmTYwePTpvn8lkomvXrqxdu/aqbdauXcuoUaPy7evevTtz5sy56vEZGRlkZGTkfZ+SknLV40SkALwCIDQAQuvY3tackz/wZKZbv8/bd2kosn61ZKaTnZFOTnYm5uwszDmZWLKzsORkYc7JghzrtsWcDTlZGLlfLdkY5mwMcxYmczYmSzYmSw4mSzZulmzcyLlmmW6G5eLz1z4mH1tO1Fgu+yriona51yfy33867fWdGm5OnjxJTk4O4eHh+faHh4eza9euq7ZJTEy86vGJiYlXPX7cuHG88cYb9ilYRArP5GYNR14BBW5iAB4XH3ZlsVhnfs7Jsn61mK0Pc87F7ZzLts359lssOZhzzOTkZGE255CTk4M5O5scsxlzTjY5OdmYzWZysq19W8xmzBe/WvK+5mCxWLCYzZC733Lp83/vx5L7vcXaznzJPosFCxYMsxkLFuvPdsn+y783LGasu637wHwxbF38Pt9x1vNLV3y9pB+wYLFYzzld9VgL13zOuPj839u5/V/cd+lzlxybey6Fi69rgbz2uedjcvv5u18ueS7//r9fw8q4bL+F3BMyF7ctl/bPJW0v7Zu/ty977u88bLnk9S6t8Rp9Xnzeku/73D6usf+yJG1c9rNeuv/y7fy5/cr+rnec2c25d1G6/GIzo0ePznemJyUlhcqVKzuxIhFxOsO4OGancLHJANwuPkTkSg2c/PpODTcVKlTAzc2NpKSkfPuTkpKIiLhyzRyAiIgIm4738vLCy8vLPgWLiIhIiefUqUY9PT1p2bIly5Yty9tnNptZtmwZbdu2vWqbtm3b5jseYMmSJdc8XkRERMoWp1+WGjVqFA8++CCtWrXipptu4oMPPiAtLY2HHnoIgKFDh1KxYkXGjRsHwDPPPEOnTp1477336N27N9OnT2fjxo18/vnnzvwxREREpIRwergZPHgwJ06c4NVXXyUxMZFmzZqxcOHCvEHD8fHxmC5Z6bhdu3ZMmzaNf//73/zrX/+idu3azJkzp0Bz3IiIiIjrc/o8N8VN89yIiIiUPrZ8fmt5XxEREXEpCjciIiLiUhRuRERExKUo3IiIiIhLUbgRERERl6JwIyIiIi5F4UZERERcisKNiIiIuBSFGxEREXEpTl9+objlTsickpLi5EpERESkoHI/twuysEKZCzepqakAVK5c2cmViIiIiK1SU1MJCgq67jFlbm0ps9nMsWPHCAgIwDAMu/adkpJC5cqVOXz4sEPWrXJk/6W5dkf3X5prL+39l+baS3v/pbn20t5/aa7dkf1bLBZSU1OJiorKt6D21ZS5Mzcmk4lKlSo59DUCAwMduiinI/svzbU7uv/SXHtp7780117a+y/NtZf2/ktz7Y7q/0ZnbHJpQLGIiIi4FIUbERERcSkKN3bk5eXFa6+9hpeXV6nrvzTX7uj+S3Ptpb3/0lx7ae+/NNde2vsvzbUXR/8FUeYGFIuIiIhr05kbERERcSkKNyIiIuJSFG5ERETEpSjciIiIiEtRuLGDlStX0qdPH6KiojAMgzlz5tit73HjxtG6dWsCAgIICwujf//+xMXF2a3/SZMm0aRJk7zJltq2bcuCBQvs1v+l3n77bQzDYOTIkXbp7/XXX8cwjHyPevXq2aXvXEePHuX++++nfPny+Pj40LhxYzZu3GiXvqtVq3ZF/YZhMHz4cLv0n5OTw5gxY6hevTo+Pj7UrFmTN998s0DrshREamoqI0eOpGrVqvj4+NCuXTs2bNhQqL5u9B6yWCy8+uqrREZG4uPjQ9euXdmzZ4/d+p81axbdunWjfPnyGIZBbGys3erPysripZdeonHjxvj5+REVFcXQoUM5duyYXWp//fXXqVevHn5+fpQrV46uXbuybt06u9R+uSeffBLDMPjggw/s1v+wYcOueA/06NHDbrXv3LmTvn37EhQUhJ+fH61btyY+Pt4u/V/t/WsYBv/3f/9nl/7PnTvHiBEjqFSpEj4+PjRo0IDJkycXqO+C9J+UlMSwYcOIiorC19eXHj16FPh9VZDPpgsXLjB8+HDKly+Pv78/d955J0lJSQWuvygUbuwgLS2Npk2bMnHiRLv3vWLFCoYPH86ff/7JkiVLyMrKolu3bqSlpdml/0qVKvH222+zadMmNm7cyG233Ua/fv3YsWOHXfrPtWHDBj777DOaNGli134bNmxIQkJC3mPVqlV26/vMmTO0b98eDw8PFixYwF9//cV7771HuXLl7NL/hg0b8tW+ZMkSAO6++2679D9+/HgmTZrEJ598ws6dOxk/fjzvvPMOH3/8sV36f/TRR1myZAnffvst27Zto1u3bnTt2pWjR4/a3NeN3kPvvPMOH330EZMnT2bdunX4+fnRvXt3Lly4YJf+09LS6NChA+PHj7e59hv1n56eTkxMDGPGjCEmJoZZs2YRFxdH37597VJ7nTp1+OSTT9i2bRurVq2iWrVqdOvWjRMnTtil/1yzZ8/mzz//JCoqqkD92tJ/jx498r0XfvjhB7v0vW/fPjp06EC9evWIjo5m69atjBkzBm9vb7v0f2nNCQkJTJkyBcMwuPPOO+3S/6hRo1i4cCHfffcdO3fuZOTIkYwYMYK5c+cWuX+LxUL//v3Zv38/v/zyC5s3b6Zq1ap07dq1QJ8vBflsevbZZ/n111/56aefWLFiBceOHWPgwIEFqr3ILGJXgGX27NkO6//48eMWwLJixQqHvUa5cuUs//vf/+zWX2pqqqV27dqWJUuWWDp16mR55pln7NLva6+9ZmnatKld+rqal156ydKhQweH9X+5Z555xlKzZk2L2Wy2S3+9e/e2PPzww/n2DRw40DJkyJAi952enm5xc3OzzJs3L9/+Fi1aWF555ZUi9X35e8hsNlsiIiIs//d//5e37+zZsxYvLy/LDz/8UOT+L3XgwAELYNm8ebPN/Rak/1zr16+3AJZDhw7Zve/k5GQLYFm6dKlNfV+v/yNHjlgqVqxo2b59u6Vq1aqW999/3+a+r9X/gw8+aOnXr1+h+rtR34MHD7bcf//9Re77Wv1frl+/fpbbbrvNbv03bNjQMnbs2Hz7Cvseu7z/uLg4C2DZvn173r6cnBxLaGio5YsvvrC5/8s/m86ePWvx8PCw/PTTT3nH7Ny50wJY1q5da3P/ttKZm1ImOTkZgJCQELv3nZOTw/Tp00lLS6Nt27Z263f48OH07t2brl272q3PXHv27CEqKooaNWowZMiQAp9uLoi5c+fSqlUr7r77bsLCwmjevDlffPGF3fq/VGZmJt999x0PP/yw3RZ0bdeuHcuWLWP37t0AbNmyhVWrVtGzZ88i952dnU1OTs4VfwH7+PjY9ewZwIEDB0hMTMz3309QUBA333wza9eutetrFZfk5GQMwyA4ONiu/WZmZvL5558TFBRE06ZN7dKn2WzmgQce4IUXXqBhw4Z26fNy0dHRhIWFUbduXf7xj39w6tSpIvdpNpuZP38+derUoXv37oSFhXHzzTfbddjApZKSkpg/fz6PPPKI3fps164dc+fO5ejRo1gsFpYvX87u3bvp1q1bkfvOyMgAyPceNplMeHl5Feo9fPln06ZNm8jKysr3vq1Xrx5VqlQplvetwk0pYjabGTlyJO3bt6dRo0Z263fbtm34+/vj5eXFk08+yezZs2nQoIFd+p4+fToxMTGMGzfOLv1d6uabb2bq1KksXLiQSZMmceDAAW655RZSU1Pt0v/+/fuZNGkStWvXZtGiRfzjH//g6aef5uuvv7ZL/5eaM2cOZ8+eZdiwYXbr8+WXX+aee+6hXr16eHh40Lx5c0aOHMmQIUOK3HdAQABt27blzTff5NixY+Tk5PDdd9+xdu1aEhIS7FD93xITEwEIDw/Ptz88PDzvudLkwoULvPTSS9x77712W1Rw3rx5+Pv74+3tzfvvv8+SJUuoUKGCXfoeP3487u7uPP3003bp73I9evTgm2++YdmyZYwfP54VK1bQs2dPcnJyitTv8ePHOXfuHG+//TY9evRg8eLFDBgwgIEDB7JixQo7Vf+3r7/+moCAALtedvn4449p0KABlSpVwtPTkx49ejBx4kQ6duxY5L5zg8bo0aM5c+YMmZmZjB8/niNHjtj8Hr7aZ1NiYiKenp5XBPjiet+WuVXBS7Phw4ezfft2u/9lXLduXWJjY0lOTubnn3/mwQcfZMWKFUUOOIcPH+aZZ55hyZIlBb7GbYtLz0A0adKEm2++mapVq/Ljjz/a5a8ns9lMq1ateOuttwBo3rw527dvZ/LkyTz44INF7v9SX375JT179rR5PMP1/Pjjj3z//fdMmzaNhg0bEhsby8iRI4mKirJL/d9++y0PP/wwFStWxM3NjRYtWnDvvfeyadMmO1TvmrKyshg0aBAWi4VJkybZrd/OnTsTGxvLyZMn+eKLLxg0aBDr1q0jLCysSP1u2rSJDz/8kJiYGLudUbzcPffck7fduHFjmjRpQs2aNYmOjqZLly6F7tdsNgPQr18/nn32WQCaNWvGmjVrmDx5Mp06dSpa4ZeZMmUKQ4YMsev/6z7++GP+/PNP5s6dS9WqVVm5ciXDhw8nKiqqyGfCPTw8mDVrFo888gghISG4ubnRtWtXevbsafNNB476bCoKnbkpJUaMGMG8efNYvnw5lSpVsmvfnp6e1KpVi5YtWzJu3DiaNm3Khx9+WOR+N23axPHjx2nRogXu7u64u7uzYsUKPvroI9zd3Yv8l9nlgoODqVOnDnv37rVLf5GRkVcEvPr169v10hfAoUOHWLp0KY8++qhd+33hhRfyzt40btyYBx54gGeffdZuZ9Fq1qzJihUrOHfuHIcPH2b9+vVkZWVRo0YNu/SfKyIiAuCKuyySkpLynisNcoPNoUOHWLJkid3O2gD4+flRq1Yt2rRpw5dffom7uztffvllkfv9448/OH78OFWqVMl7Dx86dIjnnnuOatWqFb3wq6hRowYVKlQo8vu4QoUKuLu7F8t7+I8//iAuLs6u7+Hz58/zr3/9iwkTJtCnTx+aNGnCiBEjGDx4MO+++65dXqNly5bExsZy9uxZEhISWLhwIadOnbLpPXytz6aIiAgyMzM5e/ZsvuOL632rcFPCWSwWRowYwezZs/n999+pXr26w1/TbDbnXY8tii5durBt2zZiY2PzHq1atWLIkCHExsbi5uZmh2r/du7cOfbt20dkZKRd+mvfvv0Vtzbu3r2bqlWr2qX/XF999RVhYWH07t3brv2mp6djMuV/i7u5ueX9RWsvfn5+REZGcubMGRYtWkS/fv3s2n/16tWJiIhg2bJleftSUlJYt26dXceGOVJusNmzZw9Lly6lfPnyDn09e72HH3jgAbZu3ZrvPRwVFcULL7zAokWL7FDplY4cOcKpU6eK/D729PSkdevWxfIe/vLLL2nZsqXdxjmB9b+ZrKysYnkPBwUFERoayp49e9i4cWOB3sM3+mxq2bIlHh4e+d63cXFxxMfHF8v7Vpel7ODcuXP5/so4cOAAsbGxhISEUKVKlSL1PXz4cKZNm8Yvv/xCQEBA3rXKoKAgfHx8itQ3wOjRo+nZsydVqlQhNTWVadOmER0dbZf/cQUEBFwxNsjPz4/y5cvbZczQ888/T58+fahatSrHjh3jtddew83NjXvvvbfIfYP1NsZ27drx1ltvMWjQINavX8/nn3/O559/bpf+wfoh9NVXX/Hggw/i7m7ft2OfPn3473//S5UqVWjYsCGbN29mwoQJPPzww3bpf9GiRVgsFurWrcvevXt54YUXqFevHg899JDNfd3oPTRy5Ej+85//ULt2bapXr86YMWOIioqif//+dun/9OnTxMfH5809k/uBGBERUaC/Mq/Xf2RkJHfddRcxMTHMmzePnJycvPdxSEgInp6ehe67fPny/Pe//6Vv375ERkZy8uRJJk6cyNGjRws8pcCNfjeXBzEPDw8iIiKoW7dukfsPCQnhjTfe4M477yQiIoJ9+/bx4osvUqtWLbp3717k2l944QUGDx5Mx44d6dy5MwsXLuTXX38lOjq6yLXn/r89JSWFn376iffee69AfdrSf6dOnXjhhRfw8fGhatWqrFixgm+++YYJEybYpf+ffvqJ0NBQqlSpwrZt23jmmWfo379/gQYs3+izKSgoiEceeYRRo0YREhJCYGAg//znP2nbti1t2rSx8TdVCA6/H6sMWL58uQW44vHggw8Wue+r9QtYvvrqqyL3bbFYLA8//LClatWqFk9PT0toaKilS5culsWLF9ul76ux563ggwcPtkRGRlo8PT0tFStWtAwePNiyd+9eu/Sd69dff7U0atTI4uXlZalXr57l888/t2v/ixYtsgCWuLg4u/ZrsVgsKSkplmeeecZSpUoVi7e3t6VGjRqWV155xZKRkWGX/mfMmGGpUaOGxdPT0xIREWEZPny45ezZs4Xq60bvIbPZbBkzZowlPDzc4uXlZenSpYtNv7Mb9f/VV19d9fnXXnutyP3n3l5+tcfy5cuL1Pf58+ctAwYMsERFRVk8PT0tkZGRlr59+1rWr19vt9/N5Wy9Ffx6/aenp1u6detmCQ0NtXh4eFiqVq1qeeyxxyyJiYl2q/3LL7+01KpVy+Lt7W1p2rSpZc6cOXapPddnn31m8fHxKdR/+zfqPyEhwTJs2DBLVFSUxdvb21K3bl3Le++9V+DpIm7U/4cffmipVKmSxcPDw1KlShXLv//97wL//6Egn03nz5+3PPXUU5Zy5cpZfH19LQMGDLAkJCTY8isqNONikSIiIiIuQWNuRERExKUo3IiIiIhLUbgRERERl6JwIyIiIi5F4UZERERcisKNiIiIuBSFGxEREXEpCjciUuZFR0djGMYV6+CISOmkcCMiIiIuReFGREREXIrCjYg4ndlsZty4cVSvXh0fHx+aNm3Kzz//DPx9yWj+/Pk0adIEb29v2rRpw/bt2/P1MXPmTBo2bIiXlxfVqlW7YiHDjIwMXnrpJSpXroyXlxe1atXiyy+/zHfMpk2baNWqFb6+vrRr1+6KFaVFpHRQuBERpxs3bhzffPMNkydPZseOHTz77LPcf//9rFixIu+YF154gffee48NGzYQGhpKnz59yMrKAqyhZNCgQdxzzz1s27aN119/nTFjxjB16tS89kOHDuWHH37go48+YufOnXz22Wf4+/vnq+OVV17hvffeY+PGjbi7u9ttBXURKV5aOFNEnCojI4OQkBCWLl1K27Zt8/Y/+uijpKen8/jjj9O5c2emT5/O4MGDATh9+jSVKlVi6tSpDBo0iCFDhnDixAkWL16c1/7FF19k/vz57Nixg927d1O3bl2WLFlC165dr6ghOjqazp07s3TpUrp06QLAb7/9Ru/evTl//jze3t4O/i2IiD3pzI2IONXevXtJT0/n9ttvx9/fP+/xzTffsG/fvrzjLg0+ISEh1K1bl507dwKwc+dO2rdvn6/f9u3bs2fPHnJycoiNjcXNzY1OnTpdt5YmTZrkbUdGRgJw/PjxIv+MIlK83J1dgIiUbefOnQNg/vz5VKxYMd9zXl5e+QJOYfn4+BToOA8Pj7xtwzAA63ggESlddOZGRJyqQYMGeHl5ER8fT61atfI9KleunHfcn3/+mbd95swZdu/eTf369QGoX78+q1evztfv6tWrqVOnDm5ubjRu3Biz2ZxvDI+IuC6duRERpwoICOD555/n2WefxWw206FDB5KTk1m9ejWBgYFUrVoVgLFjx1K+fHnCw8N55ZVXqFChAv379wfgueeeo3Xr1rz55psMHjyYtWvX8sknn/Dpp58CUK1aNR588EEefvhhPvroI5o2bcqhQ4c4fvw4gwYNctaPLiIOonAjIk735ptvEhoayrhx49i/fz/BwcG0aNGCf/3rX3mXhd5++22eeeYZ9uzZQ7Nmzfj111/x9PQEoEWLFvz444+8+uqrvPnmm0RGRjJ27FiGDRuW9xqTJk3iX//6F0899RSnTp2iSpUq/Otf/3LGjysiDqa7pUSkRMu9k+nMmTMEBwc7uxwRKQU05kZERERcisKNiIiIuBRdlhIRERGXojM3IiIi4lIUbkRERMSlKNyIiIiIS1G4EREREZeicCMiIiIuReFGREREXIrCjYiIiLgUhRsRERFxKQo3IiIi4lL+H4W6RQj9sX6pAAAAAElFTkSuQmCC)\n","\n","![acc.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAjsAAAHHCAYAAABZbpmkAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAWwJJREFUeJzt3Xd4VGX6xvHvpIdUQkIKkBB6R9qPqtSliEhbiqKCYAUUUFDRxYJKW0Fp1kXUXVABxVVQEJAiiFRRmnSkhp6EJKSQOb8/YmaJtGTmTGYS7s915XJyZuaZJ4nj3L7ve95jMQzDQERERKSY8nB1AyIiIiLOpLAjIiIixZrCjoiIiBRrCjsiIiJSrCnsiIiISLGmsCMiIiLFmsKOiIiIFGsKOyIiIlKsKeyIiIhIsaawIyIuUb58eQYMGODqNq7r8OHDWCwWPvroI1e3IiIOUtgRcbKPPvoIi8WCxWJh7dq1V91vGAblypXDYrFw11135bnPYrEwdOjQG9Zv1aqVrb7FYiEsLIxGjRrx4YcfYrVaC9TrqlWr8tS60Zc7+Oabb/Dw8CAhIcHVrYiIG/NydQMitwo/Pz/mzp1LixYt8hxfvXo1x44dw9fX1+7aZcuWZfz48QCcOXOGTz75hEGDBrF3714mTJiQ7zrVq1fn3//+d55jo0ePJjAwkBdeeMHu/q5lz549eHg49v9bixcvpkGDBkRFRZnUlYgURwo7IoXkzjvvZP78+UybNg0vr/+99ebOnUuDBg04e/as3bVDQkK47777bN8/+uijVK1alRkzZvDqq6/i7e2drzqRkZF56gBMmDCB8PDwq45fyWq1kpmZiZ+fX757diTc5fr2228ZOHCgw3VuJampqQQEBLi6DZFCpWkskUJyzz33cO7cOZYtW2Y7lpmZyYIFC7j33ntNfa0SJUrQpEkTUlNTOXPmDABpaWn8/vvvDoWqXLnTa3PmzKFmzZr4+vqyZMkSAN544w2aNWtGqVKl8Pf3p0GDBixYsOCqGn9ds5M73bdu3TqeeuopIiIiCAgIoHv37raf4Urbt2/n6NGjdO7cmVOnTuHl5cUrr7xy1eP27NmDxWJhxowZAJw/f56RI0dSu3ZtAgMDCQ4OplOnTvz66692/S4KUi89PZ2XX36ZKlWq4OfnR3R0ND169ODAgQO2x1itVqZOnUrt2rXx8/MjIiKCjh07snnzZuDGa4ksFgsvv/yy7fuXX34Zi8XCrl27uPfeeylZsqRtZPG3335jwIABVKhQAT8/P6Kiohg4cCDnzp27qu7x48cZNGgQMTEx+Pr6Eh8fz+OPP05mZiYHDx7EYrHw5ptvXvW8n376CYvFwqefflrQX6uIqRR2RApJ+fLladq0aZ7/8H/33XckJSXRt29f01/v4MGDeHp6EhoaCsDGjRupXr267UPfUT/88AMjRoygT58+TJ06lfLlywMwdepU6tWrx9ixYxk3bhxeXl706tWLxYsX56vuE088wa+//spLL73E448/zjfffHPNdUvffvstpUuXpmHDhkRGRtKyZUvmzZt31eM+//xzPD096dWrF5Dze/nqq6+46667mDJlCqNGjWL79u20bNmSEydOFPj3kN962dnZ3HXXXbzyyis0aNCAyZMnM2zYMJKSktixY4ftcYMGDWL48OGUK1eOiRMn8txzz+Hn58fPP/9c4N5y9erVi7S0NMaNG8fDDz8MwLJlyzh48CAPPvgg06dPp2/fvnz22WfceeedGIZhe+6JEyf4v//7Pz777DP69OnDtGnTuP/++1m9ejVpaWlUqFCB5s2bM2fOnKted86cOQQFBdG1a1e7excxhSEiTjV79mwDMDZt2mTMmDHDCAoKMtLS0gzDMIxevXoZrVu3NgzDMOLi4ozOnTvneS5gDBky5Ib1W7ZsaVSrVs04c+aMcebMGWP37t3Gk08+aQBGly5dbI9buXKlARgvvfRSgfqvWbOm0bJly6v68vDwMHbu3HnV43N/tlyZmZlGrVq1jDZt2uQ5HhcXZ/Tv39/2fe7vqV27dobVarUdHzFihOHp6WkkJibmef7tt9+e5/nvvfeeARjbt2/P87gaNWrkee309HQjOzs7z2MOHTpk+Pr6GmPHjs1zDDBmz5591c94pfzW+/DDDw3AmDJlylU1cn/eH374wQCMJ5988rqPuVFff/37vvTSSwZg3HPPPVc99q9/J8MwjE8//dQAjDVr1tiOPfDAA4aHh4exadOm6/aU+7vfvXu37b7MzEwjPDw8z99IxFU0siNSiHr37s2lS5dYtGgRFy9eZNGiRaZMYf3+++9EREQQERFB9erVmT59Op07d+bDDz+0PaZVq1YYhpFnmsMRLVu2pEaNGlcd9/f3t92+cOECSUlJ3H777WzdujVfdR955JE8Z3vdfvvtZGdn88cff9iOJSYmsn79ejp37mw71qNHD7y8vPj8889tx3bs2MGuXbvo06eP7Zivr69tYXR2djbnzp0jMDCQqlWr5rvHK+W33hdffEF4eDhPPPHEVTVyf94vvvgCi8XCSy+9dN3H2OOxxx676tiVf6f09HTOnj1LkyZNAGx9W61WvvrqK7p06ULDhg2v21Pv3r3x8/PLM7qzdOlSzp49e8O1XiKFRWFHpBBFRETQrl075s6dy5dffkl2djZ///vfHa5bvnx5li1bxvLly1m7di0JCQksWrSI8PBwE7q+tvj4+GseX7RoEU2aNMHPz4+wsDAiIiJ45513SEpKylfd2NjYPN+XLFkSyAlOuZYuXQpA+/btbcfCw8Np27Ztnqmszz//HC8vL3r06GE7ZrVaefPNN6lcuTK+vr6Eh4cTERHBb7/9lu8er5TfegcOHKBq1ap5Fqf/1YEDB4iJiSEsLKzAfdzItf5W58+fZ9iwYURGRuLv709ERITtcbl9nzlzhuTkZGrVqnXD+qGhoXTp0oW5c+fajs2ZM4cyZcrQpk0bE38SEfvobCyRQnbvvffy8MMPk5CQQKdOnWxrahwREBBAu3btHG+uAK4cGcj1448/cvfdd3PHHXfw9ttvEx0djbe3N7Nnz87zQXgjnp6e1zxuXLGO5Ntvv6V58+aEhITkeUzfvn158MEH2bZtG7fddhvz5s2jbdu2eULfuHHjGDNmDAMHDuTVV18lLCwMDw8Phg8fXuB9iZxR72auN8KTnZ193edc62/Vu3dvfvrpJ0aNGsVtt91GYGAgVquVjh072tX3Aw88wPz58/npp5+oXbs2X3/9NYMHD3Z4ewERMyjsiBSy7t278+ijj/Lzzz/nmXIpDr744gv8/PxYunRpnlPLZ8+ebdprGIbBkiVLGDly5FX3devWjUcffdT2e927dy+jR4/O85gFCxbQunVrZs2aled4YmKiXSNh+a1XsWJFNmzYQFZW1nW3AqhYsSJLly7l/Pnz1x3dyR3pSkxMzHP8ymm+m7lw4QIrVqzglVde4cUXX7Qd37dvX57HRUREEBwcnGcB9fV07NiRiIgI5syZQ+PGjUlLS+P+++/Pd08izqTILVLIAgMDeeedd3j55Zfp0qVLob2umaeeX4+npycWiyXPKMPhw4f56quvTHuNTZs2cfr06TzrdXKFhobSoUMH5s2bx2effYaPjw/dunW7qscrR4kA5s+fz/Hjx+3qJ7/1evbsydmzZ695Nlzu83v27IlhGNc8hT73McHBwYSHh7NmzZo897/99tsF6vnKmrneeuutPN97eHjQrVs3vvnmG9up79fqCcDLy4t77rmHefPm8dFHH1G7dm3q1KmT755EnEkjOyIu0L9//3w/dvPmzbz22mtXHW/VqtVVuzHfyMaNG2ndujUvvfSSaYuU/6pz585MmTKFjh07cu+993L69GlmzpxJpUqV+O2330x5jcWLF1O+fPlrLo4G6NOnD/fddx9vv/02HTp0uGqa8K677mLs2LE8+OCDNGvWjO3btzNnzhwqVKhgVz/5rffAAw/wySef8NRTT7Fx40Zuv/12UlNTWb58OYMHD6Zr1660bt2a+++/n2nTprFv3z7blNKPP/5I69atbafgP/TQQ0yYMIGHHnqIhg0bsmbNGvbu3ZvvnoODg7njjjuYNGkSWVlZlClThu+//55Dhw5d9dhx48bx/fff07JlSx555BGqV6/OyZMnmT9/PmvXrs3z+33ggQeYNm0aK1euZOLEiXb9PkWcQWFHxM1t2LCBDRs2XHX81VdfLVDYKQxt2rRh1qxZTJgwgeHDhxMfH8/EiRM5fPiwaWHn22+/5c4777zu/XfffTf+/v5cvHgxz1lYuZ5//nlSU1OZO3cun3/+OfXr12fx4sU899xzdvWT33qenp58++23vP7668ydO5cvvviCUqVK0aJFC2rXrm173OzZs6lTpw6zZs1i1KhRhISE0LBhQ5o1a2Z7zIsvvsiZM2dYsGAB8+bNo1OnTnz33XeULl06333PnTuXJ554gpkzZ2IYBu3bt+e7774jJiYmz+PKlCnDhg0bGDNmDHPmzCE5OZkyZcrQqVMnSpQokeexDRo0oGbNmuzevZt+/foV5Nco4lQW46/jmCIiburUqVNER0ezaNGiGwYecZ169eoRFhbGihUrXN2KiI3W7IhIkZGUlMSLL75I69atXd2KXMPmzZvZtm0bDzzwgKtbEclDIzsiIuKQHTt2sGXLFiZPnszZs2c5ePBggS4KK+JsGtkRERGHLFiwgAcffJCsrCw+/fRTBR1xOxrZERERkWJNIzsiIiJSrCnsiIiISLGmfXbIuZDfiRMnCAoKcujKwiIiIlJ4DMPg4sWLxMTE3PA6bAo7wIkTJyhXrpyr2xARERE7HD16lLJly173foUdICgoCMj5ZQUHB7u4GxEREcmP5ORkypUrZ/scvx6FHbBNXQUHByvsiIiIFDE3W4KiBcoiIiJSrCnsiIiISLGmsCMiIiLFmsKOiIiIFGsKOyIiIlKsKeyIiIhIsaawIyIiIsWawo6IiIgUawo7IiIiUqwp7IiIiEix5tKws2bNGrp06UJMTAwWi4Wvvvoqz/2GYfDiiy8SHR2Nv78/7dq1Y9++fXkec/78efr160dwcDChoaEMGjSIlJSUQvwpRERExJ25NOykpqZSt25dZs6cec37J02axLRp03j33XfZsGEDAQEBdOjQgfT0dNtj+vXrx86dO1m2bBmLFi1izZo1PPLII4X1I4iIiIibsxiGYbi6Cci5iNfChQvp1q0bkDOqExMTw9NPP83IkSMBSEpKIjIyko8++oi+ffuye/duatSowaZNm2jYsCEAS5Ys4c477+TYsWPExMTk67WTk5MJCQkhKSlJFwKVfEk8m8Cl1CRXtyEiUmSER5fH28fX1Jr5/fx226ueHzp0iISEBNq1a2c7FhISQuPGjVm/fj19+/Zl/fr1hIaG2oIOQLt27fDw8GDDhg107979mrUzMjLIyMiwfZ+cnOy8H0SKrOzLl0k4so+zh3/l0onf8Ti3j6CUQ0RlHaEkFwl1dYMiIkXI0X5rKFe5rkte223DTkJCAgCRkZF5jkdGRtruS0hIoHTp0nnu9/LyIiwszPaYaxk/fjyvvPKKyR1LUZWWmsyJ/dtJPLqTrFN78L2wj5JpfxCTfZwylizKXOd56YZ3ofYpIlKkWVy3csZtw44zjR49mqeeesr2fXJyMuXKlXNhR+JshtXKudPHOXVwOynHdmGc3UOJ5INEpP9BNGeodK0nWSDD8OaEZxnOlyhPZslKeEdWJbRcTWIq1qJEYEhh/xgiIkWWKz9l3TbsREVFAXDq1Cmio6Ntx0+dOsVtt91me8zp06fzPO/y5cucP3/e9vxr8fX1xdfX3HlDKaCMi7DnO7icfvPH2uFkUjp/HD2Kx/n9BKccIubyEcJJJfw6j79AEAnesVwMjMdaqjL+MdUJL1+HqNjKxHt5Ee+ULkVEpDC4bdiJj48nKiqKFStW2MJNcnIyGzZs4PHHHwegadOmJCYmsmXLFho0aADADz/8gNVqpXHjxq5qXW4mPQlmd4ZT2532EtF/fl3Jalg46VGas35xpAVXxLN0VYLK1iCqQh1KRkRT0mndiIiIK7k07KSkpLB//37b94cOHWLbtm2EhYURGxvL8OHDee2116hcuTLx8fGMGTOGmJgY2xlb1atXp2PHjjz88MO8++67ZGVlMXToUPr27ZvvM7GkkGWlw6f35gSdEuFQtpFppc+kZLAn4SKXsrIBCAwKgfAq+ERVpWRsLWIq1qKMf8B11+CIiEjx5NKws3nzZlq3bm37PncdTf/+/fnoo4945plnSE1N5ZFHHiExMZEWLVqwZMkS/Pz8bM+ZM2cOQ4cOpW3btnh4eNCzZ0+mTZtW6D+L5IM1G758CP5YC77B8MBXEFXb4bKnktN55ZudfHswZ1F62ZL+vNatFk2rlr7JM0VE5FbgNvvsuJL22SkEhgGLhsOWj8DTF+7/Esq3cKik1WowZ+MRJn33OxczLuPpYeGhFvEMa1eZEj5uO0MrIiImKfL77Egxs2p8TtCxeEDPfzkcdPYkXGT0l7+x9UgiAHXLhjCuR21qxugMKRERyUthR5xv4wewemLO7c6TocbddpdKz8pm2op9vL/mIJetBgE+nozqUJX7m5bH08NiUsMiIlKcKOyIc+1cCN+Oyrnd6nloONDuUmv3neWFr7bzx7k0ANrXiOSVrjWJDvE3o1MRESmmFHbEeQ6uhi8fAQxo9BC0fMauMudSMnht8W4W/nIcgKhgP16+uyYda11/LyUREZFcCjviHCe2wWf9IDsTanSFTpPAUrBpJsMwmL/lGOO+3U1iWhYWCzzQJI6RHaoS5KdLNYiISP4o7Ij5zh2AOX+HzIsQfwf0+AA8PAtU4uCZFJ5fuJ2fD54HoFpUEON71KZerLb+ExGRglHYEXNdPAX/6QGpZ3L20OkzB7zyf2mOzMtW3l19gBkr95N52YqftwfD21VhUIt4vD1ddxE5EREpuhR2xDzpSTCnJ1w4DCXLQ78vwC//+xZtOnye0V9uZ//pFADuqBLBa11rEVuqhHP6FRGRW4LCjpgjKz1njU7CdgiIgPsXQlBkvp6alJbFhCW7+XTjUQDCA30Yc1cN7q4bg6WA63xERET+SmFHHGfNhi8fhsM/gk8Q3PcFhFW46dMMw+Cb304y9ptdnE3JAKBvo3I816kaoSV8nN21iIjcIhR2xDGGAd+OhN1fg6cP9J0D0XVv+rTTyek888VvrNpzBoCKEQGM616bxhVKObtjERG5xSjsiGNWT4TNHwKWnLOuKrS86VMMw2D459v46cA5fDw9GNK6Eo+1qoCvV8HO2BIREckPhR2x36ZZOde8Auj8BtTslq+nrd1/1hZ0vn6iOdWidPFVERFxHp3LK/bZ9V9Y/HTO7ZbP5uyQnA9Wq8GkJXsAuK9JnIKOiIg4ncKOFNyhH+GLhwADGjwIrUbn+6nf7Uhg+/EkAnw8GdK6ovN6FBER+ZPCjhTMyd/gs3tzLgNRvUvOVczzeXr45Wwrk7/PGdV56PYKlArM/2aDIiIi9lLYkfw7fwj+0xMykiGuBfT4V4EuA7FgyzEOnk0lLMCHh26Pd2KjIiIi/6OwI/mTchr+3R1ST0NkbbhnLnj75fvp6VnZTF2xD4DBrSrqQp4iIlJoFHbk5tKTc0Z0LhyC0Di4bwH4hRSoxL/X/8HJpHRiQvy4r0mckxoVERG5msKO3NjlDPi8HyT8BiXC/7wMRFSBSiSnZzFz1X4Ahrergp+39tMREZHCo7Aj12fNhi8fgUNrwCcwZ0SnVMHPoPrXmoMkpmVRMSKAHvXLOKFRERGR61PYkWszDPjuWdj1FXh4Q5//QEy9Apc5czGDf609BMCoDlXx8tS/ciIiUrj0ySPXtnkWbPqAnMtAvAcVW9tVZubK/aRlZlO3bAgdahZs+ktERMQMCjtytTN7Yek/cm7/7RWo1dOuMkfPpzFnwx8APNOxGpZ87scjIiJiJoUdyetyJnz5EFy+BBXbQNMn7C715vK9ZGUbtKgUTvNK4SY2KSIikn8KO5LXytfh5K/gHwZd3wYP+/4V2ZNwkYW/HAdy1uqIiIi4isKO/M+hH2Hd1Jzbd0+D4Gi7S73x/R4MAzrViqJuuVBz+hMREbGDwo7kuHQBFj4GGFDv/pzrXtlp65ELLNt1Cg8LPN2+ink9ioiI2EFhR3IsHgnJxyCsAnScYHcZwzCY+N3vAPy9QVkqlQ4yq0MRERG7KOwI/DYPdiwAiyf0+AB8A+0utWbfWTYcOo+PlwfD2mlUR0REXE9h51Z34Q9Y/HTO7VbPQdmGdpeyWg3+uTRnVOf+JnGUCfU3o0MRERGHKOzcyqzZOet0MpKhXGNo8ZRD5b7dcZIdx5MJ9PVicKuCX1ZCRETEGRR2bmVr34QjP4FPEHR/Dzy97C6VlW1l8vd7AXjo9nhKBfqa1aWIiIhDFHZuVce3wqrxObfvnARh8Q6VW7DlGIfOphIW4MNDt1cwoUERERFzKOzcijJT4cuHwXoZanSDuvc4VC49K5u3lueM6gxtXYlAX/tHiERERMymsHMrWvoCnNsPQTFw15vg4DWrPll/mFPJGZQJ9adfk1iTmhQRETGHws6t5vdvYcvsnNvd34USYQ6VS07P4u1VBwAY3q4yvl6ejnYoIiJiKoWdW8nFU/D10JzbTYdChZYOl/xgzUES07KoVDqQHvXLOlxPRETEbAo7twrDgP8OhrRzEFkb2r7ocMkzFzP414+HABjZviqeHo5Nh4mIiDiDws6tYtO/YP9y8PKDnh+Al+Onhs/4YR+XsrKpWy6UDjUjTWhSRETEfAo7t4LTv8P3/8i5/bexULq6wyWPnk9j7sYjADzboSoWBxc5i4iIOIvCTnF3OQO+fAgup0OldvB/j5hS9s1le8nKNri9cjjNKoWbUlNERMQZFHaKux9eg4TtUKIUdH3b4dPMAfYkXGThtuMAjOpQ1eF6IiIizqSwU5wdWgM/Tc+5ffd0CDJnXc0/l+7BMODO2lHUKRtqSk0RERFnUdgpri5dyLnIJwbU7w/VOptSdssf51m++xSeHhaebq9RHRERcX8KO8WRYcCiEZB8HMIqQsfxJpU1mLhkDwB/r1+WihGBptQVERFxJoWd4ui3z2HnQvDwyjnN3CfAlLKr955h46Hz+Hh5MKxdZVNqioiIOJvCTnFz4TAsHplzu9VzUKaBKWWtVoN/Ls0Z1enfNI6YUH9T6oqIiDibwk5xkn0ZvnwUMi9CuSbQ4inTSi/efpKdJ5IJ9PXi8VaVTKsrIiLibAo7xcnaN+Hoz+ATBD3eAw9zLsqZlW1l8vc5ozqP3FGBsAAfU+qKiIgUBoWd4uLYFlj150Lkzm9AyfKmlZ6/+RiHz6VRKsCHgS3iTasrIiJSGBR2ioOMlJxdko1sqNkD6vQxrXR6VjZTV+wFYGibSgT6eplWW0REpDAo7BQHS5+H8wchuCzcNcWUXZJzffzTYU4lZ1Am1J97G8eaVldERKSwKOwUdbsXwdaPAQt0fwf8S5pWOulSFm+vOgDAiL9VwdfLnDVAIiIihUlhpyi7mABfP5Fzu9kTEH+HqeXfX3OApEtZVC4dSPd6ZUytLSIiUlgUdooqw4D/DoFL5yGqNrT5h6nlT19M58O1h4Gci316epg3NSYiIlKYFHaKqjN7YP9y8PSBHv8CL19Tyy/cepxLWdnULRfK32qYcwFRERERV1DYKaqObsj5Z7nGULqa6eVX7jkNQI96ZbCYuOBZRESksCnsFFVXhh2TJadnsfnwBQBaVy1ten0REZHCpLBTVDkx7Kzbd5bLVoMKEQHEliphen0REZHCpLBTFKWeg3P7c26XbWh6+dwpLI3qiIhIcaCwUxQd25jzz/CqUCLM1NKGYbByzxlAYUdERIoHhZ2iKHcKK9b8KaydJ5I5czGDEj6eNIo3b4NCERERV3HrsJOdnc2YMWOIj4/H39+fihUr8uqrr2IYhu0xhmHw4osvEh0djb+/P+3atWPfvn0u7LoQHP1zZMcJ63VW/TmF1axiuHZMFhGRYsGtw87EiRN55513mDFjBrt372bixIlMmjSJ6dOn2x4zadIkpk2bxrvvvsuGDRsICAigQ4cOpKenu7BzJ8rOguNbcm47IezYprCqRZheW0RExBXc+hLWP/30E127dqVz584AlC9fnk8//ZSNG3NGNgzD4K233uIf//gHXbt2BeCTTz4hMjKSr776ir59+7qsd6dJ+A0up+dcA6tUJVNLX0jN5JcjOaect9J6HRERKSbcemSnWbNmrFixgr179wLw66+/snbtWjp16gTAoUOHSEhIoF27drbnhISE0LhxY9avX++Snp3uyikskzf7W7PvDFYDqkYGUSbU39TaIiIiruLWIzvPPfccycnJVKtWDU9PT7Kzs3n99dfp168fAAkJCQBERua9nEFkZKTtvmvJyMggIyPD9n1ycrITuneSIz/n/LPc/5leetWfU1itNIUlIiLFiFuP7MybN485c+Ywd+5ctm7dyscff8wbb7zBxx9/7FDd8ePHExISYvsqV66cSR07mWE4bTPBbKvB6r065VxERIoftw47o0aN4rnnnqNv377Url2b+++/nxEjRjB+/HgAoqKiADh16lSe5506dcp237WMHj2apKQk29fRo0ed90OYKekYXDwJFk+IqW9q6d+OJXI+NZMgXy8axOmUcxERKT7cOuykpaXh4ZG3RU9PT6xWKwDx8fFERUWxYsUK2/3Jycls2LCBpk2bXreur68vwcHBeb6KhNxRneg64GPuZRxyz8K6vUo43p5u/a+FiIhIgbj1mp0uXbrw+uuvExsbS82aNfnll1+YMmUKAwcOBMBisTB8+HBee+01KleuTHx8PGPGjCEmJoZu3bq5tnlnsC1ObmJ66dV/7q+js7BERKS4ceuwM336dMaMGcPgwYM5ffo0MTExPProo7z44ou2xzzzzDOkpqbyyCOPkJiYSIsWLViyZAl+fn4u7NxJbOt1zF2cfOZiBr8eSwKgVRUtThYRkeLFYly5HfEtKjk5mZCQEJKSktx3SiszFcaXAyMbRuyCkDKmlf5iyzGenv8rtcoEs+iJ202rKyIi4kz5/fzW4oyi4vjWnKATXNbUoAO6yrmIiBRvCjtFxVHn7K9zOdvKmj9POdd6HRERKY4UdooKJ13885ejiSSnXya0hDe3lQs1tbaIiIg7UNgpCqzWK8KOuSM7K3/PmcJqWSUCTw9zLz8hIiLiDhR2ioJz+yA9Ebz8Iaq2qaVtVznXFJaIiBRTCjtFQe4p52UagKe3aWUTktLZfTIZiwXu0CnnIiJSTCnsFAW5YSfW3PU6q/48C+u2cqGEBfiYWltERMRdKOwUBU5anKxTzkVE5FagsOPu0s7D2b05t8s2Mq1s5mUra/edBRR2RESkeFPYcXfHNuX8M7wKlAgzrezmw+dJzcwmPNCXmjFuumu0iIiICRR23N0R52wmuNJ24c8IPHTKuYiIFGMKO+7Oaet1dMq5iIjcGhR23Fl2FhzfknPbxLBz9Hwa+0+n4OlhoUXlcNPqioiIuCOFHXeWsB0uXwK/UChV2bSyuaecN4grSYi/efv2iIiIuCOFHXd25SUiPMz7U2kKS0REbiUKO+4sdzNBE6ew0rOy+enAn6ecV9OuySIiUvwp7LgzJyxO/vngOdKzrESH+FE1Msi0uiIiIu5KYcddJR2D5GNg8YQy9U0ru+rPKaxWVUtjseiUcxERKf4UdtxV7qhOVG3wCTClpGEY/PB77iUiNIUlIiK3BoUdd+WE9TqHzqZy5Hwa3p4WmlfSKeciInJrUNhxV7awY97OyblnYTWOL0WAr5dpdUVERNyZwo47ykyFk7/l3DZxZGfVFZeIEBERuVUo7LijE7+AkQ1BMRBS1pSSqRmX2XDwPJCzOFlERORWobDjjq6cwjLpjKmfDpwjM9tKuTB/KkaYs+BZRESkKFDYcUe5Z2LFNjGtZO5VzlvrlHMREbnFKOy4G8MwfXGyYRis+v1/YUdERORWorDjbs7th0sXwMsfouqYUnLf6RROJKXj6+VBkwqlTKkpIiJSVCjsuJsjP+f8s0x98DTniuQr/xzVaVqxFP4+nqbUFBERKSoUdtyNU/bX0RSWiIjcuhR23I3JF/9MTs9i8+ELgMKOiIjcmhR23EnaeTi7J+d2WXNGdtbtO8tlq0GFiABiS5UwpaaIiEhRorDjTo5tzvlnqUoQYM5CYk1hiYjIrU5hx52YfPFPwzBs18NS2BERkVuVwo47MTns7DyRzJmLGZTw8aRRfElTaoqIiBQ1CjvuIvsyHN+Sc9uksJN74c/mlcLx9dIp5yIicmtS2HEXp3ZAVhr4hUB4FVNKagpLREREYcd95E5hlf0/8HD8z3IhNZNfjuScct6qaoTD9URERIoqhR13YfJ6nTX7zmA1oFpUEDGh/qbUFBERKYoUdtyFbTNBc/bXWfXnFFYrTWGJiMgtTmHHHSQdh6SjYPGAMg0cLpdtNVi9N3e9jqawRETk1qaw4w6O/TmqE1kLfAMdLvfbsUTOp2YS5OdF/Tidci4iIrc2hR13YPL1sHLPwrqjcgTenvoTi4jIrU2fhO4gd3FybBNTyuXur6OzsERERBR2XC/rEpz8Nee2CYuTz1zM4LdjSQC0VNgRERFR2HG541vBehmCoiGknMPlchcm1y4TQukgP4friYiIFHUKO65m21/n/8Bicbjc/65yrlEdERERUNhxPRMXJ1/OtrLmz5GdVtW0v46IiAgo7LiWYZi6c/LWI4lcTL9MyRLe1C0b6nA9ERGR4kBhx5XOHYBL58HTF6LqOFwudwqrZZUIPD0cnxITEREpDhR2XCl3VKdMffDycbjcyt//XK+jKSwREREbhR1XunJxsoNOJl3i94SLWCw5mwmKiIhIDoUdV7ItTnZ8M8HcC3/WKxdKyQDHR4lERESKC4UdV7mUCGd259w2YWTHNoWlq5yLiIjkobDjKsc25fwzrCIEhDtUKuNyNuv2nwWglcKOiIhIHgo7rmLiKeebD18gNTOb8EBfasYEO1xPRESkOFHYcRUTFydfeeFPD51yLiIikofCjitkX4ZjW3JumzCys/LPxclaryMiInI1hR1XOL0TslLBNxgiqjlU6uj5NPafTsHTw0KLyo6t/RERESmO7Ao7K1euNLuPW0vuKedlG4GHY3kzdwqrQVxJQvy9He1MRESk2LHrk7Zjx45UrFiR1157jaNHj5rdU/Fn4uJkTWGJiIjcmF1h5/jx4wwdOpQFCxZQoUIFOnTowLx588jMzDS7v+IpN+zEOhZ20rOy+elAzinnratp12QREZFrsSvshIeHM2LECLZt28aGDRuoUqUKgwcPJiYmhieffJJff/3V7D6Lj+STkHgELB5QpoFDpX4+eI70LCvRIX5UjQwyqUEREZHixeEFyvXr12f06NEMHTqUlJQUPvzwQxo0aMDtt9/Ozp07zeixeMkd1YmsCb6OBZTcS0S0qloai0WnnIuIiFyL3WEnKyuLBQsWcOeddxIXF8fSpUuZMWMGp06dYv/+/cTFxdGrVy8zey0ebNfDcmwKyzAMfrBdIkJTWCIiItdjV9h54okniI6O5tFHH6VKlSr88ssvrF+/noceeoiAgADKly/PG2+8we+//+5wg8ePH+e+++6jVKlS+Pv7U7t2bTZv3my73zAMXnzxRaKjo/H396ddu3bs27fP4dd1GpMWJyckp3PkfBpeHhaaV9Ip5yIiItfjZc+Tdu3axfTp0+nRowe+vr7XfEx4eLjDp6hfuHCB5s2b07p1a7777jsiIiLYt28fJUuWtD1m0qRJTJs2jY8//pj4+HjGjBlDhw4d2LVrF35+fg69vumyLsHJP9czObhz8pFzaQCULelPgK9df0YREZFbgl2fkitWrLh5YS8vWrZsaU95m4kTJ1KuXDlmz55tOxYfH2+7bRgGb731Fv/4xz/o2rUrAJ988gmRkZF89dVX9O3b16HXN92JbWDNgsBICI1zqNSxC5cAKFuyhAmNiYiIFF92TWONHz+eDz/88KrjH374IRMnTnS4qVxff/01DRs2pFevXpQuXZp69erxwQcf2O4/dOgQCQkJtGvXznYsJCSExo0bs379+uvWzcjIIDk5Oc9XobjyelgOLij+X9jxd7QrERGRYs2usPPee+9RrdrVlzmoWbMm7777rsNN5Tp48CDvvPMOlStXZunSpTz++OM8+eSTfPzxxwAkJCQAEBkZmed5kZGRtvuuZfz48YSEhNi+ypUrZ1rPN2TS4mSAYxf+N40lIiIi12dX2ElISCA6Ovqq4xEREZw8edLhpnJZrVbq16/PuHHjqFevHo888ggPP/yww4Fq9OjRJCUl2b4KZRdow7hiZKeJw+WOJ2oaS0REJD/sCjvlypVj3bp1Vx1ft24dMTExDjeVKzo6mho1auQ5Vr16dY4cOQJAVFQUAKdOncrzmFOnTtnuuxZfX1+Cg4PzfDnd+YOQdhY8fSG6jsPlcqexymhkR0RE5IbsWqD88MMPM3z4cLKysmjTpg2Qs2j5mWee4emnnzatuebNm7Nnz548x/bu3UtcXM7i3vj4eKKiolixYgW33XYbAMnJyWzYsIHHH3/ctD5MkTuqE1MPvK59Blt+ZVsNTiRqzY6IiEh+2BV2Ro0axblz5xg8eLDtelh+fn48++yzjB492rTmRowYQbNmzRg3bhy9e/dm48aNvP/++7z//vsAWCwWhg8fzmuvvUblypVtp57HxMTQrVs30/owxZWLkx10Kjmdy1YDb08LpYPc7PR6ERERN2NX2LFYLEycOJExY8awe/du/P39qVy58nX33LFXo0aNWLhwIaNHj2bs2LHEx8fz1ltv0a9fP9tjnnnmGVJTU3nkkUdITEykRYsWLFmyxP322DF1cXLOqE5MqD+eHrpMhIiIyI1YDMMwXN2EqyUnJxMSEkJSUpJz1u9cSoSJ5QEDRu6DwNIOlfty6zGemvcrzSuVYs5Dji92FhERKYry+/lt99a7mzdvZt68eRw5csQ2lZXryy+/tLds8XR8M2BAyXiHgw5cscdOqM7EEhERuRm7zsb67LPPaNasGbt372bhwoVkZWWxc+dOfvjhB0JCQszusegzcQoLtMeOiIhIQdgVdsaNG8ebb77JN998g4+PD1OnTuX333+nd+/exMbGmt1j0Ze7ODnWrLDz58hOmMKOiIjIzdgVdg4cOEDnzp0B8PHxITU1FYvFwogRI2xnSsmfrNlw7M+rtJs2sqMNBUVERPLLrrBTsmRJLl68CECZMmXYsWMHAImJiaSlpZnXXXFwaidkpoBvMERcfYmNgtIeOyIiIgVj1wLlO+64g2XLllG7dm169erFsGHD+OGHH1i2bBlt27Y1u8eiLXcKq2xD8PB0uNzpi9pjR0REpCDsCjszZswgPT0dgBdeeAFvb29++uknevbsyT/+8Q9TGyzyTF+cnDOqEx2iPXZERETyo8Bh5/LlyyxatIgOHToA4OHhwXPPPWd6Y8WGiTsng87EEhERKagCr9nx8vLiscces43syA1cTIDEPwALlGloSslj57VeR0REpCDsWqD8f//3f2zbts3kVoqh3CmsyJrgZ87OzDoTS0REpGDsWrMzePBgnnrqKY4ePUqDBg0ICAjIc3+dOnVMaa7IM3kKC+BYoqaxRERECsKusNO3b18AnnzySdsxi8WCYRhYLBays7PN6a6osy1ONu/6VRrZERERKRi7ws6hQ4fM7qP4sVohOyPntkkjO9pjR0REpODsCjtxcXFm91H8eHjAo2vg0gXwCzWl5OmL6WRlG3h5WIgM1h47IiIi+WFX2Pnkk09ueP8DDzxgVzPFkn9J00rlTmHFhGqPHRERkfyyK+wMGzYsz/dZWVmkpaXh4+NDiRIlFHacRHvsiIiIFJxdp55fuHAhz1dKSgp79uyhRYsWfPrpp2b3KH/SHjsiIiIFZ1fYuZbKlSszYcKEq0Z9xDzH/1ycXCZUZ2KJiIjkl2lhB3J2Vz5x4oSZJeUK/zvtXCM7IiIi+WXXmp2vv/46z/eGYXDy5ElmzJhB8+bNTWlMrqY1OyIiIgVnV9jp1q1bnu8tFgsRERG0adOGyZMnm9GX/IXVatimscqGaRpLREQkv+wKO1ar1ew+5CZOX8z43x47Qb6ubkdERKTIMHXNjjhP7hRWdKgfXp76s4mIiOSXXZ+aPXv2ZOLEiVcdnzRpEr169XK4KbmabXGyzsQSEREpELvCzpo1a7jzzjuvOt6pUyfWrFnjcFNyNS1OFhERsY9dYSclJQUfH5+rjnt7e5OcnOxwU3I1Xe1cRETEPnaFndq1a/P5559fdfyzzz6jRo0aDjclV9MeOyIiIvax62ysMWPG0KNHDw4cOECbNm0AWLFiBZ9++inz5883tUHJoWksERER+9gVdrp06cJXX33FuHHjWLBgAf7+/tSpU4fly5fTsmVLs3u85VmtBicS0wEoo7AjIiJSIHaFHYDOnTvTuXNnM3uR6ziTkkFmthVPDwtRwX6ubkdERKRIsWvNzqZNm9iwYcNVxzds2MDmzZsdbkrysu2xE6I9dkRERArKrk/OIUOGcPTo0auOHz9+nCFDhjjclOSlxckiIiL2syvs7Nq1i/r16191vF69euzatcvhpiQvnXYuIiJiP7vCjq+vL6dOnbrq+MmTJ/HysnsZkFyHzsQSERGxn11hp3379owePZqkpCTbscTERJ5//nn+9re/mdac5NDIjoiIiP3sGoZ54403uOOOO4iLi6NevXoAbNu2jcjISP7973+b2qBozY6IiIgj7Ao7ZcqU4bfffmPOnDn8+uuv+Pv78+CDD3LPPffg7e1tdo+3NKvV4LjCjoiIiN3sXmATEBBAixYtiI2NJTMzE4DvvvsOgLvvvtuc7kR77IiIiDjIrrBz8OBBunfvzvbt27FYLBiGgcVisd2fnZ1tWoO3Ou2xIyIi4hi7Pj2HDRtGfHw8p0+fpkSJEuzYsYPVq1fTsGFDVq1aZXKLt7bc9TplQjWFJSIiYg+7RnbWr1/PDz/8QHh4OB4eHnh6etKiRQvGjx/Pk08+yS+//GJ2n7csnYklIiLiGLtGdrKzswkKCgIgPDycEydOABAXF8eePXvM6050JpaIiIiD7BrZqVWrFr/++ivx8fE0btyYSZMm4ePjw/vvv0+FChXM7vGWpg0FRUREHGNX2PnHP/5BamoqAGPHjuWuu+7i9ttvp1SpUnz++eemNnirO65pLBEREYfYFXY6dOhgu12pUiV+//13zp8/T8mSJfOclSWOsVoNjiVqGktERMQRpl3IKiwszKxS8qezKRlkXs7ZYyc6RHvsiIiI2EMbt7ixo39OYUUFa48dERERe+kT1I1pcbKIiIjjFHbcmPbYERERcZzCjhvTHjsiIiKOU9hxY7nTWGUUdkREROymsOPGjuu0cxEREYcp7LgpwzBsGwqW05odERERuynsuKkzKRlkXLbiYYEo7bEjIiJiN4UdN5W7ODk6xB9v7bEjIiJiN32KuqncsKPFySIiIo5R2HFT2lBQRETEHAo7bkobCoqIiJhDYcdNaUNBERERcyjsuClNY4mIiJhDYccNaY8dERER8yjsuCHtsSMiImIehR03lDuqExXspz12REREHKRPUjekM7FERETMU6TCzoQJE7BYLAwfPtx2LD09nSFDhlCqVCkCAwPp2bMnp06dcl2TJtCZWCIiIuYpMmFn06ZNvPfee9SpUyfP8REjRvDNN98wf/58Vq9ezYkTJ+jRo4eLujSHzsQSERExT5EIOykpKfTr148PPviAkiVL2o4nJSUxa9YspkyZQps2bWjQoAGzZ8/mp59+4ueff3Zhx47RNJaIiIh5ikTYGTJkCJ07d6Zdu3Z5jm/ZsoWsrKw8x6tVq0ZsbCzr16+/br2MjAySk5PzfLkTjeyIiIiYx8vVDdzMZ599xtatW9m0adNV9yUkJODj40NoaGie45GRkSQkJFy35vjx43nllVfMbtUUhmFoZEdERMREbj2yc/ToUYYNG8acOXPw8zNvv5nRo0eTlJRk+zp69KhptR11NiVTe+yIiIiYyK3DzpYtWzh9+jT169fHy8sLLy8vVq9ezbRp0/Dy8iIyMpLMzEwSExPzPO/UqVNERUVdt66vry/BwcF5vtxF7hRWVLAfPl5u/ecREREpEtx6Gqtt27Zs3749z7EHH3yQatWq8eyzz1KuXDm8vb1ZsWIFPXv2BGDPnj0cOXKEpk2buqJlh2kKS0RExFxuHXaCgoKoVatWnmMBAQGUKlXKdnzQoEE89dRThIWFERwczBNPPEHTpk1p0qSJK1p2WG7YKaPFySIiIqZw67CTH2+++SYeHh707NmTjIwMOnTowNtvv+3qtuymM7FERETMVeTCzqpVq/J87+fnx8yZM5k5c6ZrGjLZ8UTtniwiImImrYB1M1qzIyIiYi6FHTeSs8eOprFERETMpLDjRs6lZpKeZcVigegQhR0REREzKOy4kdwpLO2xIyIiYh59oroRTWGJiIiYT2HHjWhxsoiIiPkUdtyIRnZERETMp7DjRv43sqOwIyIiYhaFHTeiaSwRERHzKey4iSv32CkTqpEdERERsyjsuInzV+6xE+rn6nZERESKDYUdN5E7hRUZ5Ievl6eLuxERESk+FHbchBYni4iIOIfCjpvQaeciIiLOobDjJnQmloiIiHMo7LgJjeyIiIg4h8KOm9DIjoiIiHMo7LiBnD12tEBZRETEGRR23MD51EwuZWVrjx0REREnUNhxA9pjR0RExHkUdtxAbtgpoyksERER0ynsuIHjiToTS0RExFkUdtyAFieLiIg4j8KOG9Bp5yIiIs6jsOMGtKGgiIiI8yjsuFjePXY0siMiImI2hR0Xu5CWRVpmNgAx2mNHRETEdAo7LpY7hRUZ7Ks9dkRERJxAYcfFNIUlIiLiXAo7LqbFySIiIs6lsONi2mNHRETEuRR2XMx2qYhQTWOJiIg4g8KOi2kaS0RExLkUdlzIMAyOaxpLRETEqRR2XCgxLYtU2x47CjsiIiLOoLDjQrnrdUoH+eLnrT12REREnEFhx4W0XkdERMT5FHZcSBsKioiIOJ/CjgtpZEdERMT5FHZcSCM7IiIizqew40LaPVlERMT5FHZcxDAMTWOJiIgUAoUdF9EeOyIiIoVDYcdFcqewIrTHjoiIiFMp7LjI8URNYYmIiBQGhR0X0ZlYIiIihUNhx0V0JpaIiEjhUNhxEZ2JJSIiUjgUdlxE01giIiKFQ2HHBXL22NE0loiISGFQ2HGBpEtZpGRcBqCM9tgRERFxKoUdF9AeOyIiIoVHYccFtDhZRESk8CjsuEDuyI6msERERJxPYccFdCaWiIhI4VHYcQGdiSUiIlJ4FHZcQGt2RERECo/CTiEzDIPjmsYSEREpNAo7hSz50mUu/rnHjkZ2REREnE9hp5Ad/XMKKzxQe+yIiIgUBoWdQqbFySIiIoVLYaeQaXGyiIhI4VLYKWTaY0dERKRwuXXYGT9+PI0aNSIoKIjSpUvTrVs39uzZk+cx6enpDBkyhFKlShEYGEjPnj05deqUizq+OU1jiYiIFC63DjurV69myJAh/PzzzyxbtoysrCzat29Pamqq7TEjRozgm2++Yf78+axevZoTJ07Qo0cPF3Z9Y7nTWGUUdkRERAqFl6sbuJElS5bk+f6jjz6idOnSbNmyhTvuuIOkpCRmzZrF3LlzadOmDQCzZ8+mevXq/PzzzzRp0sQVbV/XlXvslFPYERERKRRuPbLzV0lJSQCEhYUBsGXLFrKysmjXrp3tMdWqVSM2Npb169e7pMcbuXKPnTKhWrMjIiJSGNx6ZOdKVquV4cOH07x5c2rVqgVAQkICPj4+hIaG5nlsZGQkCQkJ162VkZFBRkaG7fvk5GSn9PxXxxJz99jxwd9He+yIiBSW7OxssrKyXN2GFJC3tzeeno5/XhaZsDNkyBB27NjB2rVrHa41fvx4XnnlFRO6KpjcxclldCaWiEihMAyDhIQEEhMTXd2K2Ck0NJSoqCgsFovdNYpE2Bk6dCiLFi1izZo1lC1b1nY8KiqKzMxMEhMT84zunDp1iqioqOvWGz16NE899ZTt++TkZMqVK+eU3q+kM7FERApXbtApXbo0JUqUcOgDUwqXYRikpaVx+vRpAKKjo+2u5dZhxzAMnnjiCRYuXMiqVauIj4/Pc3+DBg3w9vZmxYoV9OzZE4A9e/Zw5MgRmjZtet26vr6++Pr6OrX3a9GGgiIihSc7O9sWdEqVKuXqdsQO/v45n5enT5+mdOnSdk9puXXYGTJkCHPnzuW///0vQUFBtnU4ISEh+Pv7ExISwqBBg3jqqacICwsjODiYJ554gqZNm7rdmVigDQVFRApT7hqdEiX039yiLPfvl5WVVTzDzjvvvANAq1at8hyfPXs2AwYMAODNN9/Ew8ODnj17kpGRQYcOHXj77bcLudP80TSWiEjh09RV0WbG38+tw45hGDd9jJ+fHzNnzmTmzJmF0JFjcqextMeOiIhI4SlS++wUZUmXsriYrj12RESkcJUvX5633nrL1W24lFuP7BQnuaM6pQK0x46IiNxYq1atuO2220wJKZs2bSIgIMDxpoowhZ1CovU6IiJiFsMwyM7Oxsvr5h/jERERhdCRe9M0ViHRmVgiIpIfAwYMYPXq1UydOhWLxYLFYuGjjz7CYrHw3Xff0aBBA3x9fVm7di0HDhyga9euREZGEhgYSKNGjVi+fHmeen+dxrJYLPzrX/+ie/fulChRgsqVK/P111/nq7fs7GwGDRpEfHw8/v7+VK1alalTp171uA8//JCaNWvi6+tLdHQ0Q4cOtd2XmJjIo48+SmRkJH5+ftSqVYtFixbZ98vKJ43sFJLjGtkREXE5wzC4lJXtktf29/bM15lFU6dOZe/evdSqVYuxY8cCsHPnTgCee+453njjDSpUqEDJkiU5evQod955J6+//jq+vr588skndOnShT179hAbG3vd13jllVeYNGkS//znP5k+fTr9+vXjjz/+sF178nqsVitly5Zl/vz5lCpVip9++olHHnmE6OhoevfuDeScSf3UU08xYcIEOnXqRFJSEuvWrbM9v1OnTly8eJH//Oc/VKxYkV27dplySYgbUdgpJNpQUETE9S5lZVPjxaUuee1dYztQwufmH7shISH4+PhQokQJ29UAfv/9dwDGjh3L3/72N9tjw8LCqFu3ru37V199lYULF/L111/nGU35qwEDBnDPPfcAMG7cOKZNm8bGjRvp2LHjDXvz9vbOc7ml+Ph41q9fz7x582xh57XXXuPpp59m2LBhtsc1atQIgOXLl7Nx40Z2795NlSpVAKhQocJNfyeOUtgpJJrGEhERRzVs2DDP9ykpKbz88sssXryYkydPcvnyZS5dusSRI0duWKdOnTq22wEBAQQHB9suy3AzM2fO5MMPP+TIkSNcunSJzMxMbrvtNiBnp+MTJ07Qtm3baz5327ZtlC1b1hZ0CovCTiHRyI6IiOv5e3uya2wHl722o/56VtXIkSNZtmwZb7zxBpUqVcLf35+///3vZGZm3rCOt7d3nu8tFgtWq/Wmr//ZZ58xcuRIJk+eTNOmTQkKCuKf//wnGzZsAP53eYfrudn9zqKwUwiSLmWRnLvHjsKOiIjLWCyWfE0luZqPjw/Z2TdfW7Ru3ToGDBhA9+7dgZyRnsOHDzutr3Xr1tGsWTMGDx5sO3bgwAHb7aCgIMqXL8+KFSto3br1Vc+vU6cOx44dY+/evYU6uqOzsQpB7uLkUgE+ReJNJiIirlW+fHk2bNjA4cOHOXv27HVHXSpXrsyXX37Jtm3b+PXXX7n33nvzNUJjr8qVK7N582aWLl3K3r17GTNmDJs2bcrzmJdffpnJkyczbdo09u3bx9atW5k+fToALVu25I477qBnz54sW7aMQ4cO8d1337FkyRKn9QwKO4VCU1giIlIQI0eOxNPTkxo1ahAREXHdNThTpkyhZMmSNGvWjC5dutChQwfq16/vtL4effRRevToQZ8+fWjcuDHnzp3LM8oD0L9/f9566y3efvttatasyV133cW+ffts93/xxRc0atSIe+65hxo1avDMM8/kaxTLERYjPxegKuaSk5MJCQkhKSmJ4OBg0+t/uPYQYxftonPtaGb2c96/hCIi8j/p6ekcOnSI+Ph4/Pz8XN2O2OlGf8f8fn5rZKcQ5J6JpfU6IiIihU9hpxBoGktERIqCxx57jMDAwGt+PfbYY65uz25aLVsIdF0sEREpCsaOHcvIkSOveZ8zlnkUFoWdQnA8URsKioiI+ytdujSlS5d2dRum0zSWkyWnZ5F0KQuAMqEa2RERESlsCjtOlrvHTliADwG+GkgTEREpbAo7Tqb1OiIiIq6lsONkOhNLRETEtRR2nExXOxcREXEthR0n08iOiIgUtvLly/PWW2+5ug23obDjZFqzIyIi4loKO05mu1REqKaxREREXEFhx4ny7LGjkR0REcmH999/n5iYGKxWa57jXbt2ZeDAgRw4cICuXbsSGRlJYGAgjRo1Yvny5Xa/3pQpU6hduzYBAQGUK1eOwYMHk5KSkucx69ato1WrVpQoUYKSJUvSoUMHLly4AIDVamXSpElUqlQJX19fYmNjef311+3uxxkUdpwod4+dkiW8CdQeOyIirmcYkJnqmi/DyFeLvXr14ty5c6xcudJ27Pz58yxZsoR+/fqRkpLCnXfeyYoVK/jll1/o2LEjXbp04ciRI3b9Sjw8PJg2bRo7d+7k448/5ocffuCZZ56x3b9t2zbatm1LjRo1WL9+PWvXrqVLly5kZ2cDMHr0aCZMmMCYMWPYtWsXc+fOJTIy0q5enEWfwE6kM7FERNxMVhqMi3HNaz9/AnwCbvqwkiVL0qlTJ+bOnUvbtm0BWLBgAeHh4bRu3RoPDw/q1q1re/yrr77KwoUL+frrrxk6dGiB2xo+fLjtdvny5Xnttdd47LHHePvttwGYNGkSDRs2tH0PULNmTQAuXrzI1KlTmTFjBv379wegYsWKtGjRosB9OJNGdpzouM7EEhERO/Tr148vvviCjIwMAObMmUPfvn3x8PAgJSWFkSNHUr16dUJDQwkMDGT37t12j+wsX76ctm3bUqZMGYKCgrj//vs5d+4caWk5n2G5IzvXsnv3bjIyMq57v7vQyI4T6UwsERE3410iZ4TFVa+dT126dMEwDBYvXkyjRo348ccfefPNNwEYOXIky5Yt44033qBSpUr4+/vz97//nczMzAK3dPjwYe666y4ef/xxXn/9dcLCwli7di2DBg0iMzOTEiVK4O9//c+wG93nThR2nEjTWCIibsZiyddUkqv5+fnRo0cP5syZw/79+6latSr169cHchYLDxgwgO7duwOQkpLC4cOH7XqdLVu2YLVamTx5Mh4eOZM98+bNy/OYOnXqsGLFCl555ZWrnl+5cmX8/f1ZsWIFDz30kF09FAaFHSc6lqhpLBERsU+/fv2466672LlzJ/fdd5/teOXKlfnyyy/p0qULFouFMWPGXHXmVn5VqlSJrKwspk+fTpcuXVi3bh3vvvtunseMHj2a2rVrM3jwYB577DF8fHxYuXIlvXr1Ijw8nGeffZZnnnkGHx8fmjdvzpkzZ9i5cyeDBg1y6Oc3k9bsONHlbAOLRSM7IiJScG3atCEsLIw9e/Zw77332o5PmTKFkiVL0qxZM7p06UKHDh1soz4FVbduXaZMmcLEiROpVasWc+bMYfz48XkeU6VKFb7//nt+/fVX/u///o+mTZvy3//+Fy+vnPGSMWPG8PTTT/Piiy9SvXp1+vTpw+nTp+3/wZ3AYhj5PBeuGEtOTiYkJISkpCSCg4NNrZ152YqnhwVPD4updUVE5MbS09M5dOgQ8fHx+Pn5ubodsdON/o75/fzWNJaT+Xhp8ExERMSV9EksIiJSTM2ZM4fAwMBrfuXulXMr0MiOiIhIMXX33XfTuHHja97n7e1dyN24jsKOiIhIMRUUFERQUJCr23A5TWOJiIhIsaawIyIixZpOOi7azPj7KeyIiEixlLsmJfcaT1I05f79HFljpDU7IiJSLHl6ehIaGmrb4K5EiRJYLNrzrKgwDIO0tDROnz5NaGgonp6edtdS2BERkWIrKioKwO129JX8Cw0Ntf0d7aWwIyIixZbFYiE6OprSpUuTlZXl6nakgLy9vR0a0cmlsCMiIsWep6enKR+aUjRpgbKIiIgUawo7IiIiUqwp7IiIiEixpjU7/G/DouTkZBd3IiIiIvmV+7l9s40HFXaAixcvAlCuXDkXdyIiIiIFdfHiRUJCQq57v8XQPtpYrVZOnDhBUFCQqRtOJScnU65cOY4ePUpwcLBpdYtD/aLce1GvX5R7L+r1i3Lvzq5flHsv6vWLcu+GYXDx4kViYmLw8Lj+yhyN7AAeHh6ULVvWafWDg4Od8i9QcahflHsv6vWLcu9FvX5R7t3Z9Yty70W9flHt/UYjOrm0QFlERESKNYUdERERKdYUdpzI19eXl156CV9fX9UvxNqq77raqu+62kW9flHuvajXL8q955cWKIuIiEixppEdERERKdYUdkRERKRYU9gRERGRYk1hR0RERIo1hR0nWLNmDV26dCEmJgaLxcJXX31lav3x48fTqFEjgoKCKF26NN26dWPPnj2m1H7nnXeoU6eObfOnpk2b8t1335lS+1omTJiAxWJh+PDhptR7+eWXsVgseb6qVatmSm2A48ePc99991GqVCn8/f2pXbs2mzdvNqV2+fLlr+rdYrEwZMgQU+pnZ2czZswY4uPj8ff3p2LFirz66qs3vaZMfl28eJHhw4cTFxeHv78/zZo1Y9OmTXbXu9n7yDAMXnzxRaKjo/H396ddu3bs27fPlNpffvkl7du3p1SpUlgsFrZt22Za71lZWTz77LPUrl2bgIAAYmJieOCBBzhx4oQp9SHnfVCtWjUCAgIoWbIk7dq1Y8OGDabUvtJjjz2GxWLhrbfeMq33AQMGXPUe6Nixo2n1AXbv3s3dd99NSEgIAQEBNGrUiCNHjphS/1rvYYvFwj//+U+Ha6ekpDB06FDKli2Lv78/NWrU4N13381X3/mpf+rUKQYMGEBMTAwlSpSgY8eO+X5P5edzKT09nSFDhlCqVCkCAwPp2bMnp06dynf/jlDYcYLU1FTq1q3LzJkznVJ/9erVDBkyhJ9//plly5aRlZVF+/btSU1Ndbh22bJlmTBhAlu2bGHz5s20adOGrl27snPnThM6z2vTpk2899571KlTx9S6NWvW5OTJk7avtWvXmlL3woULNG/eHG9vb7777jt27drF5MmTKVmypCn1N23alKfvZcuWAdCrVy9T6k+cOJF33nmHGTNmsHv3biZOnMikSZOYPn26KfUfeughli1bxr///W+2b99O+/btadeuHcePH7er3s3eR5MmTWLatGm8++67bNiwgYCAADp06EB6errDtVNTU2nRogUTJ040vfe0tDS2bt3KmDFj2Lp1K19++SV79uzh7rvvNqU+QJUqVZgxYwbbt29n7dq1lC9fnvbt23PmzBmHa+dauHAhP//8MzExMfnuO7/1O3bsmOe98Omnn5pW/8CBA7Ro0YJq1aqxatUqfvvtN8aMGYOfn58p9a/s++TJk3z44YdYLBZ69uzpcO2nnnqKJUuW8J///Ifdu3czfPhwhg4dytdff+1w74Zh0K1bNw4ePMh///tffvnlF+Li4mjXrl2+Plvy87k0YsQIvvnmG+bPn8/q1as5ceIEPXr0yFfvDjPEqQBj4cKFTn2N06dPG4CxevVqp9QvWbKk8a9//cvUmhcvXjQqV65sLFu2zGjZsqUxbNgwU+q+9NJLRt26dU2p9VfPPvus0aJFC6fUvpZhw4YZFStWNKxWqyn1OnfubAwcODDPsR49ehj9+vVzuHZaWprh6elpLFq0KM/x+vXrGy+88ILD9f/6PrJarUZUVJTxz3/+03YsMTHR8PX1NT799FOHal/p0KFDBmD88ssvdnR98/q5Nm7caADGH3/84ZT6SUlJBmAsX77clNrHjh0zypQpY+zYscOIi4sz3nzzzQLVvVH9/v37G127drWrXn7q9+nTx7jvvvucVv+vunbtarRp08aU2jVr1jTGjh2b55i977G/1t+zZ48BGDt27LAdy87ONiIiIowPPvigwPX/+rmUmJhoeHt7G/Pnz7c9Zvfu3QZgrF+/vsD1C0ojO8VAUlISAGFhYabWzc7O5rPPPiM1NZWmTZuaWnvIkCF07tyZdu3amVoXYN++fcTExFChQgX69euX7+Hpm/n6669p2LAhvXr1onTp0tSrV48PPvjAlNp/lZmZyX/+8x8GDhxo2sVpmzVrxooVK9i7dy8Av/76K2vXrqVTp04O1758+TLZ2dlX/d+xv7+/aSNrVzp06BAJCQl5/v0JCQmhcePGrF+/3vTXc7akpCQsFguhoaGm187MzOT9998nJCSEunXrOlzParVy//33M2rUKGrWrGlCh1dbtWoVpUuXpmrVqjz++OOcO3fOlLpWq5XFixdTpUoVOnToQOnSpWncuLHpSw1ynTp1isWLFzNo0CBT6jVr1oyvv/6a48ePYxgGK1euZO/evbRv397h2hkZGQB53sMeHh74+vra9R7+6+fSli1byMrKyvOerVatGrGxsYXynlXYKeKsVivDhw+nefPm1KpVy5Sa27dvJzAwEF9fXx577DEWLlxIjRo1TKkN8Nlnn7F161bGjx9vWs1cjRs35qOPPmLJkiW88847HDp0iNtvv52LFy86XPvgwYO88847VK5cmaVLl/L444/z5JNP8vHHH5vQeV5fffUViYmJDBgwwLSazz33HH379qVatWp4e3tTr149hg8fTr9+/RyuHRQURNOmTXn11Vc5ceIE2dnZ/Oc//2H9+vWcPHnShO7zSkhIACAyMjLP8cjISNt9RUV6ejrPPvss99xzj6kXSVy0aBGBgYH4+fnx5ptvsmzZMsLDwx2uO3HiRLy8vHjyySdN6PJqHTt25JNPPmHFihVMnDiR1atX06lTJ7Kzsx2uffr0aVJSUpgwYQIdO3bk+++/p3v37vTo0YPVq1eb0H1eH3/8MUFBQaZN1UyfPp0aNWpQtmxZfHx86NixIzNnzuSOO+5wuHZu8Bg9ejQXLlwgMzOTiRMncuzYsQK/h6/1uZSQkICPj89Vgb6w3rO66nkRN2TIEHbs2GHq/z1XrVqVbdu2kZSUxIIFC+jfvz+rV682JfAcPXqUYcOGsWzZsnzPkRfElaMUderUoXHjxsTFxTFv3jyH/+/KarXSsGFDxo0bB0C9evXYsWMH7777Lv3793eo9l/NmjWLTp06FXg9xI3MmzePOXPmMHfuXGrWrMm2bdsYPnw4MTExpvT/73//m4EDB1KmTBk8PT2pX78+99xzD1u2bDGh++IpKyuL3r17YxgG77zzjqm1W7duzbZt2zh79iwffPABvXv3ZsOGDZQuXdrumlu2bGHq1Kls3brVtBHHv+rbt6/tdu3atalTpw4VK1Zk1apVtG3b1qHaVqsVgK5duzJixAgAbrvtNn766SfeffddWrZs6VD9v/rwww/p16+faf+tmz59Oj///DNff/01cXFxrFmzhiFDhhATE+PwKLm3tzdffvklgwYNIiwsDE9PT9q1a0enTp0KfBKDMz6XHKWRnSJs6NChLFq0iJUrV1K2bFnT6vr4+FCpUiUaNGjA+PHjqVu3LlOnTjWl9pYtWzh9+jT169fHy8sLLy8vVq9ezbRp0/Dy8jLl/96uFBoaSpUqVdi/f7/DtaKjo68KfNWrVzdtmizXH3/8wfLly3nooYdMrTtq1Cjb6E7t2rW5//77GTFihGkjbBUrVmT16tWkpKRw9OhRNm7cSFZWFhUqVDCl/pWioqIArjqT49SpU7b73F1u0Pnjjz9YtmyZqaM6AAEBAVSqVIkmTZowa9YsvLy8mDVrlkM1f/zxR06fPk1sbKzt/fvHH3/w9NNPU758eXMa/4sKFSoQHh5uyns4PDwcLy+vQnkf//jjj+zZs8e09/GlS5d4/vnnmTJlCl26dKFOnToMHTqUPn368MYbb5jyGg0aNGDbtm0kJiZy8uRJlixZwrlz5wr0Hr7e51JUVBSZmZkkJibmeXxhvWcVdoogwzAYOnQoCxcu5IcffiA+Pt6pr2e1Wm3zuY5q27Yt27dvZ9u2bbavhg0b0q9fP7Zt24anp6cpr5MrJSWFAwcOEB0d7XCt5s2bX3Uq5d69e4mLi3O49pVmz55N6dKl6dy5s6l109LS8PDI+5b39PS0/d+uWQICAoiOjubChQssXbqUrl27mlofID4+nqioKFasWGE7lpyczIYNG0xfX+YMuUFn3759LF++nFKlSjn9Nc14H99///389ttved6/MTExjBo1iqVLl5rUaV7Hjh3j3LlzpryHfXx8aNSoUaG8j2fNmkWDBg1MWScFOf/OZGVlFcp7OCQkhIiICPbt28fmzZvz9R6+2edSgwYN8Pb2zvOe3bNnD0eOHCmU96ymsZwgJSUlz/+FHDp0iG3bthEWFkZsbKzD9YcMGcLcuXP573//S1BQkG2+MyQkBH9/f4dqjx49mk6dOhEbG8vFixeZO3cuq1atMu0/ZEFBQVetLQoICKBUqVKmrDkaOXIkXbp0IS4ujhMnTvDSSy/h6enJPffc43DtESNG0KxZM8aNG0fv3r3ZuHEj77//Pu+//77DtXNZrVZmz55N//798fIy9+3ZpUsXXn/9dWJjY6lZsya//PILU6ZMYeDAgabUX7p0KYZhULVqVfbv38+oUaOoVq0aDz74oF31bvY+Gj58OK+99hqVK1cmPj6eMWPGEBMTQ7du3Ryuff78eY4cOWLb+yb3wzEqKipf/xd6o/rR0dH8/e9/Z+vWrSxatIjs7GzbezgsLAwfHx+H6pcqVYrXX3+du+++m+joaM6ePcvMmTM5fvx4vrYxuNnv5q/BzNvbm6ioKKpWrXrT2jerHxYWxiuvvELPnj2JioriwIEDPPPMM1SqVIkOHTo4XD82NpZRo0bRp08f7rjjDlq3bs2SJUv45ptvWLVqlSn1ISd4z58/n8mTJ+erZn5rt2zZklGjRuHv709cXByrV6/mk08+YcqUKabUnz9/PhEREcTGxrJ9+3aGDRtGt27d8rUA+mafSyEhIQwaNIinnnqKsLAwgoODeeKJJ2jatClNmjQp0O/JLk4/3+sWtHLlSgO46qt///6m1L9WbcCYPXu2w7UHDhxoxMXFGT4+PkZERITRtm1b4/vvv3e86Rsw89TzPn36GNHR0YaPj49RpkwZo0+fPsb+/ftNqW0YhvHNN98YtWrVMnx9fY1q1aoZ77//vmm1DcMwli5dagDGnj17TK1rGIaRnJxsDBs2zIiNjTX8/PyMChUqGC+88IKRkZFhSv3PP//cqFChguHj42NERUUZQ4YMMRITE+2ud7P3kdVqNcaMGWNERkYavr6+Rtu2bfP9e7tZ7dmzZ1/z/pdeesnh+rmns1/ra+XKlQ7Xv3TpktG9e3cjJibG8PHxMaKjo427777b2Lhxoym/m78q6KnnN6qflpZmtG/f3oiIiDC8vb2NuLg44+GHHzYSEhJMqZ9r1qxZRqVKlQw/Pz+jbt26xldffWVq/ffee8/w9/cv8L//N6t98uRJY8CAAUZMTIzh5+dnVK1a1Zg8eXK+t6e4Wf2pU6caZcuWNby9vY3Y2FjjH//4R77/+5Cfz6VLly4ZgwcPNkqWLGmUKFHC6N69u3Hy5MmC/IrsZvmzSREREZFiSWt2REREpFhT2BEREZFiTWFHREREijWFHRERESnWFHZERESkWFPYERERkWJNYUdERESKNYUdEZG/WLVqFRaL5arr+IhI0aSwIyIiIsWawo6IiIgUawo7IuJ2rFYr48ePJz4+Hn9/f+rWrcuCBQuA/00xLV68mDp16uDn50eTJk3YsWNHnhpffPEFNWvWxNfXl/Lly191UcaMjAyeffZZypUrh6+vL5UqVWLWrFl5HrNlyxYaNmxIiRIlaNas2VVXyxaRokFhR0Tczvjx4/nkk09499132blzJyNGjOC+++5j9erVtseMGjWKyZMns2nTJiIiIujSpQtZWVlATkjp3bs3ffv2Zfv27bz88suMGTOGjz76yPb8Bx54gE8//ZRp06axe/du3nvvPQIDA/P08cILLzB58mQ2b96Ml5eXaVeIF5HCpQuBiohbycjIICwsjOXLl9O0aVPb8Yceeoi0tDQeeeQRWrduzWeffUafPn0AOH/+PGXLluWjjz6id+/e9OvXjzNnzvD999/bnv/MM8+wePFidu7cyd69e6latSrLli2jXbt2V/WwatUqWrduzfLly2nbti0A3377LZ07d+bSpUv4+fk5+bcgImbSyI6IuJX9+/eTlpbG3/72NwIDA21fn3zyCQcOHLA97sogFBYWRtWqVdm9ezcAu3fvpnnz5nnqNm/enH379pGdnc22bdvw9PSkZcuWN+ylTp06ttvR0dEAnD592uGfUUQKl5erGxARuVJKSgoAixcvpkyZMnnu8/X1zRN47OXv75+vx3l7e9tuWywWIGc9kYgULRrZERG3UqNGDXx9fTly5AiVKlXK81WuXDnb437++Wfb7QsXLrB3716qV68OQPXq1Vm3bl2euuvWraNKlSp4enpSu3ZtrFZrnjVAIlJ8aWRHRNxKUFAQI0eOZMSIEVitVlq0aEFSUhLr1q0jODiYuLg4AMaOHUupUqWIjIzkhRdeIDw8nG7dugHw9NNP06hRI1599VX69OnD+vXrmTFjBm+//TYA5cuXp3///gwcOJBp06ZRt25d/vjjD06fPk3v3r1d9aOLiJMo7IiI23n11VeJiIhg/PjxHDx4kNDQUOrXr8/zzz9vm0aaMGECw4YNY9++fdx222188803+Pj4AFC/fn3mzZvHiy++yKuvvkp0dDRjx45lwIABttd45513eP755xk8eDDnzp0jNjaW559/3hU/rog4mc7GEpEiJfdMqQsXLhAaGurqdkSkCNCaHRERESnWFHZERESkWNM0loiIiBRrGtkRERGRYk1hR0RERIo1hR0REREp1hR2REREpFhT2BEREZFiTWFHREREijWFHRERESnWFHZERESkWFPYERERkWLt/wHSD7jskUcV5wAAAABJRU5ErkJggg==)"],"metadata":{"id":"bMw53NCzrcKL"}},{"cell_type":"markdown","source":["3.3.1 **(1pt)** What validation accuracy would you expect by chance for a 10-class problem?\n","\n","~10%\n"],"metadata":{"id":"0Z5xGWmwtvnO"}},{"cell_type":"markdown","source":["3.3.2 **(1pt)** Is your model overfitting, underfitting, or generalizing well?\n","\n","Overfitting\n"],"metadata":{"id":"YmJwbQ9SvwlN"}},{"cell_type":"markdown","source":["3.3.3 **(1pt)** If you believe the results are suspicious, what could be the problem?\n","\n","Data leakage, the validation dataset was created using the training data."],"metadata":{"id":"lNSfhRiwvzBr"}},{"cell_type":"markdown","source":["3.3.4 **(1pt)** Fix the issue by copying the problematic code from above into the cell below and correcting it there. Add a comment indicating exactly where you made the change."],"metadata":{"id":"RY7NT7Rdvjw5"}},{"cell_type":"code","source":["# number of samples\n","n_train = 100\n","n_val = 100\n","\n","# define random training images (28x28) and labels (between 0-9)\n","X_train = torch.randn(n_train, 1, 28, 28)\n","y_train = torch.randint(0, 10, (n_train,))\n","\n","# define random validation images (28x28) and labels (between 0-9)\n","X_val = torch.randn(n_val, 1, 28, 28)\n","y_val = torch.randint(0, 10, (n_val,))\n","\n","# create datasets\n","train_dataset = TensorDataset(X_train, y_train)\n","val_dataset = TensorDataset(X_val, y_val) # data leakage fixed!\n","\n","# create data loaders\n","train_loader = DataLoader(train_dataset, batch_size=10, shuffle=True)\n","val_loader = DataLoader(val_dataset, batch_size=10, shuffle=False)"],"metadata":{"id":"tB3M1uwlvlzw"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["3.3.5 **(1pt)** Retrain, plot, and evaluate the model after the fix using the same model and hyperparameters."],"metadata":{"id":"ziKqqQdVvmGv"}},{"cell_type":"code","source":["# train model\n","train_losses, val_losses, train_acc, val_acc = train(model, train_loader, val_loader, optimizer, n_epochs, criterion)\n","\n","# visualize results\n","plot(train_losses, val_losses, train_acc, val_acc, title='MLP')"],"metadata":{"id":"FUwrzjweweQT","executionInfo":{"status":"ok","timestamp":1774458199858,"user_tz":-60,"elapsed":1394,"user":{"displayName":"Olivia Lecomte","userId":"07830699269148802057"}},"colab":{"base_uri":"https://localhost:8080/","height":1000},"outputId":"d63f9e48-9674-42df-d636-672290ffde4b"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 131.55it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 1/20: train_loss: 0.0008, train_accuracy: 100.0000, val_loss: 2.9849, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 189.46it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 2/20: train_loss: 0.0008, train_accuracy: 100.0000, val_loss: 2.9878, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 191.51it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 3/20: train_loss: 0.0007, train_accuracy: 100.0000, val_loss: 2.9906, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 247.30it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 4/20: train_loss: 0.0007, train_accuracy: 100.0000, val_loss: 2.9933, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 314.87it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 5/20: train_loss: 0.0007, train_accuracy: 100.0000, val_loss: 2.9960, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 253.38it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 6/20: train_loss: 0.0007, train_accuracy: 100.0000, val_loss: 2.9987, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 231.38it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 7/20: train_loss: 0.0006, train_accuracy: 100.0000, val_loss: 3.0012, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 315.04it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 8/20: train_loss: 0.0006, train_accuracy: 100.0000, val_loss: 3.0037, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 351.10it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 9/20: train_loss: 0.0006, train_accuracy: 100.0000, val_loss: 3.0063, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 527.42it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 10/20: train_loss: 0.0006, train_accuracy: 100.0000, val_loss: 3.0089, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 521.58it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 11/20: train_loss: 0.0006, train_accuracy: 100.0000, val_loss: 3.0113, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 525.58it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 12/20: train_loss: 0.0006, train_accuracy: 100.0000, val_loss: 3.0139, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 498.42it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 13/20: train_loss: 0.0005, train_accuracy: 100.0000, val_loss: 3.0164, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 512.28it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 14/20: train_loss: 0.0005, train_accuracy: 100.0000, val_loss: 3.0188, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 508.19it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 15/20: train_loss: 0.0005, train_accuracy: 100.0000, val_loss: 3.0211, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 511.06it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 16/20: train_loss: 0.0005, train_accuracy: 100.0000, val_loss: 3.0233, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 420.79it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 17/20: train_loss: 0.0005, train_accuracy: 100.0000, val_loss: 3.0257, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 531.41it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 18/20: train_loss: 0.0005, train_accuracy: 100.0000, val_loss: 3.0280, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 552.12it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 19/20: train_loss: 0.0005, train_accuracy: 100.0000, val_loss: 3.0302, val_accuracy: 11.0000\n"]},{"output_type":"stream","name":"stderr","text":["100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:00<00:00, 403.98it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Epoch 20/20: train_loss: 0.0005, train_accuracy: 100.0000, val_loss: 3.0325, val_accuracy: 11.0000\n"]},{"output_type":"display_data","data":{"text/plain":["
"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAASUFJREFUeJzt3XlYVHX///HXALKogKJsKgLu+26Jeqcmue+WppaaLXeldy5pZeX+K7RS0zTN7tI2csul3NFcyn2jtEyz3FJAMwVBRWPO7w+/zu0IKiODA6fn47rOFXPmnPd5Dzh9XnOWORbDMAwBAACYhJurGwAAAHAmwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg2AfCsiIkJ9+/Z1dRu3dPToUVksFs2ZM+e2y23YsEEWi0UbNmy4J30BZke4AfKBOXPmyGKxyGKx6Pvvv8/0vGEYCgsLk8ViUbt27eyes1gsGjBgwG3rN23a1FbfYrEoICBA9evX18cffyyr1epQr9cH6uxMecE333wjNzc3JSYmuroVAE7i4eoGAGSft7e3YmNj1bhxY7v5Gzdu1B9//CEvL6+7rl2qVCnFxMRIks6cOaNPP/1UTz75pA4dOqTx48dnu07lypX12Wef2c0bPny4ChcurNdee+2u+8vKwYMH5eaWs89oy5cvV926dRUSEuKkrgC4GuEGyEfatGmjBQsWaOrUqfLw+N/bNzY2VnXr1tWff/5517X9/f312GOP2R7/+9//VsWKFTVt2jSNGzdOBQoUyFad4OBguzqSNH78eBUvXjzT/BtZrVZduXJF3t7e2e45J2HuuhUrVqhfv345rgMg7+CwFJCP9OjRQ2fPnlVcXJxt3pUrV7Rw4UL17NnTqdsqWLCgGjRooLS0NJ05c0aSdPHiRf3yyy85ClHXXT9c9sUXX6hq1ary8vLSqlWrJEnvvPOOGjZsqGLFisnHx0d169bVwoULM9W4+Zyb64fvNm/erCFDhigwMFCFChVS586dba/hRvv27dOJEyfUtm1bJSUlycPDQ2PGjMm03MGDB2WxWDRt2jRJ0l9//aWhQ4eqevXqKly4sPz8/NS6dWv98MMPOf693GjBggWqW7eufHx8bOHw5MmTdsskJibqiSeeUKlSpeTl5aXQ0FB17NhRR48etS2za9cutWzZUsWLF5ePj48iIyMJdDA1wg2Qj0RERCgqKkpffvmlbd7KlSuVnJysRx991Onb+/333+Xu7q4iRYpIknbs2KHKlSvbBvmc+vbbbzV48GB1795dU6ZMUUREhCRpypQpql27tsaOHas333xTHh4eeuSRR7R8+fJs1f3Pf/6jH374QaNGjdJzzz2nb775JsvzjlasWKGgoCDVq1dPwcHBatKkiebPn59puXnz5snd3V2PPPKIpGu/lyVLlqhdu3aaNGmShg0bpn379qlJkyY6derU3f9CbjBnzhx169ZN7u7uiomJ0dNPP61FixapcePGOn/+vG25rl27avHixXriiSf0/vvv64UXXtCFCxd0/PhxSdLp06fVokULHT16VK+88oree+899erVS9u2bXNKn0BexGEpIJ/p2bOnhg8frkuXLsnHx0dffPGFmjRpohIlSuSobkZGhm2PzJ9//qkZM2Zoz549at++vQoWLOiM1jM5ePCg9u3bpypVqtjNP3TokHx8fGyPBwwYoDp16mjSpElq27btHesWK1ZMa9assZ20bLVaNXXqVCUnJ8vf39+23PLly9W6dWvbct27d9e///1v7d+/X9WqVbMtN2/ePDVp0kTBwcGSpOrVq+vQoUN25/s8/vjjqlSpkj766CONGDHiLn4b/3P16lW9/PLLqlatmjZt2mQ7VNe4cWO1a9dOkydP1pgxY3T+/Hlt2bJFb7/9toYOHWpbf/jw4baft2zZonPnzmnNmjWqV6+ebf7/+3//L0c9AnkZe26AfKZbt266dOmSli1bpgsXLmjZsmVOOST1yy+/KDAwUIGBgapcubLee+89tW3bVh9//LFtmaZNm8owDI0ePTrH25OkJk2aZAo2kuyCzblz55ScnKx//etf2rNnT7bqPvPMM3ZXY/3rX/9SRkaGjh07Zpt3/vx5bd261S4sdenSRR4eHpo3b55t3v79+/Xzzz+re/futnleXl62YJORkaGzZ8+qcOHCqlixYrZ7vJ1du3bp9OnTev755+3OQWrbtq0qVapk24Pl4+MjT09PbdiwQefOncuy1vW9bsuWLdPVq1dz3BuQHxBugHwmMDBQ0dHRio2N1aJFi5SRkaGHH344x3UjIiIUFxentWvX6vvvv1diYqKWLVum4sWLO6HrrEVGRmY5f9myZWrQoIG8vb0VEBCgwMBAzZgxQ8nJydmqW7p0abvHRYsWlSS7ALB69WpJUosWLWzzihcvrubNm9sdmpo3b548PDzUpUsX2zyr1arJkyerfPny8vLyUvHixRUYGKgff/wx2z3ezvUQVrFixUzPVapUyfa8l5eXJkyYoJUrVyo4OFgPPPCA3nrrLbvL2ps0aaKuXbtqzJgxKl68uDp27KjZs2crPT09x30CeRXhBsiHevbsqZUrV2rmzJlq3bq17dN5ThQqVEjR0dFq3ry5GjVqpKCgoJw3egc37qG57rvvvlOHDh3k7e2t999/XytWrFBcXJx69uwpwzCyVdfd3T3L+Teuv2LFCjVq1MjuMJUkPfroozp06JDi4+MlSfPnz1fz5s3tQt6bb76pIUOG6IEHHtDnn3+u1atXKy4uTlWrVnX4e4FyatCgQTp06JBiYmLk7e2tESNGqHLlytq7d6+kayduL1y4UFu3btWAAQN08uRJ9evXT3Xr1lVqauo97RW4Vwg3QD7UuXNnubm5adu2bU6/SsrVvvrqK3l7e2v16tXq16+fWrdurejoaKduwzAMrVq1Ksvzdzp16iRPT0/NmzdP8fHxOnToUKaTtRcuXKhmzZrpo48+0qOPPqoWLVooOjra7kTfnAgPD5d07Zykmx08eND2/HVly5bViy++qDVr1mj//v26cuWKJk6caLdMgwYN9MYbb2jXrl364osv9NNPP2nu3LlO6RfIawg3QD5UuHBhzZgxQ6NHj1b79u3v2XadeSn4rbi7u8tisSgjI8M27+jRo1qyZInTtrFz506dPn06y3BTpEgRtWzZUvPnz9fcuXPl6empTp06Zerx5r1ICxYsyHSZ9t2qV6+egoKCNHPmTLvDRytXrtSBAwdsfV+8eFGXL1+2W7ds2bLy9fW1rXfu3LlMvdaqVUuSODQF0+JqKSCf6tOnT7aX3bVrV5ZXxzRt2jTTtx3fzo4dO9SsWTONGjXKaScV36xt27aaNGmSWrVqpZ49e+r06dOaPn26ypUrpx9//NEp21i+fLkiIiKyPJlZunbV1GOPPab3339fLVu2zHTYr127dho7dqyeeOIJNWzYUPv27dMXX3yhMmXKOKW/AgUKaMKECXriiSfUpEkT9ejRQ0lJSbbL5QcPHizp2lVlzZs3V7du3VSlShV5eHho8eLFSkpKsu1t+uSTT/T++++rc+fOKlu2rC5cuKAPP/xQfn5+atOmjVP6BfIawg3wD7B9+3Zt37490/xx48Y5FG7uhQcffFAfffSRxo8fr0GDBikyMlITJkzQ0aNHnRZuVqxYcduBvUOHDvLx8dGFCxfsrpK67tVXX1VaWppiY2M1b9481alTR8uXL9crr7zilP4kqW/fvipYsKDGjx+vl19+2fZlhBMmTLCFrbCwMPXo0UPr1q3TZ599Jg8PD1WqVEnz589X165dJV07oXjHjh2aO3eukpKS5O/vr/vuu09ffPHFLU/oBvI7i5HdM/QAwASSkpIUGhqqZcuWsecCMCnOuQHwj5KcnKyRI0eqWbNmrm4FQC5hzw0AADAV9twAAABTIdwAAABTIdwAAABTIdwAAABT+cd9z43VatWpU6fk6+trd9dgAACQdxmGoQsXLqhEiRJyc7v9vpl/XLg5deqUwsLCXN0GAAC4CydOnFCpUqVuu8w/Ltz4+vpKuvbL8fPzc3E3AAAgO1JSUhQWFmYbx2/nHxdurh+K8vPzI9wAAJDPZOeUEk4oBgAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApvKPu3EmAABwkNUqGVbJyJCsGf/72bD+33PXf8649rO7p+Qb4rJ2XRpuZsyYoRkzZujo0aOSpKpVq2rkyJFq3br1LddZsGCBRowYoaNHj6p8+fKaMGGC2rRpc486BgC4lGHcMIjePKha/zddH2Rt/7Xe9Dir+Tf9nGnZm+bbLXOLwT/T/Cz6zWp+Vv3ccX5W/WURPG7s6Y7L/99jR4XdLz25xvl//2xyabgpVaqUxo8fr/Lly8swDH3yySfq2LGj9u7dq6pVq2ZafsuWLerRo4diYmLUrl07xcbGqlOnTtqzZ4+qVavmglcAAP/HMO48GN48/64HYWfPv1WPxq0HZUcG2VsO8HcaZLMYaGW4+i+NW7G4SxY3yc392p4bV7ZiGEae+pcSEBCgt99+W08++WSm57p37660tDQtW7bMNq9BgwaqVauWZs6cma36KSkp8vf3V3Jysvz8/JzWN5Dv3G7ws/59iwHw76wHxUzLZ2MAta2TC4PwrWpkGpzvJgz8nfWyDLp5i8Xt2mDr5n7Df91ueuwuuWW13K3mZ7G+xe2mZW/82XKL+W72tex+tmSx/PV+srOdG+tlZzvZ2b5b5h4y1c39U3gdGb/zzDk3GRkZWrBggdLS0hQVFZXlMlu3btWQIUPs5rVs2VJLliy5Bx0iT8hyoLl5sMpq3u0G01sMVta/b/+p+55+gnZke9ncFoPxPWbJ5gDqwMB68/wbB7U7LZutQf+mPjLVv/n5m1+jm/3rybQdN9kPwDcvd7vnLFkse+8GWuRtLg83+/btU1RUlC5fvqzChQtr8eLFqlKlSpbLJiYmKjg42G5ecHCwEhMTb1k/PT1d6enptscpKSnOadwZbAPO3zcNpjc+vmGwuvlxpkH7VvNv+iR+46B9q3UzhYSbPrHfNjTcadnbBBG7PQBZbJcB+R7JaiC+wyDs5pE7A/bdDMJ3qpHTT/BuHnfRu8XVf1TgH8Pl4aZixYqKj49XcnKyFi5cqD59+mjjxo23DDiOiomJ0ZgxY5xS67YSfpAWPnnnQHLjYwZqJ7NcG3SyGmhsA28Wg8/tBiqHBtibP606a/C+Pv8Wr+22NTxus6zH7Qd0AMinXB5uPD09Va5cOUlS3bp1tXPnTk2ZMkUffPBBpmVDQkKUlJRkNy8pKUkhIbe+3Gz48OF2h7JSUlIUFhbmpO5vkHFVOvur8+rdODDdOPjaBukbByYP+0HpxoE8y3WzGixvGuhu/BSenXqZwsKtwkE2+75t3azW55MxAOAal4ebm1mtVrvDSDeKiorSunXrNGjQINu8uLi4W56jI0leXl7y8vJydpuZFa8g9V1xUwDwyHqwvzmg2D2+/smfgRoAgLvh0nAzfPhwtW7dWqVLl9aFCxcUGxurDRs2aPXq1ZKk3r17q2TJkoqJiZEkDRw4UE2aNNHEiRPVtm1bzZ07V7t27dKsWbNc+TKu8faTIhq5ugsAAP7xXBpuTp8+rd69eyshIUH+/v6qUaOGVq9erYceekiSdPz4cbndcOy/YcOGio2N1euvv65XX31V5cuX15IlS/iOGwAAYJPnvucmt/E9NwAA5D+OjN9cEgEAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEyFcAMAAEzFpeEmJiZG9evXl6+vr4KCgtSpUycdPHjwtuvMmTNHFovFbvL29r5HHQMAgLzOpeFm48aN6t+/v7Zt26a4uDhdvXpVLVq0UFpa2m3X8/PzU0JCgm06duzYPeoYAADkdR6u3PiqVavsHs+ZM0dBQUHavXu3HnjggVuuZ7FYFBISktvtAQCAfChPnXOTnJwsSQoICLjtcqmpqQoPD1dYWJg6duyon3766ZbLpqenKyUlxW4CAADmlWfCjdVq1aBBg9SoUSNVq1btlstVrFhRH3/8sZYuXarPP/9cVqtVDRs21B9//JHl8jExMfL397dNYWFhufUSAABAHmAxDMNwdROS9Nxzz2nlypX6/vvvVapUqWyvd/XqVVWuXFk9evTQuHHjMj2fnp6u9PR02+OUlBSFhYUpOTlZfn5+TukdAADkrpSUFPn7+2dr/HbpOTfXDRgwQMuWLdOmTZscCjaSVKBAAdWuXVuHDx/O8nkvLy95eXk5o00AAJAPuPSwlGEYGjBggBYvXqxvv/1WkZGRDtfIyMjQvn37FBoamgsdAgCA/Male2769++v2NhYLV26VL6+vkpMTJQk+fv7y8fHR5LUu3dvlSxZUjExMZKksWPHqkGDBipXrpzOnz+vt99+W8eOHdNTTz3lstcBAADyDpeGmxkzZkiSmjZtajd/9uzZ6tu3ryTp+PHjcnP73w6mc+fO6emnn1ZiYqKKFi2qunXrasuWLapSpcq9ahsAAORheeaE4nvFkROSAABA3uDI+J1nLgUHAABwBsINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFcINAAAwFZeGm5iYGNWvX1++vr4KCgpSp06ddPDgwTuut2DBAlWqVEne3t6qXr26VqxYcQ+6BQAA+YFLw83GjRvVv39/bdu2TXFxcbp69apatGihtLS0W66zZcsW9ejRQ08++aT27t2rTp06qVOnTtq/f/897BwAAORVFsMwDFc3cd2ZM2cUFBSkjRs36oEHHshyme7duystLU3Lli2zzWvQoIFq1aqlmTNn3nEbKSkp8vf3V3Jysvz8/JzWOwAAyD2OjN956pyb5ORkSVJAQMAtl9m6dauio6Pt5rVs2VJbt27Ncvn09HSlpKTYTQAAwLzyTLixWq0aNGiQGjVqpGrVqt1yucTERAUHB9vNCw4OVmJiYpbLx8TEyN/f3zaFhYU5tW8AAJC35Jlw079/f+3fv19z5851at3hw4crOTnZNp04ccKp9QEAQN7i4eoGJGnAgAFatmyZNm3apFKlSt122ZCQECUlJdnNS0pKUkhISJbLe3l5ycvLy2m9AgCAvM2le24Mw9CAAQO0ePFiffvtt4qMjLzjOlFRUVq3bp3dvLi4OEVFReVWmwAAIB9x6Z6b/v37KzY2VkuXLpWvr6/tvBl/f3/5+PhIknr37q2SJUsqJiZGkjRw4EA1adJEEydOVNu2bTV37lzt2rVLs2bNctnrAAAAeYdL99zMmDFDycnJatq0qUJDQ23TvHnzbMscP35cCQkJtscNGzZUbGysZs2apZo1a2rhwoVasmTJbU9CBgAA/xx56ntu7gW+5wYAgPwn337PDQAAQE4RbgAAgKkQbgAAgKkQbgAAgKkQbgAAgKkQbgAAgKkQbgAAgKkQbgAAgKkQbgAAgKkQbgAAgKkQbgAAgKkQbgAAgKkQbgAAgKkQbgAAgKncdbg5fPiwVq9erUuXLkmSDMNwWlMAAAB3y+Fwc/bsWUVHR6tChQpq06aNEhISJElPPvmkXnzxRac3CAAA4AiHw83gwYPl4eGh48ePq2DBgrb53bt316pVq5zaHAAAgKM8HF1hzZo1Wr16tUqVKmU3v3z58jp27JjTGgMAALgbDu+5SUtLs9tjc91ff/0lLy8vpzQFAABwtxwON//617/06aef2h5bLBZZrVa99dZbatasmVObAwAAcJTDh6XeeustNW/eXLt27dKVK1f00ksv6aefftJff/2lzZs350aPAAAA2ebwnptq1arp0KFDaty4sTp27Ki0tDR16dJFe/fuVdmyZXOjRwAAgGyzGP+wL6hJSUmRv7+/kpOT5efn5+p2AABANjgyfjt8WGrTpk23ff6BBx5wtCQAADmWkZGhq1evuroN5ICnp6fc3HJ+8wSHw03Tpk0zzbNYLLafMzIyctQQAACOMAxDiYmJOn/+vKtbQQ65ubkpMjJSnp6eOarjcLg5d+6c3eOrV69q7969GjFihN54440cNQMAgKOuB5ugoCAVLFjQ7gM38g+r1apTp04pISFBpUuXztHf0eFw4+/vn2neQw89JE9PTw0ZMkS7d+++62YAAHBERkaGLdgUK1bM1e0ghwIDA3Xq1Cn9/fffKlCgwF3XcdpdwYODg3Xw4EFnlQMA4I6un2OT1ZfLIv+5fjgqp6e4OLzn5scff7R7bBiGEhISNH78eNWqVStHzQAAcDc4FGUOzvo7OhxuatWqJYvFopuvIG/QoIE+/vhjpzQFAABwtxwON0eOHLF77ObmpsDAQHl7ezutKQAAkH0REREaNGiQBg0alONaGzZsULNmzXTu3DkVKVIkx/VcweFwEx4enht9AADwj9K0aVPVqlVL7777bo5r7dy5U4UKFcp5UyaRrXAzderUbBd84YUX7roZAABwjWEYysjIkIfHnYfqwMDAe9BRPmJkQ0RERLamyMjI7JRzqeTkZEOSkZyc7OpWAAA5dOnSJePnn382Ll26ZJtntVqNtPSrLpmsVmu2+u7Tp48hyW6aPXu2IclYsWKFUadOHaNAgQLG+vXrjcOHDxsdOnQwgoKCjEKFChn16tUz4uLi7OqFh4cbkydPtj2WZHz44YdGp06dDB8fH6NcuXLG0qVLs9Xb+vXrDUnGuXPnbPMWLlxoVKlSxfD09DTCw8ONd955x26d6dOnG+XKlTO8vLyMoKAgo2vXrrbnFixYYFSrVs3w9vY2AgICjObNmxupqalZbjurv+d1jozf2dpzc/N5NgAA5FWXrmaoysjVLtn2z2NbqqDnnYfWKVOm6NChQ6pWrZrGjh0rSfrpp58kSa+88oreeecdlSlTRkWLFtWJEyfUpk0bvfHGG/Ly8tKnn36q9u3b6+DBgypduvQttzFmzBi99dZbevvtt/Xee++pV69eOnbsmAICAhx6Tbt371a3bt00evRode/eXVu2bNHzzz+vYsWKqW/fvtq1a5deeOEFffbZZ2rYsKH++usvfffdd5KkhIQE9ejRQ2+99ZY6d+6sCxcu6Lvvvst0UZKzOXzODQAAyBl/f395enqqYMGCCgkJkST98ssvkqSxY8fqoYcesi0bEBCgmjVr2h6PGzdOixcv1tdff60BAwbccht9+/ZVjx49JElvvvmmpk6dqh07dqhVq1YO9Tpp0iQ1b95cI0aMkCRVqFBBP//8s95++2317dtXx48fV6FChdSuXTv5+voqPDxctWvXlnQt3Pz999/q0qWL7Zzd6tWrO7T9u3FX4eaPP/7Q119/rePHj+vKlSt2z02aNMkpjQEAcDd8Crjr57EtXbbtnKpXr57d49TUVI0ePVrLly+3hYVLly7p+PHjt61To0YN28+FChWSn5+fTp8+7XA/Bw4cUMeOHe3mNWrUSO+++64yMjL00EMPKTw8XGXKlFGrVq3UqlUrde7cWQULFlTNmjXVvHlzVa9eXS1btlSLFi308MMPq2jRog734QiHw826devUoUMHlSlTRr/88ouqVaumo0ePyjAM1alTJzd6BAAg2ywWS7YODeVVN1/1NHToUMXFxemdd95RuXLl5OPjo4cffjjTzoWb3Xz7AovFIqvV6vR+fX19tWfPHm3YsEFr1qzRyJEjNXr0aO3cuVNFihRRXFyctmzZojVr1ui9997Ta6+9pu3btysyMtLpvVzn8O0Xhg8frqFDh2rfvn3y9vbWV199pRMnTqhJkyZ65JFHcqNHAABMx9PTM1u3Gdi8ebP69u2rzp07q3r16goJCdHRo0dzv8H/U7lyZW3evDlTTxUqVJC7+7U9VR4eHoqOjtZbb72lH3/8UUePHtW3334r6VqoatSokcaMGaO9e/fK09NTixcvztWeHY62Bw4c0JdffnltZQ8PXbp0SYULF9bYsWPVsWNHPffcc05vEgAAs4mIiND27dt19OhRFS5c+JZ7VcqXL69Fixapffv2slgsGjFiRK7sgbmVF198UfXr19e4cePUvXt3bd26VdOmTdP7778vSVq2bJl+//13PfDAAypatKhWrFghq9WqihUravv27Vq3bp1atGihoKAgbd++XWfOnFHlypVztWeH99wUKlTItissNDRUv/32m+25P//803mdAQBgYkOHDpW7u7uqVKmiwMDAW55DM2nSJBUtWlQNGzZU+/bt1bJly3t6GkidOnU0f/58zZ07V9WqVdPIkSM1duxY9e3bV5JUpEgRLVq0SA8++KAqV66smTNn6ssvv1TVqlXl5+enTZs2qU2bNqpQoYJef/11TZw4Ua1bt87Vni2Gg9djderUSW3bttXTTz+toUOHaunSperbt68WLVqkokWLau3atbnVq1OkpKTI399fycnJ8vPzc3U7AIAcuHz5so4cOaLIyEhuA2QCt/t7OjJ+O3xYatKkSUpNTZV07Rr61NRUzZs3T+XLl+dKKQAA4HIOh5syZcrYfi5UqJBmzpzp1IYAAEDuefbZZ/X5559n+dxjjz1minHd4XNunnrqKW3YsMEpG9+0aZPat2+vEiVKyGKxaMmSJbddfsOGDbJYLJmmxMREp/QDAIDZjR07VvHx8VlO178tOb9zeM/NmTNn1KpVKwUGBurRRx/VY489ZvfNiY5IS0tTzZo11a9fP3Xp0iXb6x08eNDueFtQUNBdbR8AgH+aoKAg04+bDoebpUuX6ty5c1qwYIFiY2M1adIkVapUSb169VLPnj0VERGR7VqtW7e+qzOmg4KCVKRIEYfXAwAA5ufwYSlJKlq0qJ555hlt2LBBx44dU9++ffXZZ5+pXLlyzu4vS7Vq1VJoaKgeeuihTF8sBAAA/tly9P3UV69e1a5du2xfQhQcHOysvrIUGhqqmTNnql69ekpPT9d///tfNW3aVNu3b7/lNf/p6elKT0+3PU5JScnVHgEAgGvdVbhZv369YmNj9dVXX8lqtapLly5atmyZHnzwQWf3Z6dixYqqWLGi7XHDhg3122+/afLkyfrss8+yXCcmJkZjxozJ1b4AAEDe4XC4KVmypP766y+1atVKs2bNUvv27eXl5ZUbvWXLfffdp++///6Wzw8fPlxDhgyxPU5JSVFYWNi9aA0AALiAw+fcjB49WgkJCVq8eLEefvhhlwYbSYqPj1doaOgtn/fy8pKfn5/dBABAfhcREaF33303W8tm5+tWzMThPTdPP/200zaempqqw4cP2x4fOXJE8fHxCggIUOnSpTV8+HCdPHlSn376qSTp3XffVWRkpKpWrarLly/rv//9r7799lutWbPGaT0BAID8LUcnFOfUrl271KxZM9vj64eP+vTpozlz5ighIcHuRmJXrlzRiy++qJMnT6pgwYKqUaOG1q5da1cDAAD8s93VpeDO0rRpUxmGkWmaM2eOJGnOnDl234b80ksv6fDhw7p06ZLOnj2r9evXE2wAAPnOrFmzVKJECVmtVrv5HTt2VL9+/fTbb7+pY8eOCg4OVuHChVW/fn2n3ph63759evDBB+Xj46NixYrpmWeesd03Urp2R4D77rtPhQoVUpEiRdSoUSMdO3ZMkvTDDz+oWbNm8vX1lZ+fn+rWratdu3Y5rTdncGm4AQDA6QxDupLmmskwstXiI488YvuQft1ff/2lVatWqVevXkpNTVWbNm20bt067d27V61atVL79u3tjmbcrbS0NLVs2VJFixbVzp07tWDBAq1du1YDBgyQJP3999/q1KmTmjRpoh9//FFbt27VM888I4vFIknq1auXSpUqpZ07d2r37t165ZVXVKBAgRz35UwuPSwFAIDTXb0ovVnCNdt+9ZTkWeiOixUtWlStW7dWbGysmjdvLklauHChihcvrmbNmsnNzc3u1kbjxo3T4sWL9fXXX9tCyN2KjY3V5cuX9emnn6pQoWu9Tps2Te3bt9eECRNUoEABJScnq127dipbtqwkqXLlyrb1jx8/rmHDhqlSpUqSpPLly+eon9zg8J6bTz75RMuXL7c9fumll1SkSBE1bNjQtssKAADcXq9evfTVV1/Zvmj2iy++0KOPPio3NzelpqZq6NChqly5sooUKaLChQvrwIEDTtlzc+DAAdWsWdMWbCSpUaNGslqtOnjwoAICAtS3b1+1bNlS7du315QpU5SQkGBbdsiQIXrqqacUHR2t8ePH67fffstxT87m8J6bN998UzNmzJAkbd26VdOnT9fkyZO1bNkyDR48WIsWLXJ6kwAAZFuBgtf2oLhq29nUvn17GYah5cuXq379+vruu+80efJkSdLQoUMVFxend955R+XKlZOPj48efvhhXblyJbc6tzN79my98MILWrVqlebNm6fXX39dcXFxatCggUaPHq2ePXtq+fLlWrlypUaNGqW5c+eqc+fO96S37HA43Jw4ccJ2D6klS5aoa9eueuaZZ9SoUSM1bdrU2f0BAOAYiyVbh4ZczdvbW126dNEXX3yhw4cPq2LFirZbCW3evFl9+/a1BYbU1FQdPXrUKdutXLmy5syZo7S0NNvem82bN8vNzc3uLgC1a9dW7dq1NXz4cEVFRSk2NlYNGjSQJFWoUEEVKlTQ4MGD1aNHD82ePTtPhRuHD0sVLlxYZ8+elSStWbNGDz30kKRrf6RLly45tzsAAEysV69eWr58uT7++GP16tXLNr98+fJatGiR4uPj9cMPP6hnz56ZrqzKyTa9vb3Vp08f7d+/X+vXr9d//vMfPf744woODtaRI0c0fPhwbd26VceOHdOaNWv066+/qnLlyrp06ZIGDBhgu3H25s2btXPnTrtzcvICh/fcPPTQQ3rqqadUu3ZtHTp0SG3atJEk/fTTT4qIiHB2fwAAmNaDDz6ogIAAHTx4UD179rTNnzRpkvr166eGDRuqePHievnll5124+eCBQtq9erVGjhwoOrXr6+CBQuqa9eumjRpku35X375RZ988onOnj2r0NBQ9e/fX//+97/1999/6+zZs+rdu7eSkpJUvHhxdenSJc/dw9FiGNm8bu3/nD9/Xq+//rpOnDih5557Tq1atZIkjRo1Sp6ennrttddypVFnSUlJkb+/v5KTk7kVAwDkc5cvX9aRI0cUGRkpb29vV7eDHLrd39OR8dvhPTdFihTRtGnTMs3Pa6kNAAD8Mzl8zs2qVavs7sI9ffp01apVSz179tS5c+ec2hwAALi9L774QoULF85yqlq1qqvbcwmH99wMGzZMEyZMkHTt65tffPFFDRkyROvXr9eQIUM0e/ZspzcJAACy1qFDB91///1ZPpfXvjn4XnE43Bw5ckRVqlSRJH311Vdq166d3nzzTe3Zs8d2cjEAALg3fH195evr6+o28hSHD0t5enrq4sWLkqS1a9eqRYsWkqSAgACnnckNAIAjHLw2BnmUs/6ODu+5ady4sYYMGaJGjRppx44dmjdvniTp0KFDKlWqlFOaAgAgO64fdrl48aJ8fHxc3A1y6vo3MLu7u+eojsPhZtq0aXr++ee1cOFCzZgxQyVLlpQkrVy50nZZOAAA94K7u7uKFCmi06dPS7r2HS3X716N/MVqterMmTMqWLCgPDxydl9vh7/nJr/je24AwFwMw1BiYqLOnz/v6laQQ25uboqMjJSnp2em53L1e24kKSMjQ0uWLNGBAwckSVWrVlWHDh1yvBsJAABHWSwWhYaGKigoSFevXnV1O8gBT09Pubk5fDpwJg6Hm8OHD6tNmzY6efKk7QZbMTExCgsL0/Lly1W2bNkcNwUAgKPc3d35kA1Jd3G11AsvvKCyZcvqxIkT2rNnj/bs2aPjx48rMjJSL7zwQm70CAAAkG0O77nZuHGjtm3bpoCAANu8YsWKafz48WrUqJFTmwMAAHCUw3tuvLy8dOHChUzzU1NTszwBCAAA4F5yONy0a9dOzzzzjLZv3y7DMGQYhrZt26Znn31WHTp0yI0eAQAAss3hcDN16lSVLVtWUVFR8vb2lre3txo1aqRy5cppypQpudEjAABAtjl8zk2RIkW0dOlS/frrr/rll18kSZUrV1a5cuWc3hwAAICj7vorAMuXL6/y5cs7sxcAAIAcy1a4GTJkSLYLTpo06a6bAQAAyKlshZu9e/dmqxj38wAAAK6WrXCzfv363O4DAADAKXJ+AwcAAIA8hHADAABMhXADAABMhXADAABMhXADAABMhXADAABMhXADAABMhXADAABMhXADAABMhXADAABMhXADAABMhXADAABMhXADAABMhXADAABMhXADAABMhXADAABMhXADAABMxaXhZtOmTWrfvr1KlCghi8WiJUuW3HGdDRs2qE6dOvLy8lK5cuU0Z86cXO8TAADkHy4NN2lpaapZs6amT5+ereWPHDmitm3bqlmzZoqPj9egQYP01FNPafXq1bncKQAAyC88XLnx1q1bq3Xr1tlefubMmYqMjNTEiRMlSZUrV9b333+vyZMnq2XLlrnVJgAAyEfy1Tk3W7duVXR0tN28li1bauvWrbdcJz09XSkpKXYTAAAwr3wVbhITExUcHGw3Lzg4WCkpKbp06VKW68TExMjf3982hYWF3YtWAQCAi+SrcHM3hg8fruTkZNt04sQJV7cEAABykUvPuXFUSEiIkpKS7OYlJSXJz89PPj4+Wa7j5eUlLy+ve9EeAADIA/LVnpuoqCitW7fObl5cXJyioqJc1BEAAMhrXBpuUlNTFR8fr/j4eEnXLvWOj4/X8ePHJV07pNS7d2/b8s8++6x+//13vfTSS/rll1/0/vvva/78+Ro8eLAr2gcAAHmQS8PNrl27VLt2bdWuXVuSNGTIENWuXVsjR46UJCUkJNiCjiRFRkZq+fLliouLU82aNTVx4kT997//5TJwAABgYzEMw3B1E/dSSkqK/P39lZycLD8/P1e3AwAAssGR8TtfnXMDAABwJ4QbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKoQbAABgKnki3EyfPl0RERHy9vbW/fffrx07dtxy2Tlz5shisdhN3t7e97BbAACQl7k83MybN09DhgzRqFGjtGfPHtWsWVMtW7bU6dOnb7mOn5+fEhISbNOxY8fuYccAACAvc3m4mTRpkp5++mk98cQTqlKlimbOnKmCBQvq448/vuU6FotFISEhtik4OPgedgwAAPIyl4abK1euaPfu3YqOjrbNc3NzU3R0tLZu3XrL9VJTUxUeHq6wsDB17NhRP/30071oFwAA5AMuDTd//vmnMjIyMu15CQ4OVmJiYpbrVKxYUR9//LGWLl2qzz//XFarVQ0bNtQff/yR5fLp6elKSUmxmwAAgHm5/LCUo6KiotS7d2/VqlVLTZo00aJFixQYGKgPPvggy+VjYmLk7+9vm8LCwu5xxwAA4F5yabgpXry43N3dlZSUZDc/KSlJISEh2apRoEAB1a5dW4cPH87y+eHDhys5Odk2nThxIsd9AwCAvMul4cbT01N169bVunXrbPOsVqvWrVunqKiobNXIyMjQvn37FBoamuXzXl5e8vPzs5sAAIB5ebi6gSFDhqhPnz6qV6+e7rvvPr377rtKS0vTE088IUnq3bu3SpYsqZiYGEnS2LFj1aBBA5UrV07nz5/X22+/rWPHjumpp55y5csAAAB5hMvDTffu3XXmzBmNHDlSiYmJqlWrllatWmU7yfj48eNyc/vfDqZz587p6aefVmJioooWLaq6detqy5YtqlKliqteAgAAyEMshmEYrm7iXkpJSZG/v7+Sk5M5RAUAQD7hyPid766WAgAAuB3CDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMJU8EW6mT5+uiIgIeXt76/7779eOHTtuu/yCBQtUqVIleXt7q3r16lqxYsU96hQAAOR1Lg838+bN05AhQzRq1Cjt2bNHNWvWVMuWLXX69Oksl9+yZYt69OihJ598Unv37lWnTp3UqVMn7d+//x53DgAA8iKLYRiGKxu4//77Vb9+fU2bNk2SZLVaFRYWpv/85z965ZVXMi3fvXt3paWladmyZbZ5DRo0UK1atTRz5sw7bi8lJUX+/v5KTk6Wn5+f017H5asZ+uPcJafVuxWLJdc3katys31LPv/l5Ofu8/mvXpZ8/NvP77973Fp+/tt6ergpyNfbqTUdGb89nLplB125ckW7d+/W8OHDbfPc3NwUHR2trVu3ZrnO1q1bNWTIELt5LVu21JIlS7JcPj09Xenp6bbHKSkpOW88Cz8npKjL+1typTYAAPlJndJFtOj5Ri7bvkvDzZ9//qmMjAwFBwfbzQ8ODtYvv/yS5TqJiYlZLp+YmJjl8jExMRozZoxzGr4NDzeLihQskKvbyO19bC7eiZcjud55/v3V5OfWJfHv0pXy8a8+1xn5/q+buzw9XHvWi0vDzb0wfPhwuz09KSkpCgsLc/p2apQqoviRLZxeFwAAOMal4aZ48eJyd3dXUlKS3fykpCSFhIRkuU5ISIhDy3t5ecnLy8s5DQMAgDzPpfuNPD09VbduXa1bt842z2q1at26dYqKispynaioKLvlJSkuLu6WywMAgH8Wlx+WGjJkiPr06aN69erpvvvu07vvvqu0tDQ98cQTkqTevXurZMmSiomJkSQNHDhQTZo00cSJE9W2bVvNnTtXu3bt0qxZs1z5MgAAQB7h8nDTvXt3nTlzRiNHjlRiYqJq1aqlVatW2U4aPn78uNzc/reDqWHDhoqNjdXrr7+uV199VeXLl9eSJUtUrVo1V70EAACQh7j8e27utdz6nhsAAJB7HBm/Xf4NxQAAAM5EuAEAAKZCuAEAAKZCuAEAAKZCuAEAAKZCuAEAAKZCuAEAAKZCuAEAAKZCuAEAAKbi8tsv3GvXv5A5JSXFxZ0AAIDsuj5uZ+fGCv+4cHPhwgVJUlhYmIs7AQAAjrpw4YL8/f1vu8w/7t5SVqtVp06dkq+vrywWi1Nrp6SkKCwsTCdOnMiV+1blZv383Htu18/Pvef3+vm59/xePz/3nt/r5+fec7O+YRi6cOGCSpQoYXdD7az84/bcuLm5qVSpUrm6DT8/v1y9KWdu1s/Pved2/fzce36vn597z+/183Pv+b1+fu49t+rfaY/NdZxQDAAATIVwAwAATIVw40ReXl4aNWqUvLy88l39/Nx7btfPz73n9/r5uff8Xj8/957f6+fn3u9F/ez4x51QDAAAzI09NwAAwFQINwAAwFQINwAAwFQINwAAwFQIN06wadMmtW/fXiVKlJDFYtGSJUucVjsmJkb169eXr6+vgoKC1KlTJx08eNBp9WfMmKEaNWrYvmwpKipKK1eudFr9G40fP14Wi0WDBg1ySr3Ro0fLYrHYTZUqVXJK7etOnjypxx57TMWKFZOPj4+qV6+uXbt2OaV2REREpv4tFov69+/vlPoZGRkaMWKEIiMj5ePjo7Jly2rcuHHZui9Ldly4cEGDBg1SeHi4fHx81LBhQ+3cufOuat3pPWQYhkaOHKnQ0FD5+PgoOjpav/76q9PqL1q0SC1atFCxYsVksVgUHx/vtP6vXr2ql19+WdWrV1ehQoVUokQJ9e7dW6dOnXJK76NHj1alSpVUqFAhFS1aVNHR0dq+fbtTer/Zs88+K4vFonfffddp9fv27ZvpPdCqVSun9X7gwAF16NBB/v7+KlSokOrXr6/jx487pX5W71+LxaK3337bKfVTU1M1YMAAlSpVSj4+PqpSpYpmzpyZrdrZqZ+UlKS+ffuqRIkSKliwoFq1apXt91V2xqbLly+rf//+KlasmAoXLqyuXbsqKSkp2/3nBOHGCdLS0lSzZk1Nnz7d6bU3btyo/v37a9u2bYqLi9PVq1fVokULpaWlOaV+qVKlNH78eO3evVu7du3Sgw8+qI4dO+qnn35ySv3rdu7cqQ8++EA1atRwat2qVasqISHBNn3//fdOq33u3Dk1atRIBQoU0MqVK/Xzzz9r4sSJKlq0qFPq79y50673uLg4SdIjjzzilPoTJkzQjBkzNG3aNB04cEATJkzQW2+9pffee88p9Z966inFxcXps88+0759+9SiRQtFR0fr5MmTDte603vorbfe0tSpUzVz5kxt375dhQoVUsuWLXX58mWn1E9LS1Pjxo01YcIEh3u/U/2LFy9qz549GjFihPbs2aNFixbp4MGD6tChg1N6r1ChgqZNm6Z9+/bp+++/V0REhFq0aKEzZ844pf51ixcv1rZt21SiRIls1XWkfqtWrezeC19++aVTav/2229q3LixKlWqpA0bNujHH3/UiBEj5O3t7ZT6N/ackJCgjz/+WBaLRV27dnVK/SFDhmjVqlX6/PPPdeDAAQ0aNEgDBgzQ119/neP6hmGoU6dO+v3337V06VLt3btX4eHhio6Oztb4kp2xafDgwfrmm2+0YMECbdy4UadOnVKXLl2y1XuOGXAqScbixYtzrf7p06cNScbGjRtzbRtFixY1/vvf/zqt3oULF4zy5csbcXFxRpMmTYyBAwc6pe6oUaOMmjVrOqVWVl5++WWjcePGuVb/ZgMHDjTKli1rWK1Wp9Rr27at0a9fP7t5Xbp0MXr16pXj2hcvXjTc3d2NZcuW2c2vU6eO8dprr+Wo9s3vIavVaoSEhBhvv/22bd758+cNLy8v48svv8xx/RsdOXLEkGTs3bvX4brZqX/djh07DEnGsWPHnF47OTnZkGSsXbvWodq3q//HH38YJUuWNPbv32+Eh4cbkydPdrj2rer36dPH6Nix413Vu1Pt7t27G4899liOa9+q/s06duxoPPjgg06rX7VqVWPs2LF28+72PXZz/YMHDxqSjP3799vmZWRkGIGBgcaHH37ocP2bx6bz588bBQoUMBYsWGBb5sCBA4YkY+vWrQ7XdxR7bvKZ5ORkSVJAQIDTa2dkZGju3LlKS0tTVFSU0+r2799fbdu2VXR0tNNqXvfrr7+qRIkSKlOmjHr16pXt3c3Z8fXXX6tevXp65JFHFBQUpNq1a+vDDz90Wv0bXblyRZ9//rn69evntBu6NmzYUOvWrdOhQ4ckST/88IO+//57tW7dOse1//77b2VkZGT6BOzj4+PUvWeSdOTIESUmJtr9+/H399f999+vrVu3OnVb90pycrIsFouKFCni1LpXrlzRrFmz5O/vr5o1azqlptVq1eOPP65hw4apatWqTql5sw0bNigoKEgVK1bUc889p7Nnz+a4ptVq1fLly1WhQgW1bNlSQUFBuv/++5162sCNkpKStHz5cj355JNOq9mwYUN9/fXXOnnypAzD0Pr163Xo0CG1aNEix7XT09Mlye497ObmJi8vr7t6D988Nu3evVtXr161e99WqlRJpUuXvifvW8JNPmK1WjVo0CA1atRI1apVc1rdffv2qXDhwvLy8tKzzz6rxYsXq0qVKk6pPXfuXO3Zs0cxMTFOqXej+++/X3PmzNGqVas0Y8YMHTlyRP/617904cIFp9T//fffNWPGDJUvX16rV6/Wc889pxdeeEGffPKJU+rfaMmSJTp//rz69u3rtJqvvPKKHn30UVWqVEkFChRQ7dq1NWjQIPXq1SvHtX19fRUVFaVx48bp1KlTysjI0Oeff66tW7cqISHBCd3/T2JioiQpODjYbn5wcLDtufzk8uXLevnll9WjRw+n3VRw2bJlKly4sLy9vTV58mTFxcWpePHiTqk9YcIEeXh46IUXXnBKvZu1atVKn376qdatW6cJEyZo48aNat26tTIyMnJU9/Tp00pNTdX48ePVqlUrrVmzRp07d1aXLl20ceNGJ3X/P5988ol8fX2detjlvffeU5UqVVSqVCl5enqqVatWmj59uh544IEc174eNIYPH65z587pypUrmjBhgv744w+H38NZjU2JiYny9PTMFODv1fv2H3dX8Pysf//+2r9/v9M/GVesWFHx8fFKTk7WwoUL1adPH23cuDHHAefEiRMaOHCg4uLisn2M2xE37oGoUaOG7r//foWHh2v+/PlO+fRktVpVr149vfnmm5Kk2rVra//+/Zo5c6b69OmT4/o3+uijj9S6dWuHz2e4nfnz5+uLL75QbGysqlatqvj4eA0aNEglSpRwSv+fffaZ+vXrp5IlS8rd3V116tRRjx49tHv3bid0b05Xr15Vt27dZBiGZsyY4bS6zZo1U3x8vP788099+OGH6tatm7Zv366goKAc1d29e7emTJmiPXv2OG2P4s0effRR28/Vq1dXjRo1VLZsWW3YsEHNmze/67pWq1WS1LFjRw0ePFiSVKtWLW3ZskUzZ85UkyZNctb4TT7++GP16tXLqf+ve++997Rt2zZ9/fXXCg8P16ZNm9S/f3+VKFEix3vCCxQooEWLFunJJ59UQECA3N3dFR0drdatWzt80UFujU05wZ6bfGLAgAFatmyZ1q9fr1KlSjm1tqenp8qVK6e6desqJiZGNWvW1JQpU3Jcd/fu3Tp9+rTq1KkjDw8PeXh4aOPGjZo6dao8PDxy/MnsZkWKFFGFChV0+PBhp9QLDQ3NFPAqV67s1ENfknTs2DGtXbtWTz31lFPrDhs2zLb3pnr16nr88cc1ePBgp+1FK1u2rDZu3KjU1FSdOHFCO3bs0NWrV1WmTBmn1L8uJCREkjJdZZGUlGR7Lj+4HmyOHTumuLg4p+21kaRChQqpXLlyatCggT766CN5eHjoo48+ynHd7777TqdPn1bp0qVt7+Fjx47pxRdfVERERM4bz0KZMmVUvHjxHL+PixcvLg8Pj3vyHv7uu+908OBBp76HL126pFdffVWTJk1S+/btVaNGDQ0YMEDdu3fXO++845Rt1K1bV/Hx8Tp//rwSEhK0atUqnT171qH38K3GppCQEF25ckXnz5+3W/5evW8JN3mcYRgaMGCAFi9erG+//VaRkZG5vk2r1Wo7HpsTzZs31759+xQfH2+b6tWrp169eik+Pl7u7u5O6PZ/UlNT9dtvvyk0NNQp9Ro1apTp0sZDhw4pPDzcKfWvmz17toKCgtS2bVun1r148aLc3Ozf4u7u7rZPtM5SqFAhhYaG6ty5c1q9erU6duzo1PqRkZEKCQnRunXrbPNSUlK0fft2p54blpuuB5tff/1Va9euVbFixXJ1e856Dz/++OP68ccf7d7DJUqU0LBhw7R69WondJrZH3/8obNnz+b4fezp6an69evfk/fwRx99pLp16zrtPCfp2r+Zq1ev3pP3sL+/vwIDA/Xrr79q165d2XoP32lsqlu3rgoUKGD3vj148KCOHz9+T963HJZygtTUVLtPGUeOHFF8fLwCAgJUunTpHNXu37+/YmNjtXTpUvn6+tqOVfr7+8vHxydHtSVp+PDhat26tUqXLq0LFy4oNjZWGzZscMr/uHx9fTOdG1SoUCEVK1bMKecMDR06VO3bt1d4eLhOnTqlUaNGyd3dXT169MhxbenaZYwNGzbUm2++qW7dumnHjh2aNWuWZs2a5ZT60rVBaPbs2erTp488PJz7dmzfvr3eeOMNlS5dWlWrVtXevXs1adIk9evXzyn1V69eLcMwVLFiRR0+fFjDhg1TpUqV9MQTTzhc607voUGDBun//b//p/LlyysyMlIjRoxQiRIl1KlTJ6fU/+uvv3T8+HHbd89cHxBDQkKy9SnzdvVDQ0P18MMPa8+ePVq2bJkyMjJs7+OAgAB5enrede1ixYrpjTfeUIcOHRQaGqo///xT06dP18mTJ7P9lQJ3+t3cHMQKFCigkJAQVaxYMcf1AwICNGbMGHXt2lUhISH67bff9NJLL6lcuXJq2bJljnsfNmyYunfvrgceeEDNmjXTqlWr9M0332jDhg057v36/9tTUlK0YMECTZw4MVs1HanfpEkTDRs2TD4+PgoPD9fGjRv16aefatKkSU6pv2DBAgUGBqp06dLat2+fBg4cqE6dOmXrhOU7jU3+/v568sknNWTIEAUEBMjPz0//+c9/FBUVpQYNGjj4m7oLuX491j/A+vXrDUmZpj59+uS4dlZ1JRmzZ8/OcW3DMIx+/foZ4eHhhqenpxEYGGg0b97cWLNmjVNqZ8WZl4J3797dCA0NNTw9PY2SJUsa3bt3Nw4fPuyU2td98803RrVq1QwvLy+jUqVKxqxZs5xaf/Xq1YYk4+DBg06taxiGkZKSYgwcONAoXbq04e3tbZQpU8Z47bXXjPT0dKfUnzdvnlGmTBnD09PTCAkJMfr372+cP3/+rmrd6T1ktVqNESNGGMHBwYaXl5fRvHlzh35nd6o/e/bsLJ8fNWpUjutfv7w8q2n9+vU5qn3p0iWjc+fORokSJQxPT08jNDTU6NChg7Fjxw6n/W5u5uil4Lerf/HiRaNFixZGYGCgUaBAASM8PNx4+umnjcTERKf1/tFHHxnlypUzvL29jZo1axpLlixxSu/XffDBB4aPj89d/du/U/2EhASjb9++RokSJQxvb2+jYsWKxsSJE7P9dRF3qj9lyhSjVKlSRoECBYzSpUsbr7/+erb//5CdsenSpUvG888/bxQtWtQoWLCg0blzZyMhIcGRX9Fds/xfkwAAAKbAOTcAAMBUCDcAAMBUCDcAAMBUCDcAAMBUCDcAAMBUCDcAAMBUCDcAAMBUCDcA/vE2bNggi8WS6T44APInwg0AADAVwg0AADAVwg0Al7NarYqJiVFkZKR8fHxUs2ZNLVy4UNL/DhktX75cNWrUkLe3txo0aKD9+/fb1fjqq69UtWpVeXl5KSIiItONDNPT0/Xyyy8rLCxMXl5eKleunD766CO7ZXbv3q169eqpYMGCatiwYaY7SgPIHwg3AFwuJiZGn376qWbOnKmffvpJgwcP1mOPPaaNGzfalhk2bJgmTpyonTt3KjAwUO3bt9fVq1clXQsl3bp106OPPqp9+/Zp9OjRGjFihObMmWNbv3fv3vryyy81depUHThwQB988IEKFy5s18drr72miRMnateuXfLw8HDaHdQB3FvcOBOAS6WnpysgIEBr165VVFSUbf5TTz2lixcv6plnnlGzZs00d+5cde/eXZL0119/qVSpUpozZ466deumXr166cyZM1qzZo1t/ZdeeknLly/XTz/9pEOHDqlixYqKi4tTdHR0ph42bNigZs2aae3atWrevLkkacWKFWrbtq0uXbokb2/vXP4tAHAm9twAcKnDhw/r4sWLeuihh1S4cGHb9Omnn+q3336zLXdj8AkICFDFihV14MABSdKBAwfUqFEju7qNGjXSr7/+qoyMDMXHx8vd3V1NmjS5bS81atSw/RwaGipJOn36dI5fI4B7y8PVDQD4Z0tNTZUkLV++XCVLlrR7zsvLyy7g3C0fH59sLVegQAHbzxaLRdK184EA5C/suQHgUlWqVJGXl5eOHz+ucuXK2U1hYWG25bZt22b7+dy5czp06JAqV64sSapcubI2b95sV3fz5s2qUKGC3N3dVb16dVmtVrtzeACYF3tuALiUr6+vhg4dqsGDB8tqtapx48ZKTk7W5s2b5efnp/DwcEnS2LFjVaxYMQUHB+u1115T8eLF1alTJ0nSiy++qPr162vcuHHq3r27tm7dqmnTpun999+XJEVERKhPnz7q16+fpk6dqpo1a+rYsWM6ffq0unXr5qqXDiCXEG4AuNy4ceMUGBiomJgY/f777ypSpIjq1KmjV1991XZYaPz48Ro4cKB+/fVX1apVS9988408PT0lSXXq1NH8+fM1cuRIjRs3TqGhoRo7dqz69u1r28aMGTP06quv6vnnn9fZs2dVunRpvfrqq654uQByGVdLAcjTrl/JdO7cORUpUsTV7QDIBzjnBgAAmArhBgAAmAqHpQAAgKmw5wYAAJgK4QYAAJgK4QYAAJgK4QYAAJgK4QYAAJgK4QYAAJgK4QYAAJgK4QYAAJgK4QYAAJjK/weFWKwZs588EwAAAABJRU5ErkJggg==\n"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["
"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAjsAAAHHCAYAAABZbpmkAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAARX9JREFUeJzt3Xl4THf///HXZI9IQogsCLHvaqu1VeQWSmqrrVqULopWqKXaoqj1Lq2lpe1tqbuWoq1aioYSpcRWaiuKliKxJkGKNDm/P/wyXyOWmEwkOffzcV1zXfI5Z97znoljXj5nsxiGYQgAAMCknLK7AQAAgKxE2AEAAKZG2AEAAKZG2AEAAKZG2AEAAKZG2AEAAKZG2AEAAKZG2AEAAKZG2AEAAKZG2AGQLYoXL67u3btndxv39Mcff8hisWju3LnZ3QqATCLsAFls7ty5slgsslgs2rx5c7rlhmGoaNGislgsatmypc0yi8Wivn373rf+U089Za1vsVjk5+enWrVqafbs2UpNTX2oXjdu3GhT636PnGDFihVycnJSbGxsdrcCIAdzye4GgP8VHh4eWrBggRo0aGAzHh0drb/++kvu7u521y5SpIjGjRsnSTp//rzmzZunnj176siRIxo/fnyG65QvX17//e9/bcaGDh2qvHnz6p133rG7v7s5fPiwnJwy9/+tVatWqUaNGgoMDHRQVwDMiLADPCJPP/20lixZoqlTp8rF5f82vQULFqhGjRq6cOGC3bV9fX31/PPPW39+9dVXVbZsWU2fPl2jR4+Wq6trhuoEBATY1JGk8ePHq2DBgunGb5eamqqbN2/Kw8Mjwz1nJtyl+f7779WjR49M1/lfcu3aNXl5eWV3G8AjxW4s4BHp3LmzLl68qKioKOvYzZs3tXTpUj333HMOfa08efKoTp06unbtms6fPy9JSkpK0m+//ZapUJUmbffa/PnzVbFiRbm7u2vNmjWSpA8++ED16tVTgQIF5OnpqRo1amjp0qXpatx5zE7a7r4tW7ZowIAB8vf3l5eXl9q0aWN9D7fbt2+fTp06pRYtWiguLk4uLi4aOXJkuvUOHz4si8Wi6dOnS5IuXbqkgQMHqnLlysqbN698fHzUvHlz7d27167P4mHqXb9+Xe+9957KlCkjDw8PBQUFqW3btjp27Jh1ndTUVE2ZMkWVK1eWh4eH/P391axZM+3cuVPS/Y8lslgseu+996w/v/fee7JYLDp48KCee+455c+f3zqz+Ouvv6p79+4qUaKEPDw8FBgYqB49eujixYvp6p4+fVo9e/ZUcHCw3N3dFRoaqtdee003b97U8ePHZbFY9OGHH6Z73s8//yyLxaKFCxc+7McKOBRhB3hEihcvrrp169r8w7969WolJCSoU6dODn+948ePy9nZWfny5ZMkbd++XeXLl7d+6WfWjz/+qP79+6tjx46aMmWKihcvLkmaMmWKqlWrplGjRmns2LFycXFR+/bttWrVqgzVff3117V3716NGDFCr732mlasWHHX45a+//57FSpUSDVr1lRAQIAaNmyoxYsXp1vvq6++krOzs9q3by/p1ueybNkytWzZUpMnT9agQYO0b98+NWzYUGfOnHnozyGj9VJSUtSyZUuNHDlSNWrU0KRJk9SvXz8lJCRo//791vV69uypyMhIFS1aVBMmTNBbb70lDw8Pbdu27aF7S9O+fXslJSVp7NixevnllyVJUVFROn78uF588UVNmzZNnTp10qJFi/T000/LMAzrc8+cOaPHH39cixYtUseOHTV16lS98MILio6OVlJSkkqUKKH69etr/vz56V53/vz58vb2VqtWrezuHXAIA0CWmjNnjiHJ2LFjhzF9+nTD29vbSEpKMgzDMNq3b280atTIMAzDKFasmNGiRQub50oy+vTpc9/6DRs2NMqVK2ecP3/eOH/+vHHo0CHjjTfeMCQZERER1vU2bNhgSDJGjBjxUP1XrFjRaNiwYbq+nJycjAMHDqRbP+29pbl586ZRqVIlo3HjxjbjxYoVM7p162b9Oe1zCgsLM1JTU63j/fv3N5ydnY34+Hib5z/xxBM2z//0008NSca+ffts1qtQoYLNa1+/ft1ISUmxWefEiROGu7u7MWrUKJsxScacOXPSvcfbZbTe7NmzDUnG5MmT09VIe78//vijIcl444037rnO/fq68/c7YsQIQ5LRuXPndOve+XsyDMNYuHChIcnYtGmTdaxr166Gk5OTsWPHjnv2lPbZHzp0yLrs5s2bRsGCBW1+R0B2YWYHeIQ6dOigv//+WytXrtSVK1e0cuVKh+zC+u233+Tv7y9/f3+VL19e06ZNU4sWLTR79mzrOk899ZQMw7DZzZEZDRs2VIUKFdKNe3p6Wv98+fJlJSQk6IknntDu3bszVPeVV16xOdvriSeeUEpKiv7880/rWHx8vLZu3aoWLVpYx9q2bSsXFxd99dVX1rH9+/fr4MGD6tixo3XM3d3demB0SkqKLl68qLx586ps2bIZ7vF2Ga339ddfq2DBgnr99dfT1Uh7v19//bUsFotGjBhxz3Xs0atXr3Rjt/+erl+/rgsXLqhOnTqSZO07NTVVy5YtU0REhGrWrHnPnjp06CAPDw+b2Z21a9fqwoUL9z3WC3hUCDvAI+Tv76+wsDAtWLBA33zzjVJSUvTss89mum7x4sUVFRWldevWafPmzYqNjdXKlStVsGBBB3R9d6GhoXcdX7lyperUqSMPDw/5+fnJ399fM2bMUEJCQobqhoSE2PycP39+SbeCU5q1a9dKkpo2bWodK1iwoJo0aWKzK+urr76Si4uL2rZtax1LTU3Vhx9+qNKlS8vd3V0FCxaUv7+/fv311wz3eLuM1jt27JjKli1rc3D6nY4dO6bg4GD5+fk9dB/3c7ff1aVLl9SvXz8FBATI09NT/v7+1vXS+j5//rwSExNVqVKl+9bPly+fIiIitGDBAuvY/PnzVbhwYTVu3NiB7wSwD2djAY/Yc889p5dfflmxsbFq3ry59ZiazPDy8lJYWFjmm3sIt88MpPnpp5/0zDPP6Mknn9Qnn3yioKAgubq6as6cOTZfhPfj7Ox813HjtuNIvv/+e9WvX1++vr4263Tq1Ekvvvii9uzZo8cee0yLFy9WkyZNbELf2LFjNWzYMPXo0UOjR4+Wn5+fnJycFBkZ+dDXJcqKeg9yrxmelJSUez7nbr+rDh066Oeff9agQYP02GOPKW/evEpNTVWzZs3s6rtr165asmSJfv75Z1WuXFnLly9X7969M315AcARCDvAI9amTRu9+uqr2rZtm80uFzP4+uuv5eHhobVr19qcWj5nzhyHvYZhGFqzZo0GDhyYblnr1q316quvWj/XI0eOaOjQoTbrLF26VI0aNdKsWbNsxuPj4+2aCctovZIlSyomJkbJycn3vBRAyZIltXbtWl26dOmesztpM13x8fE247fv5nuQy5cva/369Ro5cqSGDx9uHT969KjNev7+/vLx8bE5gPpemjVrJn9/f82fP1+1a9dWUlKSXnjhhQz3BGQlIjfwiOXNm1czZszQe++9p4iIiEf2uo489fxenJ2dZbFYbGYZ/vjjDy1btsxhr7Fjxw6dO3fO5nidNPny5VN4eLgWL16sRYsWyc3NTa1bt07X4+2zRJK0ZMkSnT592q5+MlqvXbt2unDhwl3Phkt7frt27WQYxl1PoU9bx8fHRwULFtSmTZtsln/yyScP1fPtNdN89NFHNj87OTmpdevWWrFihfXU97v1JEkuLi7q3LmzFi9erLlz56py5cqqUqVKhnsCshIzO0A26NatW4bX3blzp95///1040899VS6qzHfz/bt29WoUSONGDHCYQcp36lFixaaPHmymjVrpueee07nzp3Txx9/rFKlSunXX391yGusWrVKxYsXv+vB0ZLUsWNHPf/88/rkk08UHh6ebjdhy5YtNWrUKL344ouqV6+e9u3bp/nz56tEiRJ29ZPRel27dtW8efM0YMAAbd++XU888YSuXbumdevWqXfv3mrVqpUaNWqkF154QVOnTtXRo0etu5R++uknNWrUyHoK/ksvvaTx48frpZdeUs2aNbVp0yYdOXIkwz37+PjoySef1MSJE5WcnKzChQvrhx9+0IkTJ9KtO3bsWP3www9q2LChXnnlFZUvX15nz57VkiVLtHnzZpvPt2vXrpo6dao2bNigCRMm2PV5AlmBsAPkcDExMYqJiUk3Pnr06IcKO49C48aNNWvWLI0fP16RkZEKDQ3VhAkT9Mcffzgs7Hz//fd6+umn77n8mWeekaenp65cuWJzFlaat99+W9euXdOCBQv01VdfqXr16lq1apXeeustu/rJaD1nZ2d9//33GjNmjBYsWKCvv/5aBQoUUIMGDVS5cmXrenPmzFGVKlU0a9YsDRo0SL6+vqpZs6bq1atnXWf48OE6f/68li5dqsWLF6t58+ZavXq1ChUqlOG+FyxYoNdff10ff/yxDMNQ06ZNtXr1agUHB9usV7hwYcXExGjYsGGaP3++EhMTVbhwYTVv3lx58uSxWbdGjRqqWLGiDh06pC5dujzMxwhkKYtx5zwmAORQcXFxCgoK0sqVK+8beJB9qlWrJj8/P61fvz67WwGsOGYHQK6RkJCg4cOHq1GjRtndCu5i586d2rNnj7p27ZrdrQA2mNkBAGTK/v37tWvXLk2aNEkXLlzQ8ePHH+qmsEBWY2YHAJApS5cu1Ysvvqjk5GQtXLiQoIMch5kdAABgaszsAAAAUyPsAAAAU+M6O7p1I78zZ87I29s7U3cWBgAAj45hGLpy5YqCg4Pvex82wo6kM2fOqGjRotndBgAAsMOpU6dUpEiRey4n7Ejy9vaWdOvD8vHxyeZuAABARiQmJqpo0aLW7/F7IexI1l1XPj4+hB0AAHKZBx2CwgHKAADA1Ag7AADA1Ag7AADA1Ag7AADA1Ag7AADA1Ag7AADA1Ag7AADA1Ag7AADA1Ag7AADA1Ag7AADA1LI17GzatEkREREKDg6WxWLRsmXLbJYbhqHhw4crKChInp6eCgsL09GjR23WuXTpkrp06SIfHx/ly5dPPXv21NWrVx/huwAAADlZtoada9euqWrVqvr444/vunzixImaOnWqZs6cqZiYGHl5eSk8PFzXr1+3rtOlSxcdOHBAUVFRWrlypTZt2qRXXnnlUb0FAACQw1kMwzCyuwnp1k28vv32W7Vu3VrSrVmd4OBgvfnmmxo4cKAkKSEhQQEBAZo7d646deqkQ4cOqUKFCtqxY4dq1qwpSVqzZo2efvpp/fXXXwoODs7QaycmJsrX11cJCQkOuxGoYRj6OznFIbUAAMjtPF2dH3jDzoeV0e/vHHvX8xMnTig2NlZhYWHWMV9fX9WuXVtbt25Vp06dtHXrVuXLl88adCQpLCxMTk5OiomJUZs2be5a+8aNG7px44b158TERIf3/3dyiioMX+vwugAA5EYHR4Urj1v2xI4ce4BybGysJCkgIMBmPCAgwLosNjZWhQoVslnu4uIiPz8/6zp3M27cOPn6+lofRYsWdXD3AAAgp8ixMztZaejQoRowYID158TERIcHHk9XZx0cFe7QmgAA5Faers7Z9to5NuwEBgZKkuLi4hQUFGQdj4uL02OPPWZd59y5czbP++eff3Tp0iXr8+/G3d1d7u7ujm/6NhaLJdum6wAAwP/JsbuxQkNDFRgYqPXr11vHEhMTFRMTo7p160qS6tatq/j4eO3atcu6zo8//qjU1FTVrl37kfcMAABynmyderh69ap+//13688nTpzQnj175Ofnp5CQEEVGRur9999X6dKlFRoaqmHDhik4ONh6xlb58uXVrFkzvfzyy5o5c6aSk5PVt29fderUKcNnYgEAAHPL1rCzc+dONWrUyPpz2nE03bp109y5czV48GBdu3ZNr7zyiuLj49WgQQOtWbNGHh4e1ufMnz9fffv2VZMmTeTk5KR27dpp6tSpj/y9AACAnCnHXGcnO2XFdXYAAEDWyuj3d449ZgcAAMARCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUCDsAAMDUcnTYSUlJ0bBhwxQaGipPT0+VLFlSo0ePlmEY1nUMw9Dw4cMVFBQkT09PhYWF6ejRo9nYNQAAyElydNiZMGGCZsyYoenTp+vQoUOaMGGCJk6cqGnTplnXmThxoqZOnaqZM2cqJiZGXl5eCg8P1/Xr17OxcwAAkFNYjNunSXKYli1bKiAgQLNmzbKOtWvXTp6envryyy9lGIaCg4P15ptvauDAgZKkhIQEBQQEaO7cuerUqVOGXicxMVG+vr5KSEiQj49PlrwXAADgWBn9/s7RMzv16tXT+vXrdeTIEUnS3r17tXnzZjVv3lySdOLECcXGxiosLMz6HF9fX9WuXVtbt269Z90bN24oMTHR5gEAAMzJJbsbuJ+33npLiYmJKleunJydnZWSkqIxY8aoS5cukqTY2FhJUkBAgM3zAgICrMvuZty4cRo5cmTWNQ4AAHKMHD2zs3jxYs2fP18LFizQ7t279cUXX+iDDz7QF198kam6Q4cOVUJCgvVx6tQpB3UMAABymhw9szNo0CC99dZb1mNvKleurD///FPjxo1Tt27dFBgYKEmKi4tTUFCQ9XlxcXF67LHH7lnX3d1d7u7uWdo7AADIGXL0zE5SUpKcnGxbdHZ2VmpqqiQpNDRUgYGBWr9+vXV5YmKiYmJiVLdu3UfaKwAAyJly9MxORESExowZo5CQEFWsWFG//PKLJk+erB49ekiSLBaLIiMj9f7776t06dIKDQ3VsGHDFBwcrNatW2dv8wAAIEfI0WFn2rRpGjZsmHr37q1z584pODhYr776qoYPH25dZ/Dgwbp27ZpeeeUVxcfHq0GDBlqzZo08PDyysXMAAJBT5Ojr7DwqXGcHAIDcxxTX2QEAAMgswg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1wg4AADA1u8LOhg0bHN0HAABAlrAr7DRr1kwlS5bU+++/r1OnTjm6JwAAAIexK+ycPn1affv21dKlS1WiRAmFh4dr8eLFunnzpqP7AwAAyBS7wk7BggXVv39/7dmzRzExMSpTpox69+6t4OBgvfHGG9q7d6+j+wQAALBLpg9Qrl69uoYOHaq+ffvq6tWrmj17tmrUqKEnnnhCBw4ccESPAAAAdnOx94nJycn67rvvNHv2bEVFRalmzZqaPn26OnfurPPnz+vdd99V+/btdfDgQUf2CwDAQ0tJSVFycnJ2t4GH5OrqKmdn50zXsRiGYTzsk15//XUtXLhQhmHohRde0EsvvaRKlSrZrBMbG6vg4GClpqZmusmslpiYKF9fXyUkJMjHxye72wEAOIhhGIqNjVV8fHx2twI75cuXT4GBgbJYLOmWZfT7266ZnYMHD2ratGlq27at3N3d77pOwYIFOUUdAJCt0oJOoUKFlCdPnrt+YSJnMgxDSUlJOnfunCQpKCjI7lp2hZ3169c/uLCLixo2bGhPeRunT5/WkCFDtHr1aiUlJalUqVKaM2eOatasKenWhzFixAh9/vnnio+PV/369TVjxgyVLl06068NAMi9UlJSrEGnQIEC2d0O7ODp6SlJOnfunAoVKmT3Li27DlAeN26cZs+enW589uzZmjBhgl2N3M3ly5dVv359ubq6avXq1Tp48KAmTZqk/PnzW9eZOHGipk6dqpkzZyomJkZeXl4KDw/X9evXHdYHACD3STtGJ0+ePNncCTIj7feXmWOu7Ao7n376qcqVK5duvGLFipo5c6bdzdxpwoQJKlq0qObMmaPHH39coaGhatq0qUqWLCnp1qzORx99pHfffVetWrVSlSpVNG/ePJ05c0bLli1zWB8AgNyLXVe5myN+f3aFndjY2LvuO/P399fZs2cz3VSa5cuXq2bNmmrfvr0KFSqkatWq6fPPP7cuP3HihGJjYxUWFmYd8/X1Ve3atbV169Z71r1x44YSExNtHgAAwJzsCjtFixbVli1b0o1v2bJFwcHBmW4qzfHjx63H36xdu1avvfaa3njjDX3xxReSboUuSQoICLB5XkBAgHXZ3YwbN06+vr7WR9GiRR3WMwAAOUnx4sX10UcfZXcb2cquA5RffvllRUZGKjk5WY0bN5Z066DlwYMH680333RYc6mpqapZs6bGjh0rSapWrZr279+vmTNnqlu3bnbXHTp0qAYMGGD9OTExkcADAMgxnnrqKT322GMOCSk7duyQl5dX5pvKxewKO4MGDdLFixfVu3dv6/2wPDw8NGTIEA0dOtRhzQUFBalChQo2Y+XLl9fXX38tSQoMDJQkxcXF2exWi4uL02OPPXbPuu7u7vc8ZR4AgJzOMAylpKTIxeXBX+P+/v6PoKOcza7dWBaLRRMmTND58+e1bds27d27V5cuXdLw4cMd2lz9+vV1+PBhm7EjR46oWLFikqTQ0FAFBgbanAqfmJiomJgY1a1b16G9AAByP8MwlHTzn2x5ZPQavt27d1d0dLSmTJkii8Uii8WiuXPnymKxaPXq1apRo4bc3d21efNmHTt2TK1atVJAQIDy5s2rWrVqad26dTb17tyNZbFY9J///Edt2rRRnjx5VLp0aS1fvjxDvaWkpKhnz54KDQ2Vp6enypYtqylTpqRbb/bs2apYsaLc3d0VFBSkvn37WpfFx8fr1VdfVUBAgDw8PFSpUiWtXLkyQ69vL7tvFyHJ+sFmlf79+6tevXoaO3asOnTooO3bt+uzzz7TZ599JunWLywyMlLvv/++SpcurdDQUA0bNkzBwcFq3bp1lvUFAMid/k5OUYXha7PltQ+OClcetwd/7U6ZMkVHjhxRpUqVNGrUKEmy3mvyrbfe0gcffKASJUoof/78OnXqlJ5++mmNGTNG7u7umjdvniIiInT48GGFhITc8zVGjhypiRMn6t///remTZumLl266M8//5Sfn999e0tNTVWRIkW0ZMkSFShQQD///LNeeeUVBQUFqUOHDpKkGTNmaMCAARo/fryaN2+uhIQE63G+qampat68ua5cuaIvv/xSJUuW1MGDBx1yS4j7sTvs7Ny5U4sXL9bJkyetu7LSfPPNN5luTJJq1aqlb7/9VkOHDtWoUaMUGhqqjz76SF26dLGuM3jwYF27dk2vvPKK4uPj1aBBA61Zs0YeHh4O6QEAgEfJ19dXbm5uypMnj/Vwjd9++02SNGrUKP3rX/+yruvn56eqVatafx49erS+/fZbLV++3GY25U7du3dX586dJUljx47V1KlTtX37djVr1uy+vbm6umrkyJHWn0NDQ7V161YtXrzYGnbef/99vfnmm+rXr591vbSJkXXr1mn79u06dOiQypQpI0kqUaLEgz+UTLIr7CxatEhdu3ZVeHi4fvjhBzVt2lRHjhxRXFyc2rRp49AGW7ZsqZYtW95zucVi0ahRo6zpFwCAe/F0ddbBUeHZ9tqZlXb3gDRXr17Ve++9p1WrVuns2bP6559/9Pfff+vkyZP3rVOlShXrn728vOTj42O9LcODfPzxx5o9e7ZOnjypv//+Wzdv3rQeJ3vu3DmdOXNGTZo0uetz9+zZoyJFiliDzqNiV9gZO3asPvzwQ/Xp00fe3t6aMmWKQkND9eqrr2bq3hUAAGQli8WSoV1JOdWdZ1UNHDhQUVFR+uCDD1SqVCl5enrq2WefTbfH5U6urq42P1sslgzduHvRokUaOHCgJk2apLp168rb21v//ve/FRMTI+n/bu9wLw9anlXsOkD52LFjatGihSTJzc1N165dk8ViUf/+/a3H0wAAAPu4ubkpJSXlgett2bJF3bt3V5s2bVS5cmUFBgbqjz/+yLK+tmzZonr16ql3796qVq2aSpUqpWPHjlmXe3t7q3jx4ve8h2aVKlX0119/6ciRI1nW493YFXby58+vK1euSJIKFy6s/fv3S7p1hHVSUpLjugMA4H9Q8eLFFRMToz/++EMXLly456xL6dKl9c0332jPnj3au3evnnvuuQzN0NirdOnS2rlzp9auXasjR45o2LBh2rFjh8067733niZNmqSpU6fq6NGj2r17t6ZNmyZJatiwoZ588km1a9dOUVFROnHihFavXq01a9ZkWc+SnWHnySefVFRUlCSpffv26tevn15++WV17tz5nvvpAABAxgwcOFDOzs6qUKGC/P3973kMzuTJk5U/f37Vq1dPERERCg8PV/Xq1bOsr1dffVVt27ZVx44dVbt2bes1927XrVs3ffTRR/rkk09UsWJFtWzZUkePHrUu//rrr1WrVi117txZFSpU0ODBgzM0i5UZFiOjJ/7f5tKlS7p+/bqCg4OVmpqqiRMn6ueff1bp0qX17rvv2tyVPDdITEyUr6+vEhIS5OPjk93tAAAc4Pr16zpx4oRCQ0M5QzcXu9/vMaPf3w99lNY///yjlStXKjz81tHsTk5Oeuuttx62DAAAwCPx0LuxXFxc1KtXL12/fj0r+gEAANmkV69eyps3710fvXr1yu727GbX+XePP/649uzZY71tAwAAyP1GjRqlgQMH3nVZbj7Mw66w07t3bw0YMECnTp1SjRo10p33f/vFigAAQO5QqFAhFSpUKLvbcDi7wk6nTp0kSW+88YZ1zGKxyDAMWSyWLD+qGgAAIKPsCjsnTpxwdB8AAABZwq6ww7E6AAAgt7Ar7MybN+++y7t27WpXMwAAAI5mV9i5/bbtkpScnKykpCTrLekJOwAAIKew63YRly9ftnlcvXpVhw8fVoMGDbRw4UJH9wgAAB5C8eLF9dFHH2V3GzmGXWHnbkqXLq3x48enm/UBAADITg4LO9KtqyufOXPGkSUBAAAyxa6ws3z5cpvHd999p5kzZ+r5559X/fr1Hd0jAAD/Mz777DPrjbZv16pVK/Xo0UPHjh1Tq1atFBAQoLx586pWrVpat26d3a83efJkVa5cWV5eXipatKh69+6tq1ev2qyzZcsWPfXUU8qTJ4/y58+v8PBwXb58WZKsNwQvVaqU3N3dFRISojFjxtjdT1aw6wDl1q1b2/xssVjk7++vxo0ba9KkSY7oCwAAxzMMKTkpe17bNY9ksTxwtfbt2+v111/Xhg0b1KRJE0nSpUuXtGbNGn3//fe6evWqnn76aY0ZM0bu7u6aN2+eIiIidPjwYYWEhDx0W05OTpo6dapCQ0N1/Phx9e7dW4MHD9Ynn3wiSdqzZ4+aNGmiHj16aMqUKXJxcdGGDRusFxAeOnSoPv/8c3344Ydq0KCBzp49q99+++2h+8hKFsMwjOxuIrtl9BbxAIDc4/r16zpx4oRCQ0Pl4eFxa/DmNWlscPY09PYZyc3rwevp1qRCgQIFNGvWLEm3ZntGjhypU6dOyckp/U6ZSpUqqVevXurbt6+kWwcoR0ZGKjIy8qHbXLp0qXr16qULFy5Ikp577jmdPHlSmzdvTrfulStX5O/vr+nTp+ull1566NfKiLv+Hv+/jH5/O/SYHQAAkHldunTR119/rRs3bkiS5s+fr06dOsnJyUlXr17VwIEDVb58eeXLl0958+bVoUOHdPLkSbtea926dWrSpIkKFy4sb29vvfDCC7p48aKSkm7NgKXN7NzNoUOHdOPGjXsuzyns2o3Vrl07Pf744xoyZIjN+MSJE7Vjxw4tWbLEIc0BAOBQrnluzbBk12tnUEREhAzD0KpVq1SrVi399NNP+vDDDyVJAwcOVFRUlD744AOVKlVKnp6eevbZZ3Xz5s2HbumPP/5Qy5Yt9dprr2nMmDHy8/PT5s2b1bNnT928eVN58uSRp6fnPZ9/v2U5iV0zO5s2bdLTTz+dbrx58+batGlTppsCACBLWCy3diVlxyMDx+uk8fDwUNu2bTV//nwtXLhQZcuWVfXq1SXdOli4e/fuatOmjSpXrqzAwED98ccfdn0cu3btUmpqqiZNmqQ6deqoTJky6c6qrlKlitavX3/X55cuXVqenp73XJ5T2DWzc/XqVbm5uaUbd3V1VWJiYqabAgDgf12XLl3UsmVLHThwQM8//7x1vHTp0vrmm28UEREhi8WiYcOGpTtzK6NKlSql5ORkTZs2TREREdqyZYtmzpxps87QoUNVuXJl9e7dW7169ZKbm5s2bNig9u3bq2DBghoyZIgGDx4sNzc31a9fX+fPn9eBAwfUs2fPTL1/R7JrZqdy5cr66quv0o0vWrRIFSpUyHRTAAD8r2vcuLH8/Px0+PBhPffcc9bxyZMnK3/+/KpXr54iIiIUHh5unfV5WFWrVtXkyZM1YcIEVapUSfPnz9e4ceNs1ilTpox++OEH7d27V48//rjq1q2r7777Ti4ut+ZLhg0bpjfffFPDhw9X+fLl1bFjR507d87+N54F7Doba8WKFWrbtq2ee+45NW7cWJK0fv16LVy4UEuWLEl3anpOx9lYAGA+9zuLB7mHI87Gsms3VkREhJYtW6axY8dq6dKl8vT0VJUqVbRu3To1bNjQnpIAAABZwq6wI0ktWrRQixYtHNkLAABwoPnz5+vVV1+967JixYrpwIEDj7ij7GFX2NmxY4dSU1NVu3Ztm/GYmBg5OzurZs2aDmkOAADY75lnnkn3XZ3G1dX1EXeTfewKO3369NHgwYPTfYCnT5/WhAkTFBMT45DmAACA/by9veXt7Z3dbWQ7u87GOnjw4F2P/K5WrZoOHjyY6aYAAHAU7oqUuzni92dX2HF3d1dcXFy68bNnz1pPRQMAIDul7aZJu+0Bcqe0319mdrvZlUyaNm2qoUOH6rvvvpOvr68kKT4+Xm+//bb+9a9/2d0MAACO4uzsrHz58lmv+ZInTx5ZHuIqxshehmEoKSlJ586dU758+eTs7Gx3LbvCzgcffKAnn3xSxYoVU7Vq1STdulFYQECA/vvf/9rdDAAAjhQYGChJOe4id8i4fPnyWX+P9rLrooKSdO3aNc2fP1979+61Xmenc+fOufLobi4qCADmlpKSouTk5OxuAw/J1dX1vjM6WXpRQUny8vJSgwYNFBISYr3T6urVqyXdOtUNAICcwtnZOVO7QZC72RV2jh8/rjZt2mjfvn2yWCwyDMNmP2hKSorDGgQAAMgMu87G6tevn0JDQ3Xu3DnlyZNH+/fvV3R0tGrWrKmNGzc6uEUAAAD72TWzs3XrVv34448qWLCgnJyc5OzsrAYNGmjcuHF644039Msvvzi6TwAAALvYNbOTkpJivSJjwYIFdebMGUm37rNx+PBhx3UHAACQSXbN7FSqVEl79+5VaGioateurYkTJ8rNzU2fffaZSpQo4egeAQAA7GZX2Hn33Xd17do1SdKoUaPUsmVLPfHEEypQoIC++uorhzYIAACQGXZfZ+dOly5dUv78+XPl1Sm5zg4AALlPll9n505+fn6OKgUAAOAwdh2gDAAAkFsQdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKnlqrAzfvx4WSwWRUZGWseuX7+uPn36qECBAsqbN6/atWunuLi47GsSAADkKLkm7OzYsUOffvqpqlSpYjPev39/rVixQkuWLFF0dLTOnDmjtm3bZlOXAAAgp8kVYefq1avq0qWLPv/8c+XPn986npCQoFmzZmny5Mlq3LixatSooTlz5ujnn3/Wtm3bsrFjAACQU+SKsNOnTx+1aNFCYWFhNuO7du1ScnKyzXi5cuUUEhKirVu33rPejRs3lJiYaPMAAADm5JLdDTzIokWLtHv3bu3YsSPdstjYWLm5uSlfvnw24wEBAYqNjb1nzXHjxmnkyJGObhUAAORAOXpm59SpU+rXr5/mz58vDw8Ph9UdOnSoEhISrI9Tp045rDYAAMhZcnTY2bVrl86dO6fq1avLxcVFLi4uio6O1tSpU+Xi4qKAgADdvHlT8fHxNs+Li4tTYGDgPeu6u7vLx8fH5gEAAMwpR+/GatKkifbt22cz9uKLL6pcuXIaMmSIihYtKldXV61fv17t2rWTJB0+fFgnT55U3bp1s6NlAACQw+TosOPt7a1KlSrZjHl5ealAgQLW8Z49e2rAgAHy8/OTj4+PXn/9ddWtW1d16tTJjpYBAEAOk6PDTkZ8+OGHcnJyUrt27XTjxg2Fh4frk08+ye62AABADmExDMPI7iayW2Jionx9fZWQkMDxOwAA5BIZ/f7O0QcoAwAAZBZhBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmBphBwAAmFqODjvjxo1TrVq15O3trUKFCql169Y6fPiwzTrXr19Xnz59VKBAAeXNm1ft2rVTXFxcNnUMAABymhwddqKjo9WnTx9t27ZNUVFRSk5OVtOmTXXt2jXrOv3799eKFSu0ZMkSRUdH68yZM2rbtm02dg0AAHISi2EYRnY3kVHnz59XoUKFFB0drSeffFIJCQny9/fXggUL9Oyzz0qSfvvtN5UvX15bt25VnTp1MlQ3MTFRvr6+SkhIkI+PT1a+BQAA4CAZ/f7O0TM7d0pISJAk+fn5SZJ27dql5ORkhYWFWdcpV66cQkJCtHXr1nvWuXHjhhITE20eAADAnHJN2ElNTVVkZKTq16+vSpUqSZJiY2Pl5uamfPny2awbEBCg2NjYe9YaN26cfH19rY+iRYtmZesAACAb5Zqw06dPH+3fv1+LFi3KdK2hQ4cqISHB+jh16pQDOgQAADmRS3Y3kBF9+/bVypUrtWnTJhUpUsQ6HhgYqJs3byo+Pt5mdicuLk6BgYH3rOfu7i53d/esbBkAAOQQOXpmxzAM9e3bV99++61+/PFHhYaG2iyvUaOGXF1dtX79euvY4cOHdfLkSdWtW/dRtwsAAHKgHD2z06dPHy1YsEDfffedvL29rcfh+Pr6ytPTU76+vurZs6cGDBggPz8/+fj46PXXX1fdunUzfCYWAAAwtxx96rnFYrnr+Jw5c9S9e3dJty4q+Oabb2rhwoW6ceOGwsPD9cknn9x3N9adOPUcAIDcJ6Pf3zk67DwqhB0AAHIfU15nBwAA4GERdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKkRdgAAgKm5ZHcDpmUYUnJSdncBAEDO4JpHsliy5aUJO1klOUkaG5zdXQAAkDO8fUZy88qWl2Y3FgAAMDVmdrKKa55bKRYAANz6XswmhJ2sYrFk23QdAAD4P+zGAgAApkbYAQAApkbYAQAApkbYAQAApkbYAQAApkbYAQAApkbYAQAApkbYAQAApkbYAQAApkbYAQAApkbYAQAApkbYAQAApkbYAQAApsZdzyUZhiFJSkxMzOZOAABARqV9b6d9j98LYUfSlStXJElFixbN5k4AAMDDunLlinx9fe+53GI8KA79D0hNTdWZM2fk7e0ti8XisLqJiYkqWrSoTp06JR8fH4fVNUP93Nx7bq+fm3vP7fVzc+9ZXT83957b6+fm3g3D0JUrVxQcHCwnp3sfmcPMjiQnJycVKVIky+r7+PhkyV8gM9TPzb3n9vq5uffcXj83957V9XNz77m9fm7t/X4zOmk4QBkAAJgaYQcAAJgaYScLubu7a8SIEXJ3d6f+I6xN/eyrTf3sq53b6+fm3nN7/dzce0ZxgDIAADA1ZnYAAICpEXYAAICpEXYAAICpEXYAAICpEXaywKZNmxQREaHg4GBZLBYtW7bMofXHjRunWrVqydvbW4UKFVLr1q11+PBhh9SeMWOGqlSpYr34U926dbV69WqH1L6b8ePHy2KxKDIy0iH13nvvPVksFptHuXLlHFJbkk6fPq3nn39eBQoUkKenpypXrqydO3c6pHbx4sXT9W6xWNSnTx+H1E9JSdGwYcMUGhoqT09PlSxZUqNHj37gPWUy6sqVK4qMjFSxYsXk6empevXqaceOHXbXe9B2ZBiGhg8frqCgIHl6eiosLExHjx51SO1vvvlGTZs2VYECBWSxWLRnzx6H9Z6cnKwhQ4aocuXK8vLyUnBwsLp27aozZ844pL50azsoV66cvLy8lD9/foWFhSkmJsYhtW/Xq1cvWSwWffTRRw7rvXv37um2gWbNmjmsviQdOnRIzzzzjHx9feXl5aVatWrp5MmTDql/t23YYrHo3//+d6ZrX716VX379lWRIkXk6empChUqaObMmRnqOyP14+Li1L17dwUHBytPnjxq1qxZhrepjHwvXb9+XX369FGBAgWUN29etWvXTnFxcRnuPzMIO1ng2rVrqlq1qj7++OMsqR8dHa0+ffpo27ZtioqKUnJyspo2bapr165lunaRIkU0fvx47dq1Szt37lTjxo3VqlUrHThwwAGd29qxY4c+/fRTValSxaF1K1asqLNnz1ofmzdvdkjdy5cvq379+nJ1ddXq1at18OBBTZo0Sfnz53dI/R07dtj0HRUVJUlq3769Q+pPmDBBM2bM0PTp03Xo0CFNmDBBEydO1LRp0xxS/6WXXlJUVJT++9//at++fWratKnCwsJ0+vRpu+o9aDuaOHGipk6dqpkzZyomJkZeXl4KDw/X9evXM1372rVratCggSZMmODw3pOSkrR7924NGzZMu3fv1jfffKPDhw/rmWeecUh9SSpTpoymT5+uffv2afPmzSpevLiaNm2q8+fPZ7p2mm+//Vbbtm1TcHBwhvvOaP1mzZrZbAsLFy50WP1jx46pQYMGKleunDZu3Khff/1Vw4YNk4eHh0Pq39732bNnNXv2bFksFrVr1y7TtQcMGKA1a9boyy+/1KFDhxQZGam+fftq+fLlme7dMAy1bt1ax48f13fffadffvlFxYoVU1hYWIa+WzLyvdS/f3+tWLFCS5YsUXR0tM6cOaO2bdtmqPdMM5ClJBnffvttlr7GuXPnDElGdHR0ltTPnz+/8Z///MehNa9cuWKULl3aiIqKMho2bGj069fPIXVHjBhhVK1a1SG17jRkyBCjQYMGWVL7bvr162eULFnSSE1NdUi9Fi1aGD169LAZa9u2rdGlS5dM105KSjKcnZ2NlStX2oxXr17deOeddzJd/87tKDU11QgMDDT+/e9/W8fi4+MNd3d3Y+HChZmqfbsTJ04YkoxffvnFjq4fXD/N9u3bDUnGn3/+mSX1ExISDEnGunXrHFL7r7/+MgoXLmzs37/fKFasmPHhhx8+VN371e/WrZvRqlUru+plpH7Hjh2N559/Psvq36lVq1ZG48aNHVK7YsWKxqhRo2zG7N3G7qx/+PBhQ5Kxf/9+61hKSorh7+9vfP755w9d/87vpfj4eMPV1dVYsmSJdZ1Dhw4ZkoytW7c+dP2HxcyOCSQkJEiS/Pz8HFo3JSVFixYt0rVr11S3bl2H1u7Tp49atGihsLAwh9aVpKNHjyo4OFglSpRQly5dMjw9/SDLly9XzZo11b59exUqVEjVqlXT559/7pDad7p586a+/PJL9ejRw2E3p61Xr57Wr1+vI0eOSJL27t2rzZs3q3nz5pmu/c8//yglJSXd/449PT0dNrN2uxMnTig2Ntbm74+vr69q166trVu3Ovz1slpCQoIsFovy5cvn8No3b97UZ599Jl9fX1WtWjXT9VJTU/XCCy9o0KBBqlixogM6TG/jxo0qVKiQypYtq9dee00XL150SN3U1FStWrVKZcqUUXh4uAoVKqTatWs7/FCDNHFxcVq1apV69uzpkHr16tXT8uXLdfr0aRmGoQ0bNujIkSNq2rRppmvfuHFDkmy2YScnJ7m7u9u1Dd/5vbRr1y4lJyfbbLPlypVTSEjII9lmCTu5XGpqqiIjI1W/fn1VqlTJITX37dunvHnzyt3dXb169dK3336rChUqOKS2JC1atEi7d+/WuHHjHFYzTe3atTV37lytWbNGM2bM0IkTJ/TEE0/oypUrma59/PhxzZgxQ6VLl9batWv12muv6Y033tAXX3zhgM5tLVu2TPHx8erevbvDar711lvq1KmTypUrJ1dXV1WrVk2RkZHq0qVLpmt7e3urbt26Gj16tM6cOaOUlBR9+eWX2rp1q86ePeuA7m3FxsZKkgICAmzGAwICrMtyi+vXr2vIkCHq3LmzQ2+SuHLlSuXNm1ceHh768MMPFRUVpYIFC2a67oQJE+Ti4qI33njDAV2m16xZM82bN0/r16/XhAkTFB0drebNmyslJSXTtc+dO6erV69q/PjxatasmX744Qe1adNGbdu2VXR0tAO6t/XFF1/I29vbYbtqpk2bpgoVKqhIkSJyc3NTs2bN9PHHH+vJJ5/MdO204DF06FBdvnxZN2/e1IQJE/TXX3899DZ8t++l2NhYubm5pQv0j2qb5a7nuVyfPn20f/9+h/7vuWzZstqzZ48SEhK0dOlSdevWTdHR0Q4JPKdOnVK/fv0UFRWV4X3kD+P2WYoqVaqodu3aKlasmBYvXpzp/12lpqaqZs2aGjt2rCSpWrVq2r9/v2bOnKlu3bplqvadZs2apebNmz/08RD3s3jxYs2fP18LFixQxYoVtWfPHkVGRio4ONgh/f/3v/9Vjx49VLhwYTk7O6t69erq3Lmzdu3a5YDuzSk5OVkdOnSQYRiaMWOGQ2s3atRIe/bs0YULF/T555+rQ4cOiomJUaFCheyuuWvXLk2ZMkW7d+922IzjnTp16mT9c+XKlVWlShWVLFlSGzduVJMmTTJVOzU1VZLUqlUr9e/fX5L02GOP6eeff9bMmTPVsGHDTNW/0+zZs9WlSxeH/Vs3bdo0bdu2TcuXL1exYsW0adMm9enTR8HBwZmeJXd1ddU333yjnj17ys/PT87OzgoLC1Pz5s0f+iSGrPheyixmdnKxvn37auXKldqwYYOKFCnisLpubm4qVaqUatSooXHjxqlq1aqaMmWKQ2rv2rVL586dU/Xq1eXi4iIXFxdFR0dr6tSpcnFxccj/3m6XL18+lSlTRr///numawUFBaULfOXLl3fYbrI0f/75p9atW6eXXnrJoXUHDRpknd2pXLmyXnjhBfXv399hM2wlS5ZUdHS0rl69qlOnTmn79u1KTk5WiRIlHFL/doGBgZKU7kyOuLg467KcLi3o/Pnnn4qKinLorI4keXl5qVSpUqpTp45mzZolFxcXzZo1K1M1f/rpJ507d04hISHW7ffPP//Um2++qeLFizum8TuUKFFCBQsWdMg2XLBgQbm4uDyS7finn37S4cOHHbYd//3333r77bc1efJkRUREqEqVKurbt686duyoDz74wCGvUaNGDe3Zs0fx8fE6e/as1qxZo4sXLz7UNnyv76XAwEDdvHlT8fHxNus/qm2WsJMLGYahvn376ttvv9WPP/6o0NDQLH291NRU6/7czGrSpIn27dunPXv2WB81a9ZUly5dtGfPHjk7OzvkddJcvXpVx44dU1BQUKZr1a9fP92plEeOHFGxYsUyXft2c+bMUaFChdSiRQuH1k1KSpKTk+0m7+zsbP3frqN4eXkpKChIly9f1tq1a9WqVSuH1pek0NBQBQYGav369daxxMRExcTEOPz4sqyQFnSOHj2qdevWqUCBAln+mo7Yjl944QX9+uuvNttvcHCwBg0apLVr1zqoU1t//fWXLl686JBt2M3NTbVq1Xok2/GsWbNUo0YNhxwnJd36O5OcnPxItmFfX1/5+/vr6NGj2rlzZ4a24Qd9L9WoUUOurq422+zhw4d18uTJR7LNshsrC1y9etXmfyEnTpzQnj175Ofnp5CQkEzX79OnjxYsWKDvvvtO3t7e1v2dvr6+8vT0zFTtoUOHqnnz5goJCdGVK1e0YMECbdy40WH/kHl7e6c7tsjLy0sFChRwyDFHAwcOVEREhIoVK6YzZ85oxIgRcnZ2VufOnTNdu3///qpXr57Gjh2rDh06aPv27frss8/02WefZbp2mtTUVM2ZM0fdunWTi4tjN8+IiAiNGTNGISEhqlixon755RdNnjxZPXr0cEj9tWvXyjAMlS1bVr///rsGDRqkcuXK6cUXX7Sr3oO2o8jISL3//vsqXbq0QkNDNWzYMAUHB6t169aZrn3p0iWdPHnSeu2btC/HwMDADP0v9H71g4KC9Oyzz2r37t1auXKlUlJSrNuwn5+f3NzcMlW/QIECGjNmjJ555hkFBQXpwoUL+vjjj3X69OkMXcbgQZ/NncHM1dVVgYGBKlu27ANrP6i+n5+fRo4cqXbt2ikwMFDHjh3T4MGDVapUKYWHh2e6fkhIiAYNGqSOHTvqySefVKNGjbRmzRqtWLFCGzdudEh96VbwXrJkiSZNmpShmhmt3bBhQw0aNEienp4qVqyYoqOjNW/ePE2ePNkh9ZcsWSJ/f3+FhIRo37596tevn1q3bp2hA6Af9L3k6+urnj17asCAAfLz85OPj49ef/111a1bV3Xq1Hmoz8kuWX6+1/+gDRs2GJLSPbp16+aQ+nerLcmYM2dOpmv36NHDKFasmOHm5mb4+/sbTZo0MX744YfMN30fjjz1vGPHjkZQUJDh5uZmFC5c2OjYsaPx+++/O6S2YRjGihUrjEqVKhnu7u5GuXLljM8++8xhtQ3DMNauXWtIMg4fPuzQuoZhGImJiUa/fv2MkJAQw8PDwyhRooTxzjvvGDdu3HBI/a+++sooUaKE4ebmZgQGBhp9+vQx4uPj7a73oO0oNTXVGDZsmBEQEGC4u7sbTZo0yfDn9qDac+bMuevyESNGZLp+2unsd3ts2LAh0/X//vtvo02bNkZwcLDh5uZmBAUFGc8884yxfft2h3w2d3rYU8/vVz8pKclo2rSp4e/vb7i6uhrFihUzXn75ZSM2NtYh9dPMmjXLKFWqlOHh4WFUrVrVWLZsmUPrf/rpp4anp+dD//1/UO2zZ88a3bt3N4KDgw0PDw+jbNmyxqRJkzJ8eYoH1Z8yZYpRpEgRw9XV1QgJCTHefffdDP/7kJHvpb///tvo3bu3kT9/fiNPnjxGmzZtjLNnzz7MR2Q3y/9vEgAAwJQ4ZgcAAJgaYQcAAJgaYQcAAJgaYQcAAJgaYQcAAJgaYQcAAJgaYQcAAJgaYQcA7rBx40ZZLJZ09/EBkDsRdgAAgKkRdgAAgKkRdgDkOKmpqRo3bpxCQ0Pl6empqlWraunSpZL+bxfTqlWrVKVKFXl4eKhOnTrav3+/TY2vv/5aFStWlLu7u4oXL57upow3btzQkCFDVLRoUbm7u6tUqVKaNWuWzTq7du1SzZo1lSdPHtWrVy/d3bIB5A6EHQA5zrhx4zRv3jzNnDlTBw4cUP/+/fX8888rOjraus6gQYM0adIk7dixQ/7+/oqIiFBycrKkWyGlQ4cO6tSpk/bt26f33ntPw4YN09y5c63P79q1qxYuXKipU6fq0KFD+vTTT5U3b16bPt555x1NmjRJO3fulIuLi8PuEA/g0eJGoABylBs3bsjPz0/r1q1T3bp1reMvvfSSkpKS9Morr6hRo0ZatGiROnbsKEm6dOmSihQporlz56pDhw7q0qWLzp8/rx9++MH6/MGDB2vVqlU6cOCAjhw5orJlyyoqKkphYWHpeti4caMaNWqkdevWqUmTJpKk77//Xi1atNDff/8tDw+PLP4UADgSMzsAcpTff/9dSUlJ+te//qW8efNaH/PmzdOxY8es690ehPz8/FS2bFkdOnRIknTo0CHVr1/fpm79+vV19OhRpaSkaM+ePXJ2dlbDhg3v20uVKlWsfw4KCpIknTt3LtPvEcCj5ZLdDQDA7a5evSpJWrVqlQoXLmyzzN3d3Sbw2MvT0zND67m6ulr/bLFYJN06nghA7sLMDoAcpUKFCnJ3d9fJkydVqlQpm0fRokWt623bts3658uXL+vIkSMqX768JKl8+fLasmWLTd0tW7aoTJkycnZ2VuXKlZWammpzDBAA82JmB0CO4u3trYEDB6p///5KTU1VgwYNlJCQoC1btsjHx0fFihWTJI0aNUoFChRQQECA3nnnHRUsWFCtW7eWJL355puqVauWRo8erY4dO2rr1q2aPn26PvnkE0lS8eLF1a1bN/Xo0UNTp05V1apV9eeff+rcuXPq0KFDdr11AFmEsAMgxxk9erT8/f01btw4HT9+XPny5VP16tX19ttvW3cjjR8/Xv369dPRo0f12GOPacWKFXJzc5MkVa9eXYsXL9bw4cM1evRoBQUFadSoUerevbv1NWbMmKG3335bvXv31sWLFxUSEqK33347O94ugCzG2VgAcpW0M6UuX76sfPnyZXc7AHIBjtkBAACmRtgBAACmxm4sAABgaszsAAAAUyPsAAAAUyPsAAAAUyPsAAAAUyPsAAAAUyPsAAAAUyPsAAAAUyPsAAAAUyPsAAAAU/t/LkL48GtJmFEAAAAASUVORK5CYII=\n"},"metadata":{}}]},{"cell_type":"code","source":["# evaluate on last model\n","val_loss, val_accuracy = evaluate(model, val_loader, criterion)\n","print('MLP. Validation loss: {:.2f}, validation accuracy: {:.2f}'.format(val_loss, val_accuracy))\n"],"metadata":{"id":"TaABSxS682i6","executionInfo":{"status":"ok","timestamp":1774458203876,"user_tz":-60,"elapsed":81,"user":{"displayName":"Olivia Lecomte","userId":"07830699269148802057"}},"outputId":"bc5b8a26-df61-4779-da4c-0fe035824432","colab":{"base_uri":"https://localhost:8080/"}},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["MLP. Validation loss: 3.03, validation accuracy: 11.00\n"]}]},{"cell_type":"markdown","source":["3.3.6 **(1pt)** Are your new results reasonable?\n","\n","Yes, they are as expected."],"metadata":{"id":"i09gpDVXwab4"}},{"cell_type":"markdown","metadata":{"id":"0b08471f"},"source":["---\n","### Before You Submit:\n","\n","Please **Restart Session and Run All** to ensure your notebook runs cleanly from top to bottom without errors.\n","\n","This helps us grade your work fairly and ensures everything is saved correctly.\n","\n","Thank you and congratulations! πŸ₯³"]},{"cell_type":"code","source":["## DO NOT EDIT THIS CELL\n","points"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"vz7FTMac8njP","executionInfo":{"status":"ok","timestamp":1774458153574,"user_tz":-60,"elapsed":74,"user":{"displayName":"Olivia Lecomte","userId":"07830699269148802057"}},"outputId":"f9b8fd13-15b9-4949-c099-cb131dc42180"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["64"]},"metadata":{},"execution_count":15}]}]} \ No newline at end of file