{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "76bc35cd",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Tutorial: Universal approximation theorem\n",
    "\n",
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/git/https%3A%2F%2Fgitlab.in2p3.fr%2Fenergy4climate%2Fpublic%2Feducation%2Fmachine_learning_for_climate_and_energy/master?filepath=book%2Fnotebooks%2F10_tutorial_neural_networks.ipynb)\n",
    "\n",
    "Tutorial to the class [Neural Networks](10_neural_networks.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2948ff71",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Objective\n",
    "\n",
    "The objective of this tutorial is to design a simple network in order to understand the arhitecture of a neural network. Then we will focus on the hyperparameters of the network. Last we will work with the neural network implementation in scikit learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4b8e981b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a132e546",
   "metadata": {},
   "source": [
    "In the first part of this tutorial, we want to build a simple network with 1 input, 1 hidden layer with 3 neurons and 1 output. We want a sigmoid activation function for the hidden layer and a no activation function for the output neuron.\n",
    "\n",
    "> ***Question***\n",
    ">\n",
    "> - Draw this network on paper. Think about all the arrays you are going to need"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff45d35a",
   "metadata": {},
   "source": [
    "For now, let suppose we have only one observation `x` and one corresponding output `y = sin(x)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "268e0cc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = 0.5\n",
    "y = np.sin(2*x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42931b31",
   "metadata": {},
   "source": [
    "We write `N=3` the number of hidden neurons. Declare the arrays `w1`, `b1`, `w2` and `b2` for the weights and biases. Initialize these arrays with random values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "094e0aaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here\n",
    "N = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2e9d6c3",
   "metadata": {},
   "source": [
    "Write two functions:\n",
    "- one for the sigmoid \n",
    "- and one for the derivative of the sigmoid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4ab3fb4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7414cc0",
   "metadata": {},
   "source": [
    "Plot the sigmoid and its derivative it in the range [-10,10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "581a7fc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53388962",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Hidden layer\n",
    "\n",
    "> Write a function that computes the activation level in the hidden layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8d2c8034",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "350f9156",
   "metadata": {},
   "source": [
    "Write the `feedforward` function that computes the ouput layer as a function of the input layer `x`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "334c4317",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc7fdd85",
   "metadata": {},
   "source": [
    "Write the `cost` function that returns the cost function as a function of the input data `x` and output `y`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "618461c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d46afd8",
   "metadata": {},
   "source": [
    "What is the value of cost function of your network initialize with random weights and biases?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5f8087b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fb68696",
   "metadata": {},
   "source": [
    "### Gradient of the cost function\n",
    "\n",
    "We want to optimize our network and to do so we need to change the weights and biases. As we saw in class, we need to compute the gradient of the cost function with respect to the weights and biases in order to march down the gradient of the cost function. For this simple network you may want to re-derive the expression of the gradient based on the forwarward propagation function.\n",
    "\n",
    "On a piece of paper, derive\n",
    "\n",
    "\\begin{equation}\n",
    "\\frac{\\partial C}{\\partial \\mathbf w_2}\\quad \\frac{\\partial C}{\\partial \\mathbf b_2} \\quad \\frac{\\partial C}{\\partial \\mathbf w_1}\\quad \\frac{\\partial C}{\\partial \\mathbf b_1}\n",
    "\\end{equation}\n",
    "\n",
    "you will see that as you move backward in the network, you can reuse the derivative of the layer above. Do you see why it is called the backpropagation algorithm?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91ab7102",
   "metadata": {},
   "source": [
    "Implement it in python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "538a4c60",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5ee897a",
   "metadata": {},
   "source": [
    "Choose a learning rate `eta` and increment your weights and biases by $-\\eta \\nabla C$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "05fe3059",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bed88b95",
   "metadata": {},
   "source": [
    "Compute the new cost function. It should have gone down... There are many possibilities if that is not the case: first decrease the learning rate, then check all your functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "22bc5688",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ed019b9",
   "metadata": {},
   "source": [
    "Add a loop to repeat the previous operation until you reach convergence. You will have to first define convergence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "19ed3d70",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfa87d91",
   "metadata": {},
   "source": [
    "***Congratulations!!***\n",
    "\n",
    "You have built your first neural network."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c3817f1",
   "metadata": {},
   "source": [
    "Study the properties of convergence: how many epoch do you need to reach convergence?\n",
    "When you change the learning rate, how does it affect the convergence?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "af61134a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27153d54",
   "metadata": {},
   "source": [
    "### More samples\n",
    "\n",
    "So far, we have only worked with one sample. It is actually a good news that we were able to fit a neural network on it. As you may have noticed in the begining of this tutorial, we had $y = \\sin(2*x)$. So we are trying to guess this $\\sin$ function. Below is the actual data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "31f65422",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f90b1ab9ac0>]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAoo0lEQVR4nO3deXxU5dn/8c9FSICwhTXsixhQUNaAghsuKFItaqviilWLqLTV9rG1v/q0dnlaq4+1+tQNFcUNtdaFKu4bKiKEfYewQwIJWwgkIdv1+yNDOw0JSZiZTJL5vl+vec0597nvc66cOZlrznqbuyMiIrGrUbQDEBGR6FIiEBGJcUoEIiIxTolARCTGKRGIiMS4xtEO4Fi0b9/ee/XqFe0wRETqlQULFuxy9w7ly+tlIujVqxdpaWnRDkNEpF4xs80VlevQkIhIjFMiEBGJcUoEIiIxTolARCTGKRGIiMS4sCQCM5tmZllmtryS6WZmj5hZupktNbOhQdPGmtmawLS7wxGPiIhUX7j2CJ4Dxh5l+oVASuA1CXgcwMzigEcD0/sDV5lZ/zDFJCIi1RCW+wjcfbaZ9TpKlfHA8172zOu5ZpZkZp2BXkC6u28AMLNXAnVXhiMuEZGaOHComB05BWTnHiL7wCFyC4rILywhr7CEUncamdHIoGl8HK2axdOqaWPatWhC59ZNSW7VlPi4+nm0vbZuKOsKbA0a3xYoq6j8lIpmYGaTKNuboEePHpGJUkRiwqHiElZn5rJsew4rMvazPvsAG3cdJDv30DHP0wy6tG5GSnILUjq24IROrRjcI4ne7ZrTqJGFMfrwq61EUNFa8KOUH1noPhWYCpCamqredESk2opKSpm/aQ9z1+9m7oY9LN66j8KSUgBaN4snpWMLRvftQK/2zema1IyOLZvQvmUTWjeLp1lCHM3i42jcyCh1KCl18otK2J9fxP6CInYdKCRzXz4ZOQVs3n2QdTsP8M363Rwq/vf8h/ZI4rTj23NGSgf6JrfArG4lhtpKBNuA7kHj3YAMIKGSchGRkOQVFvPFmmw+XLmTT1btZH9BMY0MTu7amhtO68WQ7kmc1LU13do0q/YXc5xBXCMjoXEjWjeLr7ReSamzPvsAi7fsY9HWvXy7cQ+fvbsKWEWnVk0Ze1Inxp3cmdSeberE3kJtJYKZwJTAOYBTgBx3zzSzbCDFzHoD24EJwNW1FJOINDAlpc7sddm8Nn8rn67O4lBxKUmJ8Zw/oBNj+iczqk87Wjat/As8XOIaGX2TW9I3uSVXDC/7rbt9Xz5fr9vFx6t28vK8LTw3ZxMdWzZh7EmduGxoNwZ1ax21PQULR5/FZjYDGA20B3YCvwHiAdz9CSv76/5G2ZVFecAP3D0t0HYc8FcgDpjm7v9T1fJSU1NdD50TkcO27snjtbStvL5gG5k5BbRrnsDFg7pwwYBODO/VhsZ17CTugUPFfLJqJ7OWZfL5mmwOFZcyoEsrrjmlJ+MHd6F5k8j8RjezBe6eekR5fey8XolARAAWbN7LE1+s5+NVOwE4M6UDE4Z359wTk0loXLe+/CuTW1DEW4szeGnuZlbvyKVFk8ZcOqQrN5/Rm57tmod1WUoEItIglJY6n63J4okv1jN/015aN4vn+pE9mTCiB12TmkU7vGPm7izcso+Xvt3MO0syKS4t5buDunDb2cfTN7llWJahRCAi9Zq78+nqLB74YA2rd+TSpXVTbj7jOK4c3j1ih1KiJWt/AU99uYGXvt1CXmEJFwxI5kfnpHBS19YhzVeJQETqrQWb93Dfe6uZv2kvvdol8uNzU7h4UJd6ewNXde09WMizX2/kuTmb2F9QzHdO7szPx/Y75kNGlSWChpVGRaRBWZ99gPveW81HK3fSoWUT/nDJSVw5vHuDTwCHtWmewE/P78fNZx7H07M38MxXG7nx9F7hP3egPQIRqWvyCov526fpPPXlBpo2jmPy6D784LReJCbE9m/X/QVFtArh8lftEYhInefufLhyJ7/750q278vne0O78ctxJ9C+RZNoh1YnhJIEjkaJQETqhIx9+dzz1nI+XZ1Fv+SWvHbLSEb0bhvtsGKCEoGIRJW78/qCbfzunyspLnXu+c6JTBzVK2bOA9QFSgQiEjVZ+wv45RvL+GR1FiN6teWByweG/USoVE2JQESi4p9LMrjnreUUFJXw3xf15wejetWJB7DFIiUCEalV+YUl/PafK3hl/lYGd0/iwSsG0adDi2iHFdOUCESk1qzbmcvtLy9k7c4D3Da6D3eO6atzAXWAEoGIRJy78/e0bfx65nJaNGnM8zeO4My+HaIdlgQoEYhIRBUUlfDrt5fzWto2RvVpx1+vHEzHVk2jHZYEUSIQkYjZkVPALS8uYMnWffzonOO547y+xOmEcJ2jRCAiEZG2aQ+TX1xIfmExT1w7lLEndY52SFKJsJylMbOxZrbGzNLN7O4Kpt9lZosDr+VmVmJmbQPTNpnZssA0PUBIpAF4+dstXPXUXFo0iePN209TEqjjQt4jMLM44FFgDGWd1M83s5nuvvJwHXd/AHggUP9i4E533xM0m7PdfVeosYhIdJWWOn96bxVPfbmRs/p24JEJQ2idGPk+giU04Tg0NAJId/cNAIEO6scDKyupfxUwIwzLFZE6JL+whDtfXcz7K3Zw/cie/Pqi/nWur2CpWDg+pa7A1qDxbYGyI5hZImUd2P8jqNiBD81sgZlNqmwhZjbJzNLMLC07OzsMYYtIuGTnHmLCU3P5YOUO/vui/vz2uwOUBOqRcOwRVHQJQGWdHFwMfF3usNBp7p5hZh2Bj8xstbvPPmKG7lOBqVDWH0GoQYtIeKRnHeCGZ+ex68Ahnrh2GBcM6BTtkKSGwpGytwHdg8a7ARmV1J1AucNC7p4ReM8C3qTsUJOI1AOLt+7j8ifmUFBUwquTRioJ1FPhSATzgRQz621mCZR92c8sX8nMWgNnAW8HlTU3s5aHh4HzgeVhiElEIuyrdbu4+qm5tGjamNcnj2JQ96RohyTHKORDQ+5ebGZTgA+AOGCau68ws8mB6U8Eql4KfOjuB4OaJwNvmtnhWF529/dDjUlEIuvdpZnc8eoi+nRowfM3jtCdwvWc+iwWkRp56dvN3PPWcob1aMMzE4fr8tB6RH0Wi0jInv5yA394dxVn9+vAY9cMo1lCXLRDkjBQIhCRannyi/X86b3VXHhSJx65aogeH92AKBGISJUe/SydBz5Yw3cGduavVw5WEmhglAhE5Kge/ngdD328lvGDu/Dg5YN0o1gDpEQgIhVydx76aC2PfJrO94Z24/7vD9QjpBsoJQIROYK7c/8Ha3j88/VcmdqdP112sjqWb8CUCETkCA99tJbHP1/P1af04A/jT1ISaOB0sE9E/sPjn6/nkU/TuTK1u5JAjFAiEJF/mT5nE39+fzXfHdSFP+pwUMxQIhARAF6bv5XfzFzBmP7JPHjFIJ0YjiFKBCLCzCUZ/OKNpZyR0p6/Xa2bxWKNPm2RGPfhih3c+epihvdqy9TrUmnSWI+NiDVKBCIx7Ov0XUx5eREndW3NtBuG69lBMUqJQCRGLd+ewy0vLKB3++ZM/8FwWjTR1eSxSolAJAZt3n2QG56dR+tm8Uy/cQRJiQnRDkmiSD8BRGJMdu4hrntmHsWlzis3jqBTa3UqE+vCskdgZmPNbI2ZpZvZ3RVMH21mOWa2OPD6dXXbikj45BYUccOz88jKLWDaDcM5vmOLaIckdUDIewRmFgc8CoyhrCP7+WY2091Xlqv6pbtfdIxtRSREh4pLmPziAlbvyOXp61MZ2qNNtEOSOiIcewQjgHR33+DuhcArwPhaaCsi1VRa6vz0tSV8nb6b+783kLNP6BjtkKQOCUci6ApsDRrfFigrb6SZLTGz98xsQA3bYmaTzCzNzNKys7PDELZI7PjjrFW8uzSTuy88ge8N6xbtcKSOCUciqOg+dC83vhDo6e6DgP8D3qpB27JC96nunuruqR06dDjWWEVizvPfbOLprzYycWRPbjnzuGiHI3VQOBLBNqB70Hg3ICO4grvvd/cDgeFZQLyZta9OWxE5dp+s2sm9M1dw3okd+fXFAzDT84PkSOFIBPOBFDPrbWYJwARgZnAFM+tkgS3QzEYElru7Om1F5Ngs357Dj2Yson+XVjw8YYgeIieVCvmqIXcvNrMpwAdAHDDN3VeY2eTA9CeA7wO3mlkxkA9McHcHKmwbakwisS5jXz43PjefpGbxTJs4nOa6a1iOwsq+j+uX1NRUT0tLi3YYInVSbkERlz/xDdv35vP6raPo16lltEOSOsLMFrh7avly/UwQaUCKSkq57aWFpGcd4LkfjFASkGpRIhBpINyd38xcwZfrdnH/9wZyekr7aIck9YQeOifSQDz/zWZe/nYLk8/qwxXDu1fdQCRAiUCkAfhyXTa/e2cl552YzM8v6BftcKSeUSIQqefWZx/gtpcWktKxBX+dMFgdzkuNKRGI1GM5eUX8cHoaCXGNeOr6VHUuI8dEW41IPVVcUsrtLy9k6948Xv7hqXRvmxjtkKSeUiIQqad+/85Kvkovu0JoeK+20Q5H6jEdGhKph16cu5np32zm5tN76wohCZkSgUg9M2f9Lu6duYLR/Trwy3EnRjscaQCUCETqkS2787jtpYX0at+cR67Sg+QkPJQIROqJvMJiJr2Qhjs8MzGVVk3jox2SNBBKBCL1gLtz1+tLWbszl0euGkLPds2jHZI0IEoEIvXAk7M38O7STO664ATO6qse+iS8lAhE6rjZa7O5//3VfOfkzkw+S11NSviFJRGY2VgzW2Nm6WZ2dwXTrzGzpYHXHDMbFDRtk5ktM7PFZqZOBkSCbNmdx49mLKJvckvu//5AdTUpERHyDWVmFgc8CoyhrA/i+WY2091XBlXbCJzl7nvN7EJgKnBK0PSz3X1XqLGINCT/PjnsPHndMPUyJhETjj2CEUC6u29w90LgFWB8cAV3n+PuewOjcynrpF5EKuHu/Pz1pazRyWGpBeFIBF2BrUHj2wJllbkJeC9o3IEPzWyBmU2qrJGZTTKzNDNLy87ODilgkbpu6uwNvLM0k7su6Mfofh2jHY40cOHY16zooGWFHSGb2dmUJYLTg4pPc/cMM+sIfGRmq9199hEzdJ9K2SElUlNT619HyyLV9OW6bP78/mrGndyJW8/qE+1wJAaEY49gGxD8sJNuQEb5SmY2EHgaGO/uuw+Xu3tG4D0LeJOyQ00iMWnL7jymvLyIlI4teeD7g3RyWGpFOBLBfCDFzHqbWQIwAZgZXMHMegBvANe5+9qg8uZm1vLwMHA+sDwMMYnUO8Enh6der5PDUntC3tLcvdjMpgAfAHHANHdfYWaTA9OfAH4NtAMeC/zCKXb3VCAZeDNQ1hh42d3fDzUmkfrG3fl/byxjzc5cnr1huE4OS60Ky08Od58FzCpX9kTQ8M3AzRW02wAMKl8uEmtenLuZtxZn8NMxfXVyWGqd7iwWibKFW/byu3dWcna/Dkw5+/hohyMxSIlAJIp2HzjE7S8tJLlVUx66Uh3PS3TobJRIlJSUOj95ZTG7Dxbyxq2jSEpMiHZIEqO0RyASJQ99tJav0nfx+/EDOKlr62iHIzFMiUAkCj5euZO/fZbOlanduXJ4j2iHIzFOiUCklm3Zncedry1mQJdW/Hb8gGiHI6JEIFKbCopKmPziAgx44tphNI2Pi3ZIIjpZLFJb3J173lrOysz9TLshle5tE6MdkgigPQKRWvPK/K28vmAbPz7neM45ITna4Yj8ixKBSC1Yum0fv3l7BWektOcn5/WNdjgi/0GJQCTC9h4s5NYXF9K+RQIPTxhCnG4akzpG5whEIqi01Lnj1cVk5x7itckjadtcN41J3aM9ApEIeuTTdXyxNptfX9yfwd2Toh2OSIWUCEQi5PM1WTz8yTouG9KVa07RTWNSdykRiETA1j153PHqYvolt+R/Lj1ZPY1JnaZEIBJmBUUl3PbSQkpKnMevHUazBN00JnVbWBKBmY01szVmlm5md1cw3czskcD0pWY2tLptReqb3/5zJcu25/DgFYPo3V49jUndF3IiMLM44FHgQqA/cJWZ9S9X7UIgJfCaBDxeg7Yi9cbf07YyY94Wbh3dh/MHdIp2OCLVEo49ghFAurtvcPdC4BVgfLk644HnvcxcIMnMOlezrUi9sCIjh3veWs7I49rxszG6aUzqj3Akgq7A1qDxbYGy6tSpTlsAzGySmaWZWVp2dnbIQYuEU05+Ebe+uJCkxHj+7+ohNI7T6TepP8KxtVZ0OYRXs0512pYVuk9191R3T+3QoUMNQxSJnNJS52evLSZjXz6PXTOU9i2aRDskkRoJx53F24DuQePdgIxq1kmoRluROu3xL9bz8aosfnNxf4b1bBvtcERqLBx7BPOBFDPrbWYJwARgZrk6M4HrA1cPnQrkuHtmNduK1Flfp+/iwQ/XcPGgLtwwqle0wxE5JiHvEbh7sZlNAT4A4oBp7r7CzCYHpj8BzALGAelAHvCDo7UNNSaR2pCZk8+PZyyiT4cW3HeZbhqT+issD51z91mUfdkHlz0RNOzA7dVtK1LXFRaXcttLCykoKuHxa4fRvIme3yj1l7ZekWPwx1mrWLRlH49ePZTjO7aIdjgiIdE1biI19Pbi7Tw3ZxM3nd6b7wzsHO1wREKmRCBSA2t35nL3P5YxvFcb7r7whGiHIxIWSgQi1ZRbUMTkFxbQvElj/nb1UOJ105g0EDpHIFIN7s7PX1/K5j15vHTzKSS3ahrtkETCRj9pRKrhma828t7yHfxibD9OPa5dtMMRCSslApEqzNu4hz+9t5qxAzrxwzOOi3Y4ImGnRCByFFn7C7j95YX0aJvIA5cP1E1j0iDpHIFIJYpKSpny8iIOFBTz4k2n0LJpfLRDEokIJQKRSjzwwRrmbdrDX68cTL9OLaMdjkjE6NCQSAXeW5bJ1NkbuH5kTy4ZUmEXGSINhhKBSDnrsw9w1+tLGdw9iV9958RohyMScUoEIkEOHirm1hcXkNC4EY9dM5QmjeOiHZJIxCkRiAS4O3e9voT0rAM8MmEIXZKaRTskkVqhRCAS8OTsDcxatoNfjD2B01PaRzsckVqjRCACzF6bzf3vr+Y7Azsz6UzdNCaxJaREYGZtzewjM1sXeG9TQZ3uZvaZma0ysxVm9pOgafea2XYzWxx4jQslHpFjsXVPHj+asYiUji154Pu6aUxiT6h7BHcDn7h7CvBJYLy8YuBn7n4icCpwu5n1D5r+kLsPDrzUU5nUqvzCEia9sAB358nrhpGYoFtrJPaEmgjGA9MDw9OBS8pXcPdMd18YGM4FVgG6MFuizt25+42lrN6xn4evGkKv9s2jHZJIVISaCJLdPRPKvvCBjkerbGa9gCHAt0HFU8xsqZlNq+jQUlDbSWaWZmZp2dnZIYYtAtO+3sTbizP42Zi+nN3vqJuuSINWZSIws4/NbHkFr/E1WZCZtQD+Adzh7vsDxY8DfYDBQCbwYGXt3X2qu6e6e2qHDh1qsmiRI8xZv4s/zlrFBQOSuW308dEORySqqjwg6u7nVTbNzHaaWWd3zzSzzkBWJfXiKUsCL7n7G0Hz3hlU5yngnZoEL3Istu/LZ8rLi+jVLpH/vXwQjRrp5LDEtlAPDc0EJgaGJwJvl69gZZdgPAOscve/lJsW3PP3pcDyEOMROaqCohImv7CAwuJSpl6fqieKihB6IrgPGGNm64AxgXHMrIuZHb4C6DTgOuCcCi4Tvd/MlpnZUuBs4M4Q4xGplLtzz1vLWbY9h4euHEyfDi2iHZJInRDStXLuvhs4t4LyDGBcYPgroMJ9b3e/LpTli9TEs19v4vUF2/jxOcczpn9ytMMRqTN0Z7HEhNlrs/nDuys5v38yd5zXN9rhiNQpSgTS4G3IPsCUlxfSN7klD105WCeHRcpRIpAGLSe/iJufT6NxXCOeuj6V5k1057BIeUoE0mCVlDo/nrGILbvzeOyaoXRvmxjtkETqJP08kgbrT7NW8cXabP546cmcely7aIcjUmdpj0AapL+nbeXprzYycWRPrj6lR7TDEanTlAikwVmweQ+/enM5px3fjv++qH/VDURinBKBNCgZ+/K55YWFdE5qyqNXD6VxnDZxkaroHIE0GAcPFXPz9DQKikqY8cNTSEpMiHZIIvWCfi5Jg3D4CqHVO/bzf1cPISW5ZbRDEqk3tEcgDcLv31nJJ6uz+P34AepbQKSGtEcg9d70OZt4bs4mbjq9N9eN7BXtcETqHSUCqdc+Xb2T3/5zBeedmMz/G3ditMMRqZeUCKTeWpGRw5SXF9G/SyseuWowcXqGkMgxUSKQemlHTgE3PZdG62bxPDNxOIkJOt0lcqxCSgRm1tbMPjKzdYH3CjufN7NNgQ5oFptZWk3biwQ7cKiYm6bPJ7egiGcmDie5VdNohyRSr4W6R3A38Im7pwCfBMYrc7a7D3b31GNsL0JhcSmTX1jA6h25/O2aofTv0iraIYnUe6EmgvHA9MDwdOCSWm4vMaS01Pn560v4Kn0X9112si4TFQmTUBNBsrtnAgTeK/vPdOBDM1tgZpOOoT1mNsnM0swsLTs7O8SwpT768wereWtxBndd0I/LU7tHOxyRBqPKM2xm9jHQqYJJv6rBck5z9wwz6wh8ZGar3X12Ddrj7lOBqQCpqalek7ZS/z379Uae/GID153ak9tG94l2OCINSpWJwN3Pq2yame00s87unmlmnYGsSuaREXjPMrM3gRHAbKBa7SW2vbM0g9+9s5ILBiRz73cHYKbLREXCKdRDQzOBiYHhicDb5SuYWXMza3l4GDgfWF7d9hLbvlm/m5++uoRhPdrw8IQhuldAJAJCTQT3AWPMbB0wJjCOmXUxs1mBOsnAV2a2BJgHvOvu7x+tvQjA8u05THohjR7tEnl6YipN4+OiHZJIgxTSXTjuvhs4t4LyDGBcYHgDMKgm7UXSsw5w/bR5tGoaz/M3jtAjpUUiSHcWS52zdU8e1z79LY3MePHmU+iS1CzaIYk0aEoEUqdk7S/g2me+Ja+wmBduGkHv9s2jHZJIg6cHtEidsS+vkOuemUd27iFevPkUTuysu4ZFaoMSgdQJBw4VM/HZ+WzcfZDnbhjO0B567JRIbdGhIYm6/MISbp4+n+Xbc3j06qGMOr59tEMSiSlKBBJV+YUl3Pz8fL7duIe/XDGIMf2Tox2SSMxRIpCoOZwE5qzfzYOXD2L84K7RDkkkJukcgURFfmEJN02fzzcbypLAZUO7RTskkZilPQKpdUoCInWL9gikVgUngb9cMYhLhygJiESb9gik1uQVFisJiNRB2iOQWpGTX8SNz81n0Za9SgIidYwSgUTcrgOHuP6ZeazLyuXRq4dy4cmdox2SiARRIpCI2r4vn+ue/paMnHyenjics/p2iHZIIlKOEoFEzIbsA1z79LfkHirmxZtOIbVX22iHJCIVUCKQiFi0ZS83TU/DgBk/PJWTuraOdkgiUomQrhoys7Zm9pGZrQu8H/GkMDPrZ2aLg177zeyOwLR7zWx70LRxocQjdcNHK3dy1VNzadGkMX+fPFJJQKSOC/Xy0buBT9w9BfgkMP4f3H2Nuw9298HAMCAPeDOoykOHp7v7rPLtpX55Ye5mbnkhjX7JLXnjtlEc16FFtEMSkSqEmgjGA9MDw9OBS6qofy6w3t03h7hcqWNKS50/v7+a/35rOWf368iMSafSvkWTaIclItUQaiJIdvdMgMB7xyrqTwBmlCubYmZLzWxaRYeWDjOzSWaWZmZp2dnZoUUtYZVfWMKPXlnE45+v56oRPXjyumEkJuj0k0h9Ye5+9ApmHwOdKpj0K2C6uycF1d3r7hV+mZtZApABDHD3nYGyZGAX4MDvgc7ufmNVQaempnpaWlpV1aQWZOzL54fPp7Eycz+/GHsCt5x5HGYW7bBEpAJmtsDdU8uXV/mzzd3PO8pMd5pZZ3fPNLPOQNZRZnUhsPBwEgjM+1/DZvYU8E5V8UjdsWDzXm55YQEFRSU8MzGVc05QXwIi9VGoh4ZmAhMDwxOBt49S9yrKHRYKJI/DLgWWhxiP1JK/p23lqqlzad4kjrduH6UkIFKPhXog9z7gNTO7CdgCXA5gZl2Ap919XGA8ERgD3FKu/f1mNpiyQ0ObKpgudUxBUQm//edKZszbwmnHt+PRq4eSlJgQ7bBEJAQhJQJ3303ZlUDlyzOAcUHjeUC7CupdF8rypXZt3n2Q215ayIqM/dw6ug8/G9OXxnF6gK1IfadLO6Ra3l++g7teX0IjM56ZmMq5J+pQkEhDoUQgR1VQVMJ9763muTmbGNStNX+7eijd2yZGOywRCSMlAqnUsm053PHqItZnH+QHp/Xi7gtPoEnjuGiHJSJhpkQgRyguKeWJL9bz14/X0a5FAi/cNIIzUvT4aJGGSolA/kN6Vi4/f30pC7fs46KBnfnDJSfpqiCRBk6JQICycwGPfZbO41+sJzGhMQ9PGMz4wV2jHZaI1AIlAmHO+l3c8+ZyNuw6yCWDu3DPRf31wDiRGKJEEMN25BRw/wereWPhdnq0TeT5G0dwprqSFIk5MZUI3F0PRAMOHirmyS/WM/XLDZSWwm2j+/Cjc1JolqArgkRiUUwlgqe/3MjsddncOaYvQ3tU+sTrBqu4pJR/LNzG/364luzcQ1w0sDO/GHuC7gsQiXExlQiaN2nMioz9XPbYHEb368Cd5/VlUPekaIcVcUUlpby5aDuPfZbOpt15DO2RxJPXDYvJZCgiR6qyP4K6KJT+CA4eKmb6N5uYOnsD+/KKOOeEjtw2ug/DerZpcIeNDhWX8Pe0bTz++Xq278vnpK6tmHJ2ChcMSG5wf6uIVK2y/ghiLhEcduBQMdPnlCWEnPwiTu7amhtG9eKiQZ3r/d2zmTn5zJi3lRnztpCde4ghPZL48TkpjO7XQQlAJIYpEVQir7CYNxZu57k5m0jPOkD7FglcPaIHlw3tRq/2zcOyjNrg7sxZv5sXvtnMR6t2UurO6L4duOn04zjt+HZKACKiRFAVd+er9F08+/UmPluThTsM7p7EpUO6ctHAzrSrg9fVuzvLtufwztJM3l2ayfZ9+bRJjOeK4d25ZkRPerTTSWAR+TclghrI2JfPzCUZvLVoO6t35BLXyBh5XDvOPqEj557QMap7CoXFpSzcspfP12Qza1kmW/bkER9nnJHSge8O6sLYkzrRNL5+H9oSkciISCIws8uBe4ETgRHuXuG3s5mNBR4G4ijruey+QHlb4FWgF2U9lF3h7nurWm5tdl6/Zkcuby3ezkcrd5KedQCAHm0TGdG7LSN6t2VYzzb0btecRo0ic+iloKiEpdtyWLhlL2mb9vDN+t0cLCwhrpExqk87Lh7YhQsGdKJ1YnxEli8iDUekEsGJQCnwJPBfFSUCM4sD1lLWVeU2YD5wlbuvNLP7gT3ufp+Z3Q20cfdfVLXc2kwEwbbszuPT1Tv5ZsNu5m3cw968IgASE+I4sXMr+nVqSe92zenRLpFubZrRoUUT2jRPIP4ovXi5OwVFpew+eIiMfQVs35fHlt35rM3KZe2OXDbuOkhxadln1Lt9c0b1aceZfTswsk87WjXVl7+IVF9liSDUripXBWZ+tGojgHR33xCo+wowHlgZeB8dqDcd+ByoMhFES492idxwWm9uOK03paVOevYBFm/dx8qM/azIyGHWskz2BZJDsJZNG9MsPo6m8XHExxklpU5RiXOouJT9+UUUlpQeuay2ifTr1JILBnRicPckhvRIqpPnKUSk/quNG8q6AluDxrcBpwSGk909E8DdM82sY2UzMbNJwCSAHj16RCjU6mvUyOib3JK+yS3/ozwnr4jNew6SsS+f7AOF7D5wiH15RRQUlVBQVEJRidM4zohrZDRp3IhWzeJJapZAm8R4uiQ1o2ubZnRNaqbj/CJSa6pMBGb2MdCpgkm/cve3q7GMinYXanw8yt2nAlOh7NBQTdvXltaJ8QxMTGJgt6RohyIiUi1VJgJ3Py/EZWwDugeNdwMyAsM7zaxzYG+gM5AV4rJERKSGKj+LGT7zgRQz621mCcAEYGZg2kxgYmB4IlCdPQwREQmjkBKBmV1qZtuAkcC7ZvZBoLyLmc0CcPdiYArwAbAKeM3dVwRmcR8wxszWUXZV0X2hxCMiIjWnG8pERGJEZZeP1sahIRERqcOUCEREYpwSgYhIjFMiEBGJcfXyZLGZZQObj7F5e2BXGMMJF8VVM4qrZhRXzdTVuCC02Hq6e4fyhfUyEYTCzNIqOmsebYqrZhRXzSiumqmrcUFkYtOhIRGRGKdEICIS42IxEUyNdgCVUFw1o7hqRnHVTF2NCyIQW8ydIxARkf8Ui3sEIiISRIlARCTGNchEYGaXm9kKMys1s0ovszKzsWa2xszSA30mHy5va2Yfmdm6wHubMMVV5XzNrJ+ZLQ567TezOwLT7jWz7UHTxtVWXIF6m8xsWWDZaTVtH4m4zKy7mX1mZqsCn/lPgqaFdX1Vtr0ETTczeyQwfamZDa1u2wjHdU0gnqVmNsfMBgVNq/AzraW4RptZTtDn8+vqto1wXHcFxbTczErMrG1gWkTWl5lNM7MsM1teyfTIblvu3uBewIlAP8r6QE6tpE4csB44DkgAlgD9A9PuB+4ODN8N/DlMcdVovoEYd1B2EwjAvcB/RWB9VSsuYBPQPtS/K5xxAZ2BoYHhlsDaoM8xbOvraNtLUJ1xwHuU9cp3KvBtddtGOK5RQJvA8IWH4zraZ1pLcY0G3jmWtpGMq1z9i4FPa2F9nQkMBZZXMj2i21aD3CNw91XuvqaKaiOAdHff4O6FwCvA+MC08cD0wPB04JIwhVbT+Z4LrHf3Y72LurpC/Xujtr7cPdPdFwaGcynr86JrmJYf7GjbS3C8z3uZuUCSlfW8V522EYvL3ee4+97A6FzKegmMtFD+5qiur3KuAmaEadmVcvfZwJ6jVInottUgE0E1dQW2Bo1v499fIMnungllXzRAxzAts6bzncCRG+GUwK7htHAdgqlBXA58aGYLzGzSMbSPVFwAmFkvYAjwbVBxuNbX0baXqupUp20k4wp2E2W/LA+r7DOtrbhGmtkSM3vPzAbUsG0k48LMEoGxwD+CiiO1vqoS0W2ryj6L6yoz+xjoVMGkX7l7dbq8tArKQr6W9mhx1XA+CcB3gV8GFT8O/J6yOH8PPAjcWItxnebuGWbWEfjIzFYHfskcszCurxaU/cPe4e77A8XHvL4qWkQFZeW3l8rqRGRbq2KZR1Y0O5uyRHB6UHHYP9MaxLWQssOeBwLnb94CUqrZNpJxHXYx8LW7B/9Sj9T6qkpEt616mwjc/bwQZ7EN6B403g3ICAzvNLPO7p4Z2P3KCkdcZlaT+V4ILHT3nUHz/tewmT0FvFObcbl7RuA9y8zepGy3dDZRXl9mFk9ZEnjJ3d8Imvcxr68KHG17qapOQjXaRjIuzGwg8DRwobvvPlx+lM804nEFJWzcfZaZPWZm7avTNpJxBTlijzyC66sqEd22YvnQ0Hwgxcx6B359TwBmBqbNBCYGhicC1dnDqI6azPeIY5OBL8PDLgUqvMIgEnGZWXMza3l4GDg/aPlRW19mZsAzwCp3/0u5aeFcX0fbXoLjvT5whcepQE7gkFZ12kYsLjPrAbwBXOfua4PKj/aZ1kZcnQKfH2Y2grLvo93VaRvJuALxtAbOImibi/D6qkpkt61wn/2uCy/K/um3AYeAncAHgfIuwKygeuMou8pkPWWHlA6XtwM+AdYF3tuGKa4K51tBXImU/UO0Ltf+BWAZsDTwYXeurbgouyphSeC1oq6sL8oOc3hgnSwOvMZFYn1VtL0Ak4HJgWEDHg1MX0bQFWuVbWthWk9VxfU0sDdo/aRV9ZnWUlxTAstdQtlJ7FF1YX0Fxm8AXinXLmLri7IffZlAEWXfXTfV5ralR0yIiMS4WD40JCIiKBGIiMQ8JQIRkRinRCAiEuOUCEREYpwSgYhIjFMiEBGJcf8fE1aAuvUTiHoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "N_sample = 200\n",
    "x = np.linspace(-1, 1, N_sample)[:, None]\n",
    "y = np.sin(2*x)\n",
    "plt.plot(x,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be8fc056",
   "metadata": {},
   "source": [
    "Below is an extension of the code you have just writen in order to compute the cost function and the gradient over all sample. The code is writen on purpose in a very compact form. \n",
    "\n",
    "> - Look at the code above and identify all the steps that we have described so far. \n",
    "> - Add comments in the code to describe what the code is doing\n",
    "> - Change the code to define 3 hyper parameters: number of neurons in the hidden layer, learning rate and the number of epochs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "fe32b4f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tanh(x):\n",
    "    return np.tanh(x)\n",
    "def derivative_tanh(x):\n",
    "    return 1 - tanh(x)**2\n",
    "w1 = np.random.uniform(0, 1, (1, 10))\n",
    "w2 = np.random.uniform(0, 1, (10, 1))\n",
    "b1 = np.full((1, 10), 0.1)\n",
    "b2 = np.full((1, 1), 0.1)\n",
    "for i in range(5000):\n",
    "    a1 = x\n",
    "    z2 = a1.dot(w1) + b1\n",
    "    a2 = tanh(z2)\n",
    "    z3 = a2.dot(w2) + b2\n",
    "    cost = np.sum((z3 - y)**2)/2\n",
    "    z3_delta = z3 - y\n",
    "    dw2 = a2.T.dot(z3_delta)\n",
    "    db2 = np.sum(z3_delta, axis=0, keepdims=True)\n",
    "    z2_delta = z3_delta.dot(w2.T) * derivative_tanh(z2)\n",
    "    dw1 = x.T.dot(z2_delta)\n",
    "    db1 = np.sum(z2_delta, axis=0, keepdims=True)\n",
    "    for param, gradient in zip([w1, w2, b1, b2], [dw1, dw2, db1, db2]):\n",
    "        param -= 0.0001 * gradient"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc6abfd6",
   "metadata": {},
   "source": [
    "In the same figure, plot the output as a function of input parameters \n",
    "- for the observations\n",
    "- for the output of the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "343d0690",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e2f7d43",
   "metadata": {},
   "source": [
    "Modify the code above to get the cost function at each epoch and plot it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "3fd27b92",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5faf4814",
   "metadata": {},
   "source": [
    "What hyper parameter has the most impact to get an accurate reconstruction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c46fc6a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f1da9cf",
   "metadata": {},
   "source": [
    "### Neural network with scikit-learn\n",
    "\n",
    "Scikit-learn has a built-in Neural network function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "edb23716",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neural_network import MLPRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cc402fc",
   "metadata": {},
   "source": [
    "Look at the documentation of the `MLPRegressor`. This class of model is extremely similar to what we have been working on so far. Identify the parameters that we have already explored."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "193316f5",
   "metadata": {},
   "source": [
    "We are now trying to predict a more complicated function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "bc175f99",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f90a6f402e0>]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABE3klEQVR4nO29aXRj13Wg+22AMwHO4EzWQLJY86RSabJlybIUSY4l24nzpPjZaidptZ6tvDidznvK6n5pJ+msuJNO0p3EsSMnjoek7dgrdqw4kjVZsmRbQ5WkGlkDh5pYHADOBFkcAJz3497LglhkcQAucC9wvrW4CNxx8/Lcs8/eZ5+9RSmFRqPRaLIXT7oF0Gg0Gk160YpAo9FoshytCDQajSbL0YpAo9FoshytCDQajSbLyUm3AOuhqqpKbdy4Md1iaDQajat46623hpRSgcXbXakINm7cyOHDh9Mthkaj0bgKEbmw1HbtGtJoNJosRysCjUajyXK0ItBoNJosRysCjUajyXK0ItBoNJosJymKQES+IiJBETmxzH4Rkb8QkS4ROSYi++P23SsiZ8x9TyRDHo1Go9GsnmRZBF8F7r3O/vuANvPnUeCLACLiBb5g7t8OPCwi25Mkk0aj0WhWQVLWESilXhGRjdc55EHg68rIef26iJSJSB2wEehSSvUAiMi3zGM7kiGXJrm80TPMuaEpfAU53LezDq9H0i1S1jAXifF8xyAjU7PcuKmCrbUl6RYpq+gcnORo7zjz0Rgf2lOPL9+VS7CWJVV/TQNwKe57r7ltqe03LXUBEXkUw5qgubnZHik1SxKejfD//csJvvfO5YVt2+q6+fP/Y4/ukFLAkUtj/PuvHyY0Obuw7ZO3bOC/fmiHVsY2E4spnny1hz959gzRmFG75U+fO8N/+/BO7t1Zl2bpkkeqJouXaq3qOtuv3ajUk0qpA0qpA4HANSukNTYRiyke+8ZbfP/IZT77gTZ++sT7+atf3sdQeJZP/f0hgpMz6RYxozk9MMEjX3mTglwPX/3Ujbz6/9zJr9y2ia+/doH/+O0j6MJS9vI/XzjL5585zc/tqOFHv/U+/vn/uoX6skJ+/Zvv8Oa5kXSLlzRSpQh6gaa4741A33W2axzCF3/czU+6hvjDj+zisx/YQkNZIT+/u56vfupGRqfn+PQ/vL0wUtIkl9lIlMe+8RYFuR7+96/dzB3t1TRVFPG7H9rOb35gC98/0sdTR/XrYhev9wzzly918Qv7G/nCL+9nc8DHDRsq+Mav3ERTeRGP/cNbBCcyYyCUKkXwFPBJM3roZmBcKdUPHALaRGSTiOQBD5nHahxAdyjMnz1/lg/uruOhG5vetW9HfSl/+OFdHL4w+i6XkSZ5/O2r5zg/PM2f/OIemiqK3rXv8fe3sq+5jM89dZKh8OwyV9Csl9lIlN/69lE2Vhbz+w/uQOSq86K0KJcnP3mAyZl5/vyFs2mUMnkkK3z0m8BrQLuI9IrIr4rIYyLymHnI00AP0AV8Gfg0gFIqAjwOPAucAr6tlDqZDJk0ifOXL3aS5/Xwew+8+0Ww+Oj+BnY3lvJnz51hZj6aBgkzl8GJGf7qR13cs72G27dc6wr1eoQ//oXdTMxE+OLL3WmQMLP5zuFeLo9d4fce2EHxEhPDrdU+PnHzRv7p0CU6ByfTIGFySYoiUEo9rJSqU0rlKqUalVJ/p5T6klLqS+Z+pZT6jFKqRSm1Syl1OO7cp5VSW8x9f5gMeTSJ0xUM89TRPj55ywaqfPlLHiMiPHHvVvrGZ/jHNy6mWMLM5u9/ep7ZSJT/8sHlo6nbavw8sKeeb755kfHp+RRKl9nMRWJ88eVu9jWX8d62qmWPe/z9rRTn5fCnz7nfKtArizVL8qUfd5Of4+Xf3775usfd2lrFjRvL+fpr54npuYKkcGUuyjffvMjP7ailubLousf+h/dtZnouyjdeP58a4bKAfzlymctjV/iNu9qWtIQtKorz+MQtG3iuY4C+sSsplDD5aEWguYbJmXl+cKyPD++rX9YaiOf/vHkDF4an+Vn3cAqky3z+5chlxq/M8+9u3bjisVtrS7ijPcBXf3aB+WjMfuGygG++eZG2ah/vW8Ilt5iHDzajgG8durTisU5GKwLNNfzgWD8z8zF+6UDTygcD9+6spaI4j398Y8maF5o18o3XLrCtroSDmypWdfwvH2xmKDzLTzqHbJYs8+kKTvLOxTE+dqDxutaARVNFEXdsCfCtNy+6WhFrRaC5hm8fvkRbtY+9TWWrOj4/x8vHbmjk+Y5BHcGSIF3BSTr6J/jYDavriADuaK+mvCiX7+rorYT5zuFevB7hI/saV33Ox2/aQHBylpfPhGyUzF60ItC8i55QeE0jIosP72sgElM8d3LQRukyn3892o8IfHD36let5uV4+Pnd9Tx3coDJGT1pvF6iMcV337nMne3VBPwru0Qt3tceoLQwl2eO99sonb1oRaB5Fz88OQDAz++uX9N5W2v9bKoq5pkT7n0Z0o1Sin891sdNmyqoKSlY07kf3tfAbCTGD08M2CRd5vP2xVFCk7M8sHdtbT/X6+Ge7TU8f2qQ2Yg7w6i1ItC8i+dODrK7sZT6ssI1nSci3Luzlte6hxmbnrNJuszmVP8kPaEpPrRnbR0RwP7mMupLC3i+Q1tk6+W5kwPkeoU729eewub+XXVMzkT4WZc7Aya0ItAsMDgxw5FLY9yzvWZd59+/s45ITOnOaJ083zGICNy7o3bN54oI799WzaudQ3px3zpQSvFcxyC3tlThL8hd8/m3tVbhL8jh31zqHtKKQLPAc2YH/nPr6IgAdjaU0FBWuHAdzdp46UyQPY1lVK4iZHcp7tpWw5X5KK/1uHNUmk46g2EuDE9z9zoHQXk5Hu7aWs2PTgdduZ5GKwLNAi+eGmRjZRGt1b51nS8ivK89wGvdw64OpUsHI1NzHO0d4451uCUsbtlcSWGulx+dCiZRsuzAsmLXqwjAmDQemZrjZN9EssRKGVoRaAAjydYbPSO8b0tgTdFCi7m9LUB4NsI7F8eSJ1wW8MrZEErBne3V675GQa6X97RV8eKpQZ2eeo282hlie13Jmifp43lvm6HEX+l0XxipVgQaAN6+MMaV+SjvaUus1sOtrZV4PcIrZ933MqSTl88EqSzOY1dDaULXuaM9QN/4DOeGppIkWeYzPRfh7Qtj180rtBqqfPnsqC/hxy5s+1oRaAD4adcQXo9w0+bVrWZdjpKCXPY1lblyVJQulFK82jnE7VsCeBKsOHZbi9GZ6XQfq+fQ+VHmojFua01MEQDcviXA2xdGXbeeQysCDQA/6Rpib1MZJeuImFjMe9sCHL88zsiUDiNdDZ3BMMNTc9zSUpnwtTZUFlFfWsBrWhGsmp92DZHn9XDjxsQGQWC4RiMx5brnrxWBhvHpeY71jiVlRATwnrZKlCKjSvnZyetmlM8tmxNXBCLCLS1VvNYz7MrolXTwk84hbthQTmGeN+Fr7d9QRl6Ox3VtXysCDW+eHyGm4LYkjEgBdjWUke/ClyFdvNEzQn1pAY3la1vEtxy3tlQyMjXHmQwomGI3o1NzdPRPcFtrctp+fo6XfU1lvHneXW0/WRXK7hWRMyLSJSJPLLH/t0XkiPlzQkSiIlJh7jsvIsfNfYevvbrGbg6fHyHP62HPKpPMrURejod9zWW8ed5d5nE6UErxes8wN2+uTChaKx7LxaTnCVbmrQujABzclBxFAHDTpgpOXB531TxBwopARLzAF4D7gO3AwyLyrrJKSqk/UUrtVUrtBX4H+LFSKl5l3mnuP5CoPJq1c+j8CLsbSynITdw0tji4qZKOvglXvQzpoMucH7g5CW4hi/qyQpoqCjnsslFpOjh0wRgE7W5MLFornoObKompq0rGDSTDIjgIdCmlepRSc8C3gAevc/zDwDeTcF9NEpiZj3L88jgHkjBRFs9Nmypc9zKkg9dN91mi0VqLObChgsMXRvV6ghU4fH6UXUkeBO3fUEaOR1zlGk2GImgA4svz9JrbrkFEioB7gX+O26yA50TkLRF5dLmbiMijInJYRA6HQjo0MVkcvTTGfFRx48bypF53X7P7XoZ08M6FUap8+TRXXL8k5VrZv6Gc0OQsvaPuLqFoJzPzUY71jnEgyW2/KC+HnQ2lvOGitp8MRbCUY3O5YciHgJ8ucgvdppTaj+Fa+oyI3L7UiUqpJ5VSB5RSBwKBxBY9aa5yyHQf3LAh+S/DjoZSDp/XFsH1ePviKPuby5I2P2BxQ7Px/9QW2fIc6x03BkEbkmuNAdy4sZzjl8eZi7gj1UoyFEEvEF/TsBHoW+bYh1jkFlJK9Zm/g8D3MFxNmhRx6PwobdU+yorykn7tfU1lHL88TkTnHVqS4fAs54en2Z9kJQzQXuvHl5/D4QvuGZWmGrsGQQB7m8qZi8Q41e+OvEPJUASHgDYR2SQieRid/VOLDxKRUuB9wPfjthWLiN/6DNwDnEiCTJpVoJTiaO+YLS8CGO6hK/NRHca4DFY+pn1JitaKx+sR9jWX8daFsaRfO1N45+IYmwPFlBfbMAhqLjPv4Q6LLGFFoJSKAI8DzwKngG8rpU6KyGMi8ljcoR8BnlNKxSdBqQF+IiJHgTeBf1NK/TBRmTSr48LwNGPT80kLG13MviZDwRy5NGbL9d3O2xdHyfEIuxvLbLn+/uZyzgxMEJ6N2HJ9N6OU4silsVXX5V4rdaUF1JTku6bt5yTjIkqpp4GnF2370qLvXwW+umhbD7AnGTJo1s7R3jEA9tjUETVVFFJRnMc7F8f4+E0bbLmHm3n74ijb6kqSsqJ1KfY2lRFTcOLyeFLDUzOB/vEZhsKztikCEWFvUxnvuEQR6JXFWcyRS2MU5nrZUrO++gMrYb0MbhkVpZJoTHGsd3zBhWAHVmz8MVPha65y1GyTdg2CAPY1l3NheNoVObe0Ishijl4aY1dDKTle+5rBvqYyuoJhxq/ohWXx9ITCTM9Fbe2IKn35NJQVcrR33LZ7uJUjvWPkeT1srfPbdg/L2jhyyfnzBFoRZCnz0Rgn+ibY05S8FZVLsdcc8R7XndG7OGY+j11JXNG6FHuaSrVFsARHL42xrb6E/Bx73HIAuxpKEYGjl5zf9rUiyFLODEwyF4nZNlFssbPe6OhO9Dn/ZUglxy+PU5jrpSVgj1vOYndjGZdGrrjCPZEqojHF8d5x9tqshIvzc2gJ+DjpgravFUGWcuKyOSJNsCLWSpQX59FQVrhwP43B8cvj7GwowZtgIZqV0PME13JuaIqpuSi7bHTLWexqKOW4C9q+VgRZSkf/BP78HJrKk5vaYCl2NZRqRRBHJBrjZN84uxrKbL+X5Z44pl1zC3SYi7x21JfYfq+dDaUMTswSnJyx/V6JoBVBltLRN8G2upKESyOuhl2NpZwfnmZCZyIFoDs0xcx8jF2N9ndE/oJcNlUWu8I9kSo6+ibI83psd8sB7DSVzcnLzl5hrBVBFhKLKU71T7A9BSMiuDrycvrLkCosN00qLAKAbXUlnOrXq7stOvonaKvxkZdjf/e3w7TInO4e0oogC7k4Ms3UXJTtdalRBNY8hHYPGXT0T1CY62VTVXFK7retzs/FkWldG8Kko28iZW3fl5/DpqpirQg0zsPykabKIqj05VNfWuD4lyFVnOqfoL3Wb/tEscU2s9M7PaCtguCksaI4VW0f3DFHphVBFtLRN0GOR2ittt9HarG9vtQ1mRjtRCnF6YHJhc45FVj30s/faPtAyiwC61794zOMTTs3hFcrgiyko3+C1mpfUqsyrcS2Oj89Q1PMRqIpu6cTGZyYZWx6nm02rmhdTF1pAaWFuVoRcNUa3ppCRbDVBRaZVgRZSCp9pBbttX6iMUVXMJzS+zoNqzPeWpu65y8ibK8roUNPGNPRN0FjeSGlhbkpu+fWWkPpn9GKwBm81j3MN147n24x0spweJaBiZmU+kjhasd3Oss7o1MD1og0dRYBGO6hMwMTRGPZXcO4oz/1g6Bqfz7lRbmcHnCuRZZViuC5jgH+6JnTxLL4ZbDCCFP9MmysLCI/x+PolyEVnOqfpKGskJKC1I1IwXDNzczHOD88tfLBGcr0XIRzQ1MpHwSJCFtrnR3CmxRFICL3isgZEekSkSeW2H+HiIyLyBHz53dXe24yaa/xMz0XzeqC3h39RvRCKicrAXK8HtpqfI72k6aC0/0TKZ0fsLD+39ZkaTZyZmASpVI/CALDNXpmYNKxg9CEFYGIeIEvYBSf3w48LCLblzj0VaXUXvPn99d4blJoN3112Twq7eiboL60wJbyfCuxtbYkqxXBzHyUnqGplCthgLYaHzkeyeoJ41SHTcezrc7PlfkoF0emU37v1ZAMi+Ag0KWU6lFKzQHfAh5Mwblrpq3GUARns7iGbkcKVxQvZmutn9DkLMPh2bTcP910BcNEYyqlE8UW+TlGptOsVgR9E5QU5NBQVpjyey/MkTl0IJQMRdAAXIr73mtuW8wtInJURJ4RkR1rPDcp+PJzaKoodOw/w25m5qN0h6bSYhrD1ZfBydETdrIQMZQG1xAYo1In+6ntpqPfyK8lkpqFfPFsqfEj4lxvRDIUwVJPdbEj7G1gg1JqD/CXwL+s4VzjQJFHReSwiBwOhULrlZX2Gn/WWgRnByeJxlT6LAKzAzyVtYpgkoJcDxsrU5NaYjHb60sYmJjJytoE0ZjidP9k2tp+YZ6XjZXFjo2aS4Yi6AWa4r43An3xByilJpRSYfPz00CuiFSt5ty4azyplDqglDoQCATWLeyWGj89oSnmIrF1X8OtpCOGPZ4qXz5VvnxOZ6l74vTABO01qUstsZhsXmF8YXiKK/PRtMzPWGyt9We0RXAIaBORTSKSBzwEPBV/gIjUimmPichB877Dqzk32bTX+onEFD1D2bew6exgmIJcD80V9tcgWI5tdX7OZKFFppSR8TWdHVE2KwLLC2At7koHW2tLuDAyzfRcJG0yLEfCikApFQEeB54FTgHfVkqdFJHHROQx87BfBE6IyFHgL4CHlMGS5yYq0/Vod8EqP7s4OzhJa7UvJTUIlqO9xgijy7aFTcHJWUan59PaEVX58gn48xeiZ7KJs4PGwC+V+bUWs7XOj1JXZXESOcm4iOnueXrRti/Fff4r4K9We66dbK4ywuiyURF0Doa5taUyrTJsrSthNmIsbEpFYRCncCoNOW6Wor3Gn5VpPs4OTtJUUUhRXlK6vHVhDQJO90+w1+Za4Wslq1YWA+TleNgcKM66CePxK/MMTMwshNCmi6svQ3Y9fytSbVua5mcsWqt9dAXDjl3YZBedg2G2VKe37TeVF1GU53Vk1GLWKQKA9ixc2NQVNP7eLTXpHYW3VvvweoQzDp00s4uzg5NU+/MpLUptaonFbDFX118ey57V9fPRGD1DYVrT3PY9HqG91u/IOZrsVAQ1PnpHrxCedd6kjV1YfsktabYICszKXNkWQtodmkqrf9qizewMs8k9dGF4mvmoSrtFAFdX1yvlLIssKxXBlixcYXx2cJLCXG9aVlUuxsq7ki0opegOhp2hCEwZsqntdw5a1nD6FcGWGh/jV+YJOWx1fVYqAiuO/mwWdUadg2HaatIbMWTRVu3j0ug0M/PZUaQmODlLeDbiiMnxsqI8Av58OrPIIjg7GEYkvRFDFm2mVdLlsMihrFQEjeWFFOV5syqe/ezg5EIjTDet1T6Ugp5QdqREttwwTuiIwBiVdmZT2w9O0lReRGFe6iryLYflmnOaIs5KReDxCG012eOeGJ+eJzg5m/aJYgtLIXUGs+P5d4eMl94JFgEYz78zGHacn9ouOgcnHdP2q/35+AtyHDdHk5WKAIwJ42xRBGeDzvGRAmysKsLrEce9DHbRHQzjy8+hpiQ/3aIAxqg0WyKH5qMxzg1NpT1s2kJEaK32OW4QlLWKYEuNn+GpuaxIwGVNDLY5ZFSUn+NlQ0VR1iiCrlCYlkBxWrJeLsVViyzzn//5oSkjYsghbR+MOTKntf2sVQQt1dkTRtc5GKY4zxkRQxbGqCjznz1Ad3Bqob05AStyyGkTlnZghU07ZX4MDFmGwnOMOmgQmrWKoDWQPYrg7OAkrTV+x4xIwVAExmgts7PATs4YK7qdMj8AUF6cR5UvPytCSM8OTiLinPkZYGFhW1fIOX1P1iqChrJCCnO9WaIIwgujQKfQVuMjElNcyPBi6lZklFMihiy21GSHRdYZnKS5whkRQxbWILTTQRZZ1ioCj0fYHCh2lFa2g9GpOYbCzokYsljwUzvoZbADa6DhpBEpXPVTZ3rkUOdg2FFuIbg6CHXShHHWKgIwRmndGT4qujpR7KyXYXPAqNKV6RZZdyhMjkfYUJm+GhBL0VbjJzwboX98Jt2i2MZcxIgYctogyOORheR/TiG7FUHAx+WxK0xlcM4hy/x3SuioRVFeDo3lhRnvnugOhdlQWUSu11mvWjakmjg/PEUkphzX9gGtCJyE5bfN5BWu3aEwRXle6koK0i3KNbRlQeRQVzDsOLcQXB0YOKkzSjZOW9EdT2u1j/7xGSZn5tMtCpAkRSAi94rIGRHpEpEnltj/cRE5Zv78TET2xO07LyLHReSIiBxOhjyrxWogXaHMHRV1h6bYHCh2RI6hxbRW++gJhTO2Wtl8NMaF4WlHdkRG5FBeRlsEltvXckM6Ccsi63bIIDRhRSAiXuALwH3AduBhEdm+6LBzwPuUUruBPwCeXLT/TqXUXqXUgUTlWQsbKoszfoVrt0NHpGBMGM9GYvSOTqdbFFu4MDxNJKYc+/xbq32O6YjsoDsUpqEsvVXJlsOas3NKzqdkWAQHgS6lVI9Sag74FvBg/AFKqZ8ppUbNr68DjUm4b8Lk5XjYUJm5K1yvmGkEHNsR1TgvjC6ZWDmGnGgRAGwOZHbkkGUNO5Gm8kLyvB7H9D3JUAQNwKW4773mtuX4VeCZuO8KeE5E3hKRR5c7SUQeFZHDInI4FAolJHA8rQFnTdokk54hZ4YuWlx1zWXm8+9ysGsCjHYxfmU+I9OsKKXoDjnXGs7xGiVzndL3JEMRLOV8XnKIISJ3YiiC/zdu821Kqf0YrqXPiMjtS52rlHpSKXVAKXUgEAgkKvMCrdU+s4JR5q1wtcz+lmpndkQlBbnUlORntEVQW1KAvyC95SmXo8VUUJnoHhqYmGF6Luqo1B6LcVKalWQogl6gKe57I9C3+CAR2Q38LfCgUmrY2q6U6jN/B4HvYbiaUkZrdeaucO0OGgU5NlY6UxGAFUbnDD9psukOhh2rhOGqpdidgRZZd9AcBDnUGgOj7V8anebKXPoLNCVDERwC2kRkk4jkAQ8BT8UfICLNwHeBTyilzsZtLxYRv/UZuAc4kQSZVk1rBief6w6FaSovoiDXOcvrF9NW7ac7NJVxfmrDNTHlWNcEGCtc83M89GSiIrDmZxz8/K0CTeeG0j8ITXg6XSkVEZHHgWcBL/AVpdRJEXnM3P8l4HeBSuCvzcRnETNCqAb4nrktB/jfSqkfJirTWmjJ4ORzRkfk3BERGCO28GyE4OQsNQ5c67BerPKUTp0oBmOF66aq4ox0DXWHwvjzcwj4nVEDYiniLbLt9SVplSUpcVVKqaeBpxdt+1Lc518Dfm2J83qAPYu3p5Li/BzqSwsyThHEYoqeUJjbWirTLcp1WXgZguGMUgROzTG0mJZqHycuj6dbjKTTHQqzudrnqIy7i9lUVYyIM1xzWb2y2KKl2pdxkSuXx64wG4k5erIM4upCZNjzd3roqEVLwMelkWlmI+n3UyeT7qDzreGCXC9N5UWOsMi0IsBKPjdFLINWuDqtTu5yVPvz8eXnZFzyvy6zPGW1g10TYLjmYspY/JYphGcjjqsBsRwtDgkh1YoAQxFcmY8yMJE5mRgXQkcdPioSEVoCmeen7g6FaXG4awLe7ZrLFHpcMggCQ8aeUDjtg1CtCMjMMLruUJiyolwqivPSLcqKtFT7MurZg5VsztlKGK4udsuk53/VLef8599S7WM2EuPy2JW0yqEVAZk5KuoOhmkNOH9ECsbz7x+fIZwh6cAnZ+YZnJh1/PwAGOnA60sLMsoi6w5O4fUIzRUuUAQOGYRqRQBU+fIoKcjJrJfBwcvrF2PJeS5Dnv9Vt5xLnn+GWWTdoTAbKorIy3F+9+aU1d3Of1IpQEQy6mUYm55jKDzn6FWt8VgmfKakA+92cB78pTD81JmzqK87FGazS5RwRXEeZUW5ae97tCIwaQlkjiJw24i0ucJIB26lBXA7VnnK5gpnladcjvhFfW4nEo1xfmjaNYMgEaE1kP6SuVoRmLQEfAxOzDqmYlAiuCV01CIvx8OGiqKMUcRdQWeWp1yOzRk0R9Y7eoW5aMw1bR+cMQh1R0tNAZavLhPKVnaHwuR5PTSWF6ZblFWz2QEvQ7LoDoVd4xYC50xYJgO3DYLAyA48FJ5jbDp96cC1IjBpqc6glyE4xcaqInJcMiIFw59+fmiaiMvTgVvlKd3UEdWU5FOc5037hGUyuKoI3OEagnhFnL7n756ewmaaK4rI8UhGKIIeF0UMWbQEipmLxugdTW88daJY5SndZBFkUrBEd3CKKl8eZUXOXz9j4QSLTCsCk1yvh+bKItdPWM5FYlwYcdeIFDLHInNLsrnFWJFDbsdNEUMWjWbZSq0IHIITJm0S5eLIFNGYck3UhEVLVWakA19wTbjIIgDYXFXM5bErTM+5e1Gfm9bPWOR4PWysSu8gVCuCOFoCPs4PT7naT90VdFfoqEVpUS5VvnzXK+LuoFGe0peflAzvKcNSXG62Ckam5hidnnfV/IBFa7UvrQWCtCKIoyVQzHxUccnFfmqrI3WbeQxkRPI5t0UMWVgDhx4HVMtaL261xsB4/hdGppmLpGcQmhRFICL3isgZEekSkSeW2C8i8hfm/mMisn+156aSBT+1i90TVsF0t41IwawLEQy7doXr1fKU7huRbqgswiMub/vWim5XDoJ8RNNYOz1hRSAiXuALwH3AduBhEdm+6LD7gDbz51Hgi2s4N2VYfmo3uye6Q1Oumx+waA34GL8yz8hU+uKpE2FwwihP6cYRaUGulyaXL+rrDoXJz/FQX+ae9TMW6Y4cSoZFcBDoUkr1KKXmgG8BDy465kHg68rgdaBMROpWeW7KsPzUbvWTKqXoCbpvssziauSQO5+/GwqmX4/NLq9f3B2aYlOVka7EbWxOc/K5ZCiCBuBS3Pdec9tqjlnNuQCIyKMiclhEDodCoYSFXg7DT+3OUVFocpbJ2Qibq9xpEbS4PDf+QuioCy0CMEal54bSXyRlvbgxYsiiOD+HutKCtLnmkqEIllK/i1vScses5lxjo1JPKqUOKKUOBAKBNYq4ety8sMaq+9ta7U+zJOujvrSQglyPa0NIu0Nh/C4oT7kcLdU+ZuZj9I27L1hiZj7KpZFp1yphSG/4ejIUQS/QFPe9Eehb5TGrOTeltAR8jE6700+9kHXUpXMEHo+wucrFijgYZrMLylMuhxNSHayXC8PTxJS7UkssxoqaS0ewRDIUwSGgTUQ2iUge8BDw1KJjngI+aUYP3QyMK6X6V3luSnGze6I7GKYoz0ttSUG6RVk3brbIukNh184PQFzbd6FF5sZkc4tprfalLR14wopAKRUBHgeeBU4B31ZKnRSRx0TkMfOwp4EeoAv4MvDp652bqEyJ4OaylZaP1K0jUjA6o97RK8zMR9MtypqwylO61RoDo0hKaWH6i6SsB+t93exqiyB9q+uTEmyulHoao7OP3/aluM8K+Mxqz00nDWWF5OekN+/HeukJTXHjxvJ0i5EQrdU+lIJzQ1NsqytJtzirxnKnuNkiEBHXBkt0h8I0lBVSlOe+9TMW8fm2bmutSum99criRXg8YubGd5efdHouwuWxK642jSH98dTrpdvlEUMWbk0+1x2acrU1AFDtz8eXn5MWb4RWBEuw2YWjop6FiWJ3d0SbqooRcV/yua5QmFyve8pTLkdLtY/g5CwTLqrUZ6zodm/oqMVViyz1ilgrgiVoCfi4NDLtKj91JkyWgbHCtbG80HUWWXcwzIbKYteUp1yOhZxDLnr+AxMzTM9FXT8IgvSFkLq71dpES6CYmDJC0txCd2gKjxg5Y9xOiwOKea+VrlDY1aGLFptdGDnUvZBx1/3Pv6XaR//4DOHZ1KYD14pgCdzop+4OhWmqKKIg15tuURKmJeCjx0UrXOejMS4OT7sy6+hi3Fipz+2pPeKx+p5zKbbItCJYAneOitzvI7VoCbhrheuF4SkiMZURzz/X62FDZZGrXEPWiu6AS1d0x9Nqhh93hSZTel+tCJagKC+HhrJC14yKojHFuSF3pj9eilaXJZ+zigFlgkUAhiLucknbB7M8pYtXdMfTXGEkzUt1tTKtCJZhs4uKpPSNXWE2EsuIESm4b4Wrm4sBLUVLtc+wclxSqa87mDmDoLwcDxvSkA5cK4JlsGbv3VAkxRq9ZULUBBgrXMuKcl0zKu0OhqkrdWcxoKVoDfiYjyoujjg/WCI8G2FgYiZjBkGAuY5JKwJH0FLtY3ouysDETLpFWZGFxUwZ8jIY8dTuiRzKhBj2eKwBhRvWcvRkSNh0PC3VxZwfmk6pRaYVwTJYpqYbJs26Q1OUF+VSUZyXblGShlvqF1vlKTNlfgDiEy86//kvRAy5OMfTYloCPuaiMXpTWDtdK4JlaHVRCGmmjUjBeBmGwrOMTzt7hetCecoM8VED+AtyqSnJd4VF0B2cwusRmisy5/mnI3xdK4JlCPjz8acp78da6clQRQDQPeTs5+/2qmTLkc4iKWuhOxRmQ0UReTmZ05VZg9BUKuLMeXpJRkTYXO385HNj03MMhedcnf54KRZCSB2uiDNpMVM8rdXGHI3TgyW6Q+GMidaysGqna4vAIbghJe9CVbIMexkaywvJ83ocr4i7gpmzmCmeloCPydkIoTQUSVktkWiM80PTGTcIgtTPkWlFcB1aAunJ+7EWMiXZ3GJyvB42VhU53k/dHQrTkiGLmeJpdUHkUO/oFeaimbN+Jp6Wah9dKbTIElIEIlIhIs+LSKf5+5qqKCLSJCIvicgpETkpIr8Rt+9zInJZRI6YP/cnIk+ySVfej7XQHQqT5/XQWF6YblGSjpEb37kdEWTmRD3EVcty8PPP1EEQGH/T+JXU1U5P1CJ4AnhRKdUGvGh+X0wE+C2l1DbgZuAzIrI9bv+fK6X2mj+OqVQG7qhf3B2cYmNVETkuT3+8FC0BHxdGppmLOHOF64RZnjKTQkctakrSVyRltWTq/AykPoQ30d7jQeBr5uevAR9efIBSql8p9bb5eRKjNnFDgvdNCc2VRUbeDwcrgkyMGLJoqS4mGlNcHHGmRXZ1IV/m+aitIimOtgiCU1T58iktyk23KEkn1SGkiSqCGqVUPxgdPlB9vYNFZCOwD3gjbvPjInJMRL6ylGsp7txHReSwiBwOhUIJir068nO8NKch78dqmYvEuDAynbmKYCGMzpmKwPKft9X40yyJPbRU+1Ke/GwtdGdIDYilaCgrpCDXk7I5mhUVgYi8ICInlvh5cC03EhEf8M/AZ5VSE+bmLwItwF6gH/jT5c5XSj2plDqglDoQCATWcuuEaAkUO/ZluDgyRTSmMjJqApxfF6IrGCYvx0NTBs7PgPH8ByZmmHRo2Uproj4T8XiEzVWpW8uxYpYspdQHltsnIoMiUqeU6heROiC4zHG5GErgH5VS34279mDcMV8GfrAW4VNBS8DHK2eHiMYUXo+zIkO6gpkZOmpRnJ9DXWmBY/3UncEwm6uKM3J+Bq5GDvWEptjTVJZeYRYxMjXH6PR8xrZ9MCyyI5dGU3KvRFvwU8Aj5udHgO8vPkCMuLq/A04ppf5s0b66uK8fAU4kKE/SuZr3w3mZGDMt/fFSOHmFa2dwMmPdQhDvmnPe878aMZSZ1jAYf1vv6JWU1E5PVBF8HrhbRDqBu83viEi9iFgRQLcBnwDev0SY6B+LyHEROQbcCfxmgvIkHcvt4sTOqDsYprYkc9IfL4W1sMZpK1yn5yL0jl6hLUNdE2DUv3Zq2cquDMu4uxQtAR9Kwbkh+13TCfUgSqlh4K4ltvcB95uffwIs6VNRSn0ikfungs1VVqqDKd6/Nc3CLOJscJK2msx9EcAwj8OzEYKTs9SUFKRbnAV6QlMoRUYrAqtspRMtgrODkxTmemkoy8z5GXj3HNm2uhJb75WZzs0kUl6cR2VxnuNGRbGYoisYZksGuyYg7mVwWGd0dtCoKZvpiri12pllKzsHw7TV+PA4bN4umWyqKkaElASraEWwClqqfXQ6rCO6NDrNzHyMLRneETk1cqgzGCbHI2yozFwfNUBbtZ8Lw9PMRuz3U6+Fs4OTGbmQL57CPG/KaqdrRbAKttT4ODs46Sg/9dlBqyBHZlsECytcHZbmo3MwzKaqYnIzNGLIYkutn2hMOapA0/j0PMHJ2Yy3hsG0yFIwCM3sVpwk2mv8TM5EHFW2MltcEwsrXB1mkXVlwfwMsGBxWu3NCZwNGrJkujUMZr6toTCxmL2DUK0IVoEVImiNwp1A5+AkdaUFlBRk3vL6xTgthHRmPsrFkemMt8bA8FN7PUKng9r+wiAoC55/S8DHzHyMvnF7y1ZqRbAKLBP07IBzRkWdwXBGx7DH01LtrHTgPaEpYhkeMWSRn+NlU1UxZxxkEXQOhinKy+yIIQtrHsRuRawVwSqoKM6jypfvGPM4akUMZUFHBHGK2CHPvzOYHW45iy01Pjod8uzBXMhXndkRQxapcs1pRbBK2mt9nHWIn/rSyDSzkVhWTJaBMUcDzrHIuoJhPGK4TbKBtmo/F0amuTLnjMihs4PZYw2XFeVRU5LPGZvbvlYEq6St2k/n4KTtkzarIVsmii0aywspyvNy2iGKoHMwzMbKYvJzvOkWJSW01/pRyhkhvGPTc4QmZ7NiotiivbbEdtecVgSrZEuNn+m5KJfH7J20WQ2dGZ7+eDEej9BW43eUayjTY9jjcVLkkBWwkS1tH6C9xljHFLVxEKoVwSppr3XSyzBJQ1lhRucYWkx7jc9283g1zEVinB+ezhq3HMCGymLyvB5HTBhb7182Pf8tNX6z3dm3lkMrglVihQo6IYT0rLm8Pptory1heGqOofBsWuU4P2zUgMim55/r9bA5UOyIENLOwUl8+TnUlzon75TdbK018gzZOUemFcEqKS3Mpa60IO0WQTSm6A6FsyJ0MR6nTBhbVkk2uYYAx7jmzg6Gaa32YWS3zw6MvxdbLTKtCNaAE16Gi2Yx92zykYIxYQmkfcL49MAEOR7JOkXQXuOjd/QKU2ley9EZnMyqiWIwcg5trCy21TWqFcEaaK8x8n7YOWmzEtnoIwWo8uVRUZyXdkV8qn+SloAvayKGLKyBRzqTL45MzTEUnsu6tg/GhL1jLQIRqRCR50Wk0/y9ZPF5ETlvFqA5IiKH13q+U2ir8TMbiXFxJH3VyjoXltdn16hIRGiv8affIuifYGtdNnZE6XfNXQ2bzr7n317j5/zQlG3VyhK1CJ4AXlRKtQEvmt+X406l1F6l1IF1np92LD91OqNXzgyGaSgrpDiLIoYs2mvTu5ZjfHqevvGZhcm7bKK5ooj8HE9aLbLOBWs4uwZBYARLxJR9ZUMTVQQPAl8zP38N+HCKz08pV/N+pO9lONU/YXu1IqeypcbPVBrXcpwamABgWxZaBF5zXiSdq+tPDUxSWphLrYMq1aUKu8PXE1UENUqpfgDzd/UyxyngORF5S0QeXcf5jqA4P4emisK0xVNfmYvSEwqzvT47FYE1YZwui+x0v6UIsvT51/g5YyrDdNDRN8H2upKsihiy2GjzWo4VFYGIvCAiJ5b4eXAN97lNKbUfuA/4jIjcvlZBReRRETksIodDodBaT08a7TUlafNTnxmcJKZge5Z2RJZLIF2K+FT/JBXFeVT789Ny/3Sztc7P4MQsI1NzKb93NKY4PTCRtYOgHK+Hlmr7FlWuqAiUUh9QSu1c4uf7wKCI1AGYv4PLXKPP/B0EvgccNHet6nzz3CeVUgeUUgcCgcBa/saksr2+hJ5QOC0JuDr6jNHYjix9GfwFuTSUFabPIhiYYGutPytHpAA76ksBONk3nvJ7nxuaYmY+lrWDIDCiFu2arE/UNfQU8Ij5+RHg+4sPEJFiEfFbn4F7gBOrPd9p7Kg3Jm1OpcFE7ugfx5+fQ2N55udhX4722vSs5YjGFGcGJ7NyotjC6oStAUkq6TDdctlqEYAxYdw3PsP4lfmkXztRRfB54G4R6QTuNr8jIvUi8rR5TA3wExE5CrwJ/JtS6ofXO9/JWKPxk2l4GU71T7KtPjt9pBbttX66Q2Hmo7GU3vf8sDEizcaJYovy4jzqSwvS0vY7+ibI9QotgeyLGLKwJoztCFZJKAZRKTUM3LXE9j7gfvNzD7BnLec7mYayQkoLc+lIsXkciylO9U/wSweaUnpfp7G11s981CjMk8pJ29P9xsuXrRPFFtvrS9PiGuron6Ct2k9eTvaugd1eV8ov3tBoS+h49j7VdSIi7KgvSfmo6MLINNNz0az2kcJViyzV7onTAxMLIZTZzI76EnqGppieS22qiY6+7J0otqgtLeB/fGyPLYMRrQjWwY56I3IokkL3hNXxZfvLsKnKR0GuJ+WK+FT/BJuriinIza7UEovZUV+CUqnN+RScnGEoPJv1gyA70YpgHeyoL2UuEqM7ZF9+8MV09I9nZbKzxXg9wtbakpS7J071T2a9WwiuDkRSqYj1IMh+tCJYB1cnjFPXGXX0TdBa7cv6ESkYz7+jfwKlUpNqYmJmnstjV7Iyx9Bi0jFH1pHlC/lSgVYE62BTVTH5Oal1T3T0T2jT2GR7fQmTMxF6R1OTamJhojiLQ0ct0jFH1tE3QWO5oYA09qAVwTrI8XrYWpc698RweJbBiVk9IjJJ9cIm6z7aNWGQ6jkyPQiyH60I1smO+hI6+lLjnjhljkh1R2SwtdaP1yMpG5UevzxOtT+fmixMdrYUqZwjm56LcG5oSrd9m9GKYJ3sqC9hIkXuiY5+Y0SqLQKDglwvrQEfxy+nxiI43jvOrobSlNzLDWxP4RzZ6YFJVBbn10oVWhGsk1S6Jzr6JqgrLaCiOM/2e7mFXY2lHOsdt90im5qN0BUKs6tRKwKLzSmcI9MRQ6lBK4J1YrknUrGw6djl8QXFozHY01jKyNSc7RbZyb4JlILdWhEsYM2RpaLtn7g8TmmhkWxQYx9aEayTglwvLYFi20dF49Pz9ISm2NdcZut93MbuxjIA291D1vV3atfQuzAih+y3yI5cGmNPU1lW59dKBVoRJMDOhlKO2uyeOHZ5DIA9ZsenMdha5yfXKxztHbP1Psd7x6gtKaDaryeK49ndUMrETITzw/bV756ajXB2cJK92hqzHa0IEmBfUxlD4VlbSycevTQGoH3Ui8jP8bKtroRjl+y1CI5dHtfWwBLsNS3Udy6O2naPE5fHiamr99LYh1YECbCvuRyAdy6O2XaPI5fG2Rwo1otplmB3Y6nRWdhUzH5sek675ZahrdpPcZ6XI+ZAxQ4sa2+3toZtRyuCBGiv9ZOf47FNESilOHJpjL36RViSPY1lTM5G6Bmyp6D6O2YnpxXBtXg9wu7GMnsVwaVxGssLqfJlZ2nQVKIVQQLkej3sbizlyCV7zOP+cSPr4p6mMluu73b2bzAssrcu2PP837kwikf0/Mxy7G0uo6Nvgpl5e8q2Hrk0xl7d9lNCQopARCpE5HkR6TR/ly9xTLuIHIn7mRCRz5r7Picil+P23Z+IPOlgX3M5J/ommIskf7n926b/Vb8MS7O5qpiyolzbFMHbF8fYWltiSyGQTGBvUxmRmLJlLU1wYobLY1d0208RiVoETwAvKqXagBfN7+9CKXVGKbVXKbUXuAGYxihgb/Hn1n6l1NOLz3c6e5vKmIvEFjIkJpPD50cpzPXqxTTLICLc0FxuiyKIxgy33P4NZUm/dqawz+yk7XCNHjb/pwc2ViT92pprSVQRPAh8zfz8NeDDKxx/F9CtlLqQ4H0dw/5m+9wTh86PsK+5jFyv9uAtxw0by+kOTTE6NZfU63YFw4RnIwv/X821VJcU0FRRyKHzI0m/9qHzIxTkehZSvmvsJdEepkYp1Q9g/q5e4fiHgG8u2va4iBwTka8s5VqyEJFHReSwiBwOhUKJSZ1EaksLaK4o4s1zw0m9bng2wqn+CQ5s0B3R9bjB7KjfTnIYo6XYtSK4Pgc3VnLo/GjS19IcOj/CvqZyPQhKESs+ZRF5QUROLPHz4FpuJCJ5wAPAd+I2fxFoAfYC/cCfLne+UupJpdQBpdSBQCCwllvbzsFNFbx5biSpL8M7F0eJKW0ar8TuxjJyPJJ0i+yNc8NU+/PZUFmU1OtmGjdtqmBkao7uUPIit8KzETr6Jrhxo1bCqWJFRaCU+oBSaucSP98HBkWkDsD8HbzOpe4D3lZKDcZde1ApFVVKxYAvAwcT+3PSw8FNFYxOz9MVTN7LcOi8EbGiQxevT2Gel50NpbxxLnnuCaUUr/cMc/PmSp3aYAVu3GQMVJL5/K1BkHVtjf0kanc9BTxifn4E+P51jn2YRW4hS4mYfAQ4kaA8aeEmG16GQ+dG2Fpbgr9ALyRbiVtaKjl6aYyp2UhSrnd+eJrBiVlu2qw7opXYWFlEwJ/Pm8ls+wuDIG0RpIpEFcHngbtFpBO42/yOiNSLyEIEkIgUmfu/u+j8PxaR4yJyDLgT+M0E5UkLzRVF1JQk72W4MhflrQuj3NpSmZTrZTq3tlQSiamkTVq+3mPM99y8WT//lRCRpLtGf9Y1xK6GUnw6bDdlJKQIlFLDSqm7lFJt5u8Rc3ufUur+uOOmlVKVSqnxRed/Qim1Sym1Wyn1gDXx7DaMl6GS13qGk/IyHL4wwlw0xm1tVUmQLvM5sKGCXK/wWndyJuxf7xkm4M9nc1VxUq6X6dy8qYL+8ZmkJKCbnJnnnUtjvEe3/ZSip+STxHtbqwhNznJ6YDLha/2ka4hcryy4nDTXpzDPy77mcn6WBEVgzQ/ctKlCzw+skve2GcEbr3YmHs33Rs8I0ZjitlatCFKJVgRJ4r1bjIb7ytnEX4afdg2xv7mcojxtGq+WW1sqOdE3zvj0fELXOT0wyeDELLe3OSsyzclsrCqmuaIoKW3/J11DFOR6uEGHTacUrQiSRF1pIVtqfLyS4KhoZGqOk30TvEePiNbEe1qrUApe7Urs+b98xjj/fe1aEayF97ZV8Vr3cMKpVn7SNcTBTZXk53iTJJlmNWhFkETetyXAoXOjTM+tP3rl1c4QSqHnB9bIvuZyyopy+dGp60Uwr8xLZ4JsqyuhpkQXolkLt28JMDUXTWhhX9/YFbqCYd7TqifpU41WBEnk9i0B5qKxhaiT9fBcxyBVvnyd8XKNeD3Cne3VvHQmSHSd9QkmZuZ568Iod2prYM3c2lJJjkcWLKr18MIpY4nRXdtqkiWWZpVoRZBEbtxYgS8/h+dODq588BLMRqL8+EyIu7dX4/Xoicq1cte2akan59ddNesnnUNEY4o72lfKlKJZjL8gl5s2V/DcyYF1R849d3KQlkAxLQFfkqXTrIRWBEmkINfLXduqefbkAJHo2n2lr3UPE56NcPd2PSJaD7dvCZDjEV5Yp3vomRMDlBflsl+v5l4X9+2so2doirODa19hPz49z+s9w9yzo9YGyTQroRVBkrlvZx2j0/PrWmX8fMcgRXlebm3R8wProaQgl1taKnnmRP+aR6XTcxFe6Bjkvl115OhEZ+vinh01iMDTx9e+HOilM0EiMcXPaUWQFnSLTzJ3tAcoyvPyb2t8GeajMX54YoA726spyNURE+vlgT31XBieXigzuVp+dDrIlfkoH9pdb49gWUC1v4AbN1bwwxMDaz73B8f6qSnJZ3dDqQ2SaVZCK4IkU5Dr5c6t1fzwxMCaQul+dDrI8NQcv3BDg43SZT737qylINfD996+vKbz/vVoH9X+fA7qRXwJcf/OWs4MTnJqDYWagpMzvHQmyEf2NeLRc2NpQSsCG/jYDY2MTM3x7MnVj4y+c/gS1f58vZApQfwFudy9vZZ/Pda3akU8MjXHS2dC3L+rTk/SJ8iDexvIy/HwzTcvrvqc7719mWhM8bEDjTZKprkeWhHYwO1tARrLC/mH11dXiM0YEYX46P5G7Z9OAh/d38DY9DzPdaxOEX/zzYvMRWJ8/KZmmyXLfMqL8/jgrjq+9/blVa2nUUrx7cOXOLChXEcLpRHd69iAxyP88k3NvHFuhK7gyrmH/uG1C3pElERubwuwqaqYv/lxz4qTxvPRGN947QLvaa2ircafIgkzm4/f1MzkbISnjvSteOzPuofpDk3xSweaUiCZZjm0IrCJXzrQRF6Oh79+ufu6x41fmefvf3qee3fU6hFRkvB6hEdv38zxy+MrZiR95sQAAxMzfOq2jakRLgu4YUM52+pK+JtXepi/Thi1Uor/9UIntSUFPLBXT9KnE60IbKLKl8+nbt3I9965TEff8hNnf//Tc0zORvj1u1pTKF3m85F9DVT58vmLH3UuaxXMzEf5H8+eoa3apxeRJRER4T/ds4VzQ1P806FLyx73Wvcwb54f4dN3tuhIuTSjFYGNfPqOVkoKcvmjZ04t2RmdH5riyVd6uGd7DTvqddhcMinI9fIbH2jj9Z4RvvNW75LHfPmVHi6OTPO5B3boSeIk8/6t1dy4sZz/+UInEzPXZoSdmY/y+z/ooLakQLuFHEBCikBEPiYiJ0UkJiIHrnPcvSJyRkS6ROSJuO0VIvK8iHSavzMq92xpUS6/+YE2Xu0cusZFNBeJ8X9/6x1yvR4+98CONEmY2Xz8YDMHN1Xw337QQe/ou4umvHVhlC+83MV9O2t17nsbEBH+8we3MzY9x298851r8j/90dOnOD0wyR99dJe2BhxAohbBCeCjwCvLHSAiXuALGMXrtwMPi8h2c/cTwItKqTbgRfN7RvHIrRt5cG89f/LsGf7mx93MRWKEJmf51a8d4ljvOJ//6C7qywrTLWZG4vEI//0XdqOAX/ziaxy9NIZSilc7Q3zq79+ktqSA339wZ7rFzFj2NpXxXx/YwUtnQvz2d44yOjXHbCTKH/ygg6+9doFfe88m7tyqXXJOQJJRWlFEXgb+k1Lq8BL7bgE+p5T6OfP77wAopf5IRM4Adyil+s1C9i8rpdpXut+BAwfU4cPX3MqxzMxH+fQ/vs2PTgcpzPUyE4mS5/Xwew/s4KGDOmTRbk4PTPDIV95kcGIWX34O4dkIDWWF/NN/uJnG8qJ0i5fx/NlzZ/jCy914PUIspojEFJ+8ZQP/5YPbycvR3ulUIiJvKaWu8d6kogRWAxA/Y9QL3GR+rrHqFJvKYNnhgYg8CjwK0Nzsrs6zINfL3z1ygJfPhnjpdJDyojx+fnedDldMEVtrS/jBr7+XH54c4ETvOAc2lvOhPfXaJZEi/uM97Xxwdz3/dOgSBbkebmmpXChvqXEGKyoCEXkBWCoT1H9WSn1/FfdYahZuzWaIUupJ4EkwLIK1np9uRIx8+Xfq6JS0EPDn84mbN6RbjKylvdbP735o+8oHatLCiopAKfWBBO/RC8SHBTQC1kqTQRGpi3MNJVZeSqPRaDRrJhUOukNAm4hsEpE84CHgKXPfU8Aj5udHgNVYGBqNRqNJIomGj35ERHqBW4B/E5Fnze31IvI0gFIqAjwOPAucAr6tlDppXuLzwN0i0gncbX7XaDQaTQpJStRQqnFb1JBGo9E4geWihnTslkaj0WQ5WhFoNBpNlqMVgUaj0WQ5WhFoNBpNluPKyWIRCQGrK/91LVXAUBLFSRZarrWh5VobWq614VS5IDHZNiilrlnW7UpFkAgicnipWfN0o+VaG1qutaHlWhtOlQvskU27hjQajSbL0YpAo9FospxsVARPpluAZdByrQ0t19rQcq0Np8oFNsiWdXMEGo1Go3k32WgRaDQajSYOrQg0Go0my8lIRSAiHxORkyISE5Flw6xE5F4ROSMiXSLyRNz2ChF5XkQ6zd/lSZJrxeuKSLuIHIn7mRCRz5r7Picil+P23Z8quczjzovIcfPeh9d6vh1yiUiTiLwkIqfM//lvxO1L6vNarr3E7RcR+Qtz/zER2b/ac22W6+OmPMdE5Gcisidu35L/0xTJdYeIjMf9f353tefaLNdvx8l0QkSiIlJh7rPleYnIV0QkKCInltlvb9tSSmXcD7ANaAdeBg4sc4wX6AY2A3nAUWC7ue+PgSfMz08A/z1Jcq3puqaMAxiLQAA+h1EbOtnPa1VyAeeBqkT/rmTKBdQB+83PfuBs3P8xac/reu0l7pj7gWcwqvLdDLyx2nNtlutWoNz8fJ8l1/X+pymS6w7gB+s51065Fh3/IeBHKXhetwP7gRPL7Le1bWWkRaCUOqWUOrPCYQeBLqVUj1JqDvgW8KC570Hga+bnrwEfTpJoa73uXUC3Umq9q6hXS6J/b9qel1KqXyn1tvl5EqPmRUOS7h/P9dpLvLxfVwavA2ViVN5bzbm2yaWU+plSatT8+jpGlUC7SeRvTuvzWsTDwDeTdO9lUUq9Aoxc5xBb21ZGKoJV0gBcivvey9UOpEYp1Q9GRwMkq9DwWq/7ENc2wsdN0/AryXLBrEEuBTwnIm+JyKPrON8uuQAQkY3APuCNuM3Jel7Xay8rHbOac+2UK55fxRhZWiz3P02VXLeIyFEReUZEdqzxXDvlQkSKgHuBf47bbNfzWglb29aKNYudioi8ANQuses/K6VWU/JSltiWcCzt9eRa43XygAeA34nb/EXgDzDk/APgT4FfSaFctyml+kSkGnheRE6bI5l1k8Tn5cN4YT+rlJowN6/7eS11iyW2LW4vyx1jS1tb4Z7XHihyJ4YieE/c5qT/T9cg19sYbs+wOX/zL0DbKs+1Uy6LDwE/VUrFj9Ttel4rYWvbcq0iUEp9IMFL9AJNcd8bgT7z86CI1Cml+k3zK5gMuURkLde9D3hbKTUYd+2FzyLyZeAHqZRLKdVn/g6KyPcwzNJXSPPzEpFcDCXwj0qp78Zde93Pawmu115WOiZvFefaKRcishv4W+A+pdSwtf06/1Pb5YpT2CilnhaRvxaRqtWca6dccVxjkdv4vFbC1raVza6hQ0CbiGwyR98PAU+Z+54CHjE/PwKsxsJYDWu57jW+SbMztPgIsGSEgR1yiUixiPitz8A9cfdP2/MSEQH+DjillPqzRfuS+byu117i5f2kGeFxMzBuurRWc65tcolIM/Bd4BNKqbNx26/3P02FXLXm/w8ROYjRHw2v5lw75TLlKQXeR1ybs/l5rYS9bSvZs99O+MF46XuBWWAQeNbcXg88HXfc/RhRJt0YLiVreyXwItBp/q5IklxLXncJuYowXojSRed/AzgOHDP/2XWpkgsjKuGo+XPSKc8Lw82hzGdyxPy5347ntVR7AR4DHjM/C/AFc/9x4iLWlmtrSXpOK8n1t8Bo3PM5vNL/NEVyPW7e9yjGJPatTnhe5vd/B3xr0Xm2PS+MQV8/MI/Rd/1qKtuWTjGh0Wg0WU42u4Y0Go1Gg1YEGo1Gk/VoRaDRaDRZjlYEGo1Gk+VoRaDRaDRZjlYEGo1Gk+VoRaDRaDRZzv8Py14JtdBjnPcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.linspace(-1, 1, 200)[:, None]\n",
    "y = np.sin(10*x).squeeze()\n",
    "plt.plot(x,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe81a040",
   "metadata": {},
   "source": [
    "Try to fit the MLP with all default parameters. Are you satisfied with the result?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "b067de4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bffd327",
   "metadata": {},
   "source": [
    "There are several things we can do to get a better fit. Let's go through the options one by one. Let's focus first on the hyper-parameters we know.\n",
    "\n",
    "- What is the default activation. Do you see an improvement if you revert it to the smooth functions we have see so far?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "90bc7b1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5ac7345",
   "metadata": {},
   "source": [
    "- Try to increase/decrease the number of hidden layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "2d55ae22",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44fa86bc",
   "metadata": {},
   "source": [
    "As you can see in the documentation, the choice of solver can make a big difference. DO you confirm this is the case for your dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "929454ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6779378b",
   "metadata": {},
   "source": [
    "With all these options, you should be able to find a very small neural network that fits your data better than the default options. What is your minimal neural network?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "3c394d13",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "043b0fa1",
   "metadata": {},
   "source": [
    "## Neural network for classification\n",
    "\n",
    "Neural networks are also used for classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "87783dce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAACXCAYAAAARS4GeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAALBklEQVR4nO3dX2yd510H8O+vi8ooW2tnE0wU1sSdBAK0mqZTmZBQqjnSuJgcMRJNG2iuNCXiBiJx4dxAHY2hBCHkCooWEGoZMFgjIJ0mFdSIuqMXgGLhTipsF21amNikQp1uHfsjwcvFcUbUpmnzvufkxE8+HymSz+n5vs9j95dzvnlfH7u6rgsAQMtumPYGAAAmTeEBAJqn8AAAzVN4AIDmKTwAQPMUHgCgeU0Xnqp6tKo+Ou7Hcn0xRwxlhhgHczRMXWs/h6eqXr7o5k1Jvp3kf7ZuH+667s+u/q7Gq6rel+SBJO9M8o9Jlrque366u2pL63NUVTcm+XSSu5LcluSeruvWprqpxlwHM/RTST6eZE9Gn9dakl/uuu4r09xXa66DOfqxJJ9KcvvWXesZzdG/TG9Xl3bNneHpuu4tF/4k+bckH7jovu8ORlXtmN4u+6uqtyf5qyS/lmRnkrNJPjPVTTWo9Tna8mSSX0jy1WlvpEXXwQzNJvmDJLsyKs1fT/LgNDfUoutgjv4jyc9n9Hr29iSfTfIXU93Ra7jmCs9rqaq9VfXlqlquqq8mebCqZqvqc1X1QlVtbn38Qxdl1qrqY1sfL1XVk1X121uPPVdVP9vzsbur6vNV9fWqOlNVD1TVn77BT+Xnkjzddd2pruu+lWQlyR1V9aPDv0q8nlbmqOu673Rdt9p13ZP5/38tchU0NEOPbj0Pfa3ruv9O8ntJfnpMXyZeR0NzdL7ruue60eWiyuj56F3j+SqN17YpPFvekVGLvC3JoYz2/+DW7Xcm+WZGf2lfy91JvpRRC/2tJH9UVdXjsZ9O8k9J3pZRYfnFi4NV9YWq+vBrHPfHkzx14UbXdd9I8szW/VwdLcwR09XiDP1Mkqff4GMZj2bmqKrOJ/lWkt9N8puXe+y0bLdTaP+b5L6u6769dfubSf7ywn+sqk8kefwy+ee7rvvDrcf+cZLfT/IDufQlgUs+tkbfO/GeJO/ruu47SZ6sqs9eHOy67t2X2cNbkrzwivteSvLWy2QYrxbmiOlqaoaq6t1Jfj3J4ht5PGPTzBx1XTdTVd+X5KNJrsnvSd1uZ3he2LoMlCSpqpuq6mRVPV9VX0vy+SQzVfWm18h/dwi2TuEmowJyJY/9wSQvXnRfkvz7FXwOLye5+RX33ZzR9XOujhbmiOlqZoaq6l1JHk3yK13X/f2V5hmkmTnaOu43knwyyaeq6vv7HGOStlvheeVbyn41yY8kubvrupszOiWbjK4jTspXkuysqpsuuu+HryD/dJI7LtzYasS3x6nkq6mFOWK6mpihqrotyZkkH++67k/GuTnekCbm6BVuyOjdaLcO2tUEbLfC80pvzegU4Pmq2pnkvkkvuPX28bNJVqrqxqp6b5IPXMEh/jrJT1TVB6vqzRmdRv5C13VfnMB2eWO24xylqr5na4aS5MaqevNlrt8zWdtuhqrq1iR/l+SBrus+OaFtcmW24xztq6qfrKo3VdXNSX4nyWaSf53Mjvvb7oVnNcn3JvnPJP+Q5G+u0rofSfLeJP+V5Dcyelv5hWuwqaqnq+ojlwp2XfdCkg8m+URGQ3F3kg9NesNc1mq22Rxt+VJGT463JvnbrY9vm9huuZzVbL8Z+liSuST3VdXLF/5MesNc1mq23xzNJPnzjL4X9ZmM3qH1/osv1V0rrrkfPLgdVdVnknyx67qJt3HaZY4YygwxDq3O0XY/wzMVVfWeqrq9qm6oqvdn9M6G01PeFtuMOWIoM8Q4XC9ztN3eln6teEdGPy35bUm+nOSXuq775+luiW3IHDGUGWIcros5ckkLAGieS1oAQPNe75LWVE7/nDp1alB+eXm5d3bfvn29s8ePH++dnZ2d7Z0dg0m/lXlbnkbcu3dv7+z58+d7Z48dO9Y7u7g41R+UO8k52pYztLa21ju7f//+3tn5+fne2SF7HoMmn4tOnDgxKH/06NHe2d27d/fOrq+v985ei69pzvAAAM1TeACA5ik8AEDzFB4AoHkKDwDQPIUHAGiewgMANE/hAQCap/AAAM1TeACA5ik8AEDzFB4AoHkKDwDQPIUHAGjejmlv4FKWl5cH5c+dO9c7u7m52Tu7c+fO3tmHH364dzZJDhw4MCjPq83MzPTOPvHEE72zjz/+eO/s4uJi7yyvtrGxMSh/zz339M7ecsstvbPPPfdc7yyXdvTo0d7Zoc/vJ0+e7J09fPhw7+z6+nrv7MLCQu/spDjDAwA0T+EBAJqn8AAAzVN4AIDmKTwAQPMUHgCgeQoPANA8hQcAaJ7CAwA0T+EBAJqn8AAAzVN4AIDmKTwAQPMUHgCgeTsmdeAhv1b+3Llzg9Z+5plnemfn5uZ6Z/ft29c7O+TrlSQHDhwYlG/RxsbGoPza2tpY9nGl5ufnp7Iur3b69OlB+TvuuKN3dv/+/b2zx44d653l0g4dOtQ7u7y8PGjtPXv29M7u3r27d3ZhYaF39lrkDA8A0DyFBwBonsIDADRP4QEAmqfwAADNU3gAgOYpPABA8xQeAKB5Cg8A0DyFBwBonsIDADRP4QEAmqfwAADNU3gAgOYpPABA83ZM6sCbm5u9s3feeeegtefm5gbl+9qzZ89U1m3Z6upq7+zKysqgtV966aVB+b727t07lXV5tSNHjgzK79q1ayprLy4u9s5yaUNeV5599tlBa587d653dmFhoXd2yOv47Oxs7+ykOMMDADRP4QEAmqfwAADNU3gAgOYpPABA8xQeAKB5Cg8A0DyFBwBonsIDADRP4QEAmqfwAADNU3gAgOYpPABA8xQeAKB5OyZ14CG/Vn7fvn1j3MnVM+Rznp2dHeNO2nHkyJHe2aWlpUFrT+v/yfnz56eybquGfD1XV1cHrX369OlB+b4eeuihqazLpc3NzQ3Kv/jii72zCwsLU8meOXOmdzaZzPOvMzwAQPMUHgCgeQoPANA8hQcAaJ7CAwA0T+EBAJqn8AAAzVN4AIDmKTwAQPMUHgCgeQoPANA8hQcAaJ7CAwA0T+EBAJq3Y1IHHvKr3dfX18e4kyuzubnZO3v27Nne2YMHD/bO0paNjY3e2fn5+bHtoxUrKyu9s/fff//4NnKFTp8+3Ts7MzMztn0wfUNeT8+cOdM7e/jw4d7ZEydO9M4myfHjxwflL8UZHgCgeQoPANA8hQcAaJ7CAwA0T+EBAJqn8AAAzVN4AIDmKTwAQPMUHgCgeQoPANA8hQcAaJ7CAwA0T+EBAJqn8AAAzdsxqQPPzc31zp49e3bQ2qdOnZpKdojl5eWprAutW1pa6p1dW1sbtPZTTz3VO7t///7e2cXFxd7Ze++9t3d26NqtOnr06KD8wsJC7+zm5mbv7GOPPdY7e/Dgwd7ZSXGGBwBonsIDADRP4QEAmqfwAADNU3gAgOYpPABA8xQeAKB5Cg8A0DyFBwBonsIDADRP4QEAmqfwAADNU3gAgOYpPABA8xQeAKB5OyZ14Lm5ud7ZEydODFp7eXm5d/auu+7qnV1fX++dZfxmZmYG5RcXF3tnH3nkkd7ZtbW13tmlpaXe2VbNz8/3zm5sbAxae0h+ZWWld3bI/O3atat3Nhn296ZVs7Ozg/KHDh0a006uzMGDB3tnT548OcadjIczPABA8xQeAKB5Cg8A0DyFBwBonsIDADRP4QEAmqfwAADNU3gAgOYpPABA8xQeAKB5Cg8A0DyFBwBonsIDADRP4QEAmldd1017DwAAE+UMDwDQPIUHAGiewgMANE/hAQCap/AAAM1TeACA5v0fWRndI4po5XUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x216 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# degraded version of the original MNIST data set\n",
    "from sklearn import datasets\n",
    "digits = datasets.load_digits()\n",
    "\n",
    "_, axes = plt.subplots(nrows=1, ncols=4, figsize=(10, 3))\n",
    "for ax, image, label in zip(axes, digits.images, digits.target):\n",
    "    ax.set_axis_off()\n",
    "    ax.imshow(image, cmap=plt.cm.gray_r, interpolation=\"nearest\")\n",
    "    ax.set_title(\"Training: %i\" % label)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4c51a0a",
   "metadata": {},
   "source": [
    "Use this data set to train a neural network to recognize hand writen digits. Look at the [Documention](https://scikit-learn.org/stable/modules/neural_networks_supervised.html) to get help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3847b4c7",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "- https://github.com/MorvanZhou/simple-neural-networks\n",
    "- https://www.manning.com/books/deep-learning-with-python-second-edition\n",
    "- http://neuralnetworksanddeeplearning.com/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e186998",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "***\n",
    "## Credit\n",
    "\n",
    "[//]: # \"This notebook is part of [E4C Interdisciplinary Center - Education](https://gitlab.in2p3.fr/energy4climate/public/education).\"\n",
    "Contributors include Bruno Deremble and Alexis Tantet.\n",
    "\n",
    "<br>\n",
    "\n",
    "<div style=\"display: flex; height: 70px\">\n",
    "    \n",
    "<img alt=\"Logo LMD\" src=\"images/logos/logo_lmd.jpg\" style=\"display: inline-block\"/>\n",
    "\n",
    "<img alt=\"Logo IPSL\" src=\"images/logos/logo_ipsl.png\" style=\"display: inline-block\"/>\n",
    "\n",
    "<img alt=\"Logo E4C\" src=\"images/logos/logo_e4c_final.png\" style=\"display: inline-block\"/>\n",
    "\n",
    "<img alt=\"Logo EP\" src=\"images/logos/logo_ep.png\" style=\"display: inline-block\"/>\n",
    "\n",
    "<img alt=\"Logo SU\" src=\"images/logos/logo_su.png\" style=\"display: inline-block\"/>\n",
    "\n",
    "<img alt=\"Logo ENS\" src=\"images/logos/logo_ens.jpg\" style=\"display: inline-block\"/>\n",
    "\n",
    "<img alt=\"Logo CNRS\" src=\"images/logos/logo_cnrs.png\" style=\"display: inline-block\"/>\n",
    "    \n",
    "</div>\n",
    "\n",
    "<hr>\n",
    "\n",
    "<div style=\"display: flex\">\n",
    "    <a rel=\"license\" href=\"http://creativecommons.org/licenses/by-sa/4.0/\"><img alt=\"Creative Commons License\" style=\"border-width:0; margin-right: 10px\" src=\"https://i.creativecommons.org/l/by-sa/4.0/88x31.png\" /></a>\n",
    "    <br>This work is licensed under a &nbsp; <a rel=\"license\" href=\"http://creativecommons.org/licenses/by-sa/4.0/\">Creative Commons Attribution-ShareAlike 4.0 International License</a>.\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": true,
   "autocomplete": false,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}