![\"gradient_descent\"](\"images/gradient_descent.png\")
"
]
},
{
"cell_type": "markdown",
"id": "d368cd0e",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"> - Try different values of the regularization parameter $\\alpha$ in `MLPRegressor`.\n",
"> - What is the optimal value of $\\alpha$ for a well trained model?"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "31bea560",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# your code here\n"
]
},
{
"cell_type": "markdown",
"id": "4f5b625e",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"In a similar way as the LASSO regularization, we can use the L1 norm to penalize the coefficients of the neural network\n",
"\n",
"\\begin{equation}\n",
"C = C_0 + \\alpha \\sum_i | w_i |\\,\n",
"\\end{equation}\n",
"\n",
"Also like in LASSO, this strategy will actually set some weights exactly equal to zero (cf. discussion in the regularization notebook)."
]
},
{
"cell_type": "markdown",
"id": "7fb96a93",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Work with small networks\n",
"\n",
"With the ridge and LASSO techniques, we reduce the size of the coefficients for some neurons. For our simple network that has only one single hidden layer, it means that some neurons will actually be completely useless. A strategy to get the same result would be to train a smaller network."
]
},
{
"cell_type": "markdown",
"id": "25c32381",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Le's first analyze the magnitude of the weights in your best neural network with L2 regularization.\n",
"\n",
"> - Plot the magnitude of the weights of the hidden layer and the weights of the output layer.\n",
"> - How many coefficients are significantly different from 0? \n",
"> - This should give you an indication for how to build a lightweight network that has the same behavior as the regularized network.\n",
"> - Build this new network and comment the results.\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "843bf607",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# your code here"
]
},
{
"cell_type": "markdown",
"id": "d7a786bc",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Ensemble of Network\n",
"\n",
"Another strategy to regularize the solution is to train several networks. If choose different initial conditions for all networks, we are almost sure that each network will converge to a different solution. Each network may overfit the data but in a different way and so by averaging the output of all trained network, we can get a more generic (less overfitting) solution."
]
},
{
"cell_type": "markdown",
"id": "24ff5af3",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Neuron dropout\n",
"\n",
"Depending on the problem we are working on, the solution of ensemble of networks may be too expensive. A similar approach is to temporarily alter the network by randomly deleting neurons. At each step of the gradient descent we remove a fraction of the neurons and rescale the other neurons so that the weighted sum remains on the same order of magnitude. \n",
"\n",
"Neuron dropout is a very popular approach and seem to really improve the regularization even if this approach does not rely on solid mathematical background. It is unfortunately not implemented in scikit-learn (see [here](10_intro_deep_learning.ipynb#Towards-more-advanced-libraries) for libraries dedicated to neural networks)."
]
},
{
"cell_type": "markdown",
"id": "1c16ca7c",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Increase sample size\n",
"\n",
"Of course, another obvious strategy is to train a model with enough samples. While this is not always possible because we are limited by our observations, we can test the idea here for our synthetic data set.\n",
"\n",
"> - Increase the size of the training data set while keeping 20 neurons in the hidden layer. When the training data set is big enough, do you still overfit the data?"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "85d25e17",
"metadata": {},
"outputs": [],
"source": [
"# your code here"
]
},
{
"cell_type": "markdown",
"id": "d0cd2d86",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## Deep networks\n",
"\n",
"The universal approximation theorem guarantees that we can approximate any smooth function with one single hidden layer provided that this layer holds enough neurons. At the same time, the idea of adding extra layer was driven by our intuition that a network would naturally segment the problem in task and subtask (for image recognition, that could be edge recognition, pattern recognition, etc.). This strategy is also inspired by the way we build computer softwares: we first build modules to handle elementary operations and then build on top of these modules more elaborate functions. In the end we only use *high level* computer program with a cascade of layers of programs behind.\n"
]
},
{
"cell_type": "markdown",
"id": "99e19d04",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"However, as you may have noticed for the perceptron experiment, the optimal weights do not carry any particular structure. And the absence of structure persists even when we train a network with more layers. Probably guided by the fact that there *should be* some structure in the weights, Yann LeCun and colleagues proposed in the late 90' to modify the architecture of neural networks. More specifically, they proposed to change the *weighted sum* in the activation function by a *convolution*."
]
},
{
"cell_type": "markdown",
"id": "4d4ef1aa",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Convolution\n",
"\n",
"Convolution is the main operation that we perform to apply a filter in image processing.\n",
"\n",
"Since we only work with discrete convolution, let me skip the mathematical definition of convolution and write instead its discrete 2D form: the discrete expression of a 2D convolution is given by\n",
"\n",
"\n",
"\\begin{equation}\n",
"g[i,j]=(f*\\omega)[i,j]=\\sum _{m=-\\infty}^{\\infty}{\\sum_{n=-\\infty}^{\\infty}{f [i,j]\\omega[i-n,j-m]}},\n",
"\\end{equation}\n",
"\n",
"where $g$ is the filtered image, $f$ is the original image, $\\omega$ is the filter kernel. Notice that convolution is commutative: $f*\\omega$ = $\\omega*f$. However in practice one of the two function is only non-zero on a small area compared to the other image (at least in image filtering). We call this function the kernel."
]
},
{
"cell_type": "markdown",
"id": "c3ab198f",
"metadata": {},
"source": [
"Below is an example of the convolution of the matrix I by the kernel K. The operation of convolution consists in sliding the kernel over the image and taking the weighted sum as indicated in the figure.\n",
"\n",
"![\"convolution\"](\"images/convolution-operation-in-2D.png\")
\n"
]
},
{
"cell_type": "markdown",
"id": "a6f91240",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### Application of convolution to blur\n",
"\n",
"Below is an image of the Deep Water Horizon oil spill. There are many information in this picture and we may be interested in knowing the position of the oil spill, finding how many boats are present in this figure, etc. Let's see how we can extract information from this image with convolutions.\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "e7552dd0",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"![\"alexnet\"](\"images/alexnet.png\")
\n"
]
},
{
"cell_type": "markdown",
"id": "bbbdb6c9",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Issue with deep networks\n",
"\n",
"Convolutional neural networks are usually deeper than fully connected networks because we want to do several convolutions and finally aggregate the results in the final layer.\n",
"\n",
"The problem of deep networks is that they are hard to train. Let's illustrate this point with a minimalist fully connected network. "
]
},
{
"cell_type": "markdown",
"id": "b3e223b7",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"The network below is a line of $L$ neurons connected between the input and the output (there are $L-2$ hidden layers with $1$ neuron per layer). This network cannot predict much but it will help us understand why it is hard to train a deep networks.\n",
"\n",
"> - Read the code below and train this network with $L=5$."
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "577a3b9d",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# network hyper parameters\n",
"L = 5 # total length of the network (including input and output)\n",
"eta = 1e-2 # learning rate\n",
"n_epoch = 100 # number of iterations\n",
"\n",
"# activation functions\n",
"def sigmoid(x):\n",
" return (1./(1.+np.exp(-x)))\n",
"\n",
"def sigmoid_prime(x):\n",
" return sigmoid(x)*(1.-sigmoid(x))\n",
"\n",
"\n",
"# network structure\n",
"w = np.random.rand(L)\n",
"b = np.random.rand(L)\n",
"\n",
"a = np.zeros(L)\n",
"z = np.zeros(L)\n",
"d = np.zeros(L)\n",
"\n",
"dw = np.zeros(L)\n",
"db = np.zeros(L)\n",
"\n",
"# input and output\n",
"x = a[0]= 0.5\n",
"y = np.sin(x)\n",
"\n",
"# training loop\n",
"for i in range(0,n_epoch):\n",
" for l in range(1,L):\n",
" z[l] = w[l]*a[l-1] + b[l]\n",
" a[l] = sigmoid(z[l])\n",
"\n",
" C = 0.5*(a[-1]-y)**2\n",
"\n",
" d[-1] = np.sqrt(2*C)\n",
" for l in range(L-2,0, -1):\n",
" d[l] = d[l+1]*w[l+1]*sigmoid_prime(z[l])\n",
"\n",
" dw[1:] = d[1:]*a[:-1]\n",
" db[1:] = d[1:]\n",
"\n",
" # update weights\n",
" w -= eta*dw \n",
" b -= eta*db"
]
},
{
"cell_type": "markdown",
"id": "371e8357",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"At each iteration, all the weights are adjusted according to the gradient descent algorithm.\n",
"\n",
"> - Can you monitor the evolution of the magnitude of the gradient of the cost function with respect to all weights?\n",
"> - Plot this information. It may hep to use a log scale for the $y$ axis to see the evolution (you can do this with `plt.semilogy()`)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "65aad949",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# your code here"
]
},
{
"cell_type": "markdown",
"id": "0f43f4d0",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"> - Derive an analytical expression for the gradient of the cost function with respect to all weights (you can use the recurrence relation that we derived in the [previous notebook](9_neural_networks.ipynb) or you can use the recurrence relation in the code above)\n",
"> - What is the maximum value of the derivative of the sigmoid?\n",
"> - If we initialize the weights with random numbers between 0 and 1, can you characterize the magnitude of the gradient as you go from the input layer to the output layer with the help of the recurrence relation?\n",
"\n",
"This problem is called the *vanishing gradient problem*."
]
},
{
"cell_type": "markdown",
"id": "3a5b2107",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"As you observed, the weights of a deep network will evolve much slower for a layer that is near the input layer during the training phase. You either need to train your model for long enough or use more elaborate methods than simple gradient descent. Another possibility is to use the ReLU activation function instead of the sigmoid.\n",
"\n",
"> - Can you explain why the ReLU activation function does not suffer the vanishing gradient problem?\n",
"\n",
"We describe below two modern libraries which handle all these points and which will help you tackle the problem of deep learning."
]
},
{
"cell_type": "markdown",
"id": "fa683ccc",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Towards more advanced libraries\n",
"\n",
"We are now at the limit of what scikit-learn can do for neural networks. If you want to study more advanced neural networks you should probably start to use [Keras](https://keras.io/) or [PyTorch](https://pytorch.org/). Note that Keras is built on top of [TensorFlow](https://www.tensorflow.org/) which is a library developed by Google to handle tensors and gradients (via automatic differentiation). Keras provides a very clean and very intuitive way to work with Neural networks. PyTorch on the other hand is more research oriented. You have full control on the architecture of the neural network. It is actively developed by FaceBook.\n",
"\n",
"![\"keras\"](\"images/keras_logo.svg.png\"){kind=logo}
\n",
"\n",
"![\"PyTorch\"](\"images/pytorch-logo.png\"){kind=logo}
\n"
]
},
{
"cell_type": "markdown",
"id": "72fa0f09",
"metadata": {},
"source": [
"Take a moment to look at the Keras and PyTorch implementation for the image classification problem\n",
"\n",
"- Keras implementation: https://keras.io/examples/vision/mnist_convnet/\n",
"\n",
"- PyTorch implementation: https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html"
]
},
{
"cell_type": "markdown",
"id": "27193d2e",
"metadata": {},
"source": [
"### Further reading\n",
"\n",
"-Autoencoder: How to use neural networks to do a Principal Component Analysis"
]
},
{
"cell_type": "markdown",
"id": "631f9b78",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## References\n",
"\n",
"- Deep Learning, *Goodfellow, Bengio, Courville* (2016), [Chap 7](https://www.deeplearningbook.org/contents/regularization.html) and [Chap 9](https://www.deeplearningbook.org/contents/convnets.html)\n",
"- Neural Networks and Deep Learning, *Nielsen* (2019) [Chap. 3](http://neuralnetworksanddeeplearning.com/chap3.html)\n",
"- Pattern recognition and machine learning, *Bishop C. M.* (2006) Springer (Chap. 5)"
]
},
{
"cell_type": "markdown",
"id": "5e186998",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"***\n",
"## Credit\n",
"\n",
"[//]: # \"This notebook is part of [E4C Interdisciplinary Center - Education](https://gitlab.in2p3.fr/energy4climate/public/education).\"\n",
"Contributors include Bruno Deremble and Alexis Tantet.\n",
"\n",
"\n", "\n", "
\n",
" \n",
"![\"Logo](\"images/logos/logo_lmd.jpg\")
\n",
"\n",
"![\"Logo](\"images/logos/logo_ipsl.png\")
\n",
"\n",
"![\"Logo](\"images/logos/logo_e4c_final.png\")
\n",
"\n",
"![\"Logo](\"images/logos/logo_ep.png\")
\n",
"\n",
"![\"Logo](\"images/logos/logo_su.png\")
\n",
"\n",
"![\"Logo](\"images/logos/logo_ens.jpg\")
\n",
"\n",
"![\"Logo](\"images/logos/logo_cnrs.png\")
\n",
" \n",
"
\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
" \n",
"\n", "\n", "
\n",
" ![\"Creative](\"https://i.creativecommons.org/l/by-sa/4.0/88x31.png\")
\n",
"
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.\n", "
"
]
}
],
"metadata": {
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
},
"latex_envs": {
"LaTeX_envs_menu_present": true,
"autoclose": true,
"autocomplete": false,
"bibliofile": "biblio.bib",
"cite_by": "apalike",
"current_citInitial": 1,
"eqLabelWithNumbers": true,
"eqNumInitial": 1,
"hotkeys": {
"equation": "Ctrl-E",
"itemize": "Ctrl-I"
},
"labels_anchors": false,
"latex_user_defs": false,
"report_style_numbering": false,
"user_envs_cfg": false
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
},
"varInspector": {
"cols": {
"lenName": 16,
"lenType": 16,
"lenVar": 40
},
"kernels_config": {
"python": {
"delete_cmd_postfix": "",
"delete_cmd_prefix": "del ",
"library": "var_list.py",
"varRefreshCmd": "print(var_dic_list())"
},
"r": {
"delete_cmd_postfix": ") ",
"delete_cmd_prefix": "rm(",
"library": "var_list.r",
"varRefreshCmd": "cat(var_dic_list()) "
}
},
"types_to_exclude": [
"module",
"function",
"builtin_function_or_method",
"instance",
"_Feature"
],
"window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 5
}
\n",
" This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.\n", "