We will solve a regression problem using neural network. \n", " \n", "
Data is generated from \\begin{equation}y=\\sin(x) + \\epsilon, \\end{equation} where $\\epsilon \\sim N(0, 0.1)$ denotes a random noise." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Dataset generation" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2022-05-26T11:29:42.441872Z", "start_time": "2022-05-26T11:29:42.215828Z" }, "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "N = 10 # Number of samples\n", "x = torch.linspace(0, 2*np.pi, N).view(-1,1) # Input data\n", "y = # Target data\n", "\n", "plt.plot(x, y, 'bo', label='Data points')\n", "plt.plot(np.linspace(0,2*np.pi,100), np.sin(np.linspace(0,2*np.pi,100)), 'r', label='Target Function')\n", "plt.legend(fontsize=13)\n", "plt.title('Regression', fontsize=15)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Neural Network" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2022-05-26T11:29:43.627206Z", "start_time": "2022-05-26T11:29:43.622598Z" }, "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "# Build a neural network\n", "\n", "class model(nn.Module) :\n", " def __init__(self, hidden_dims) : # Hidden_dims : [h1, h2, h3, ..., hn]\n", " super(model, self).__init__()\n", " \n", " self.layers = []\n", " for i in range(len(hidden_dims)-1) :\n", " self.layers.append(nn.Linear(hidden_dims[i], hidden_dims[i+1])) # hidden layers\n", " self.layers = nn.ModuleList(self.layers)\n", " \n", " for layer in self.layers : # Weight initialization\n", " nn.init.xavier_uniform_(layer.weight) # Also known as Glorot initialization\n", "\n", " self.act = nn.Tanh() # Nonlinear activation function\n", " \n", " def forward(self, x) :\n", " for layer in self.layers[:-1] :\n", " x = self.act(layer(x))\n", " x = self.layers[-1](x)\n", " return x" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2022-05-26T11:29:43.802976Z", "start_time": "2022-05-26T11:29:43.796295Z" } }, "outputs": [], "source": [ "model([1,64,64,1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Loss function, Optimizer" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2022-05-26T11:29:44.143410Z", "start_time": "2022-05-26T11:29:44.135128Z" } }, "outputs": [], "source": [ "# Prepare for training\n", "\n", "network = # Pass the network to GPU\n", " # Pass data to GPU\n", " # Pass data to GPU\n", "\n", "loss_f = # Mean Square Error loss function\n", "optimizer = # Stochstic Gradient Descent optimizer\n", "EPOCHS = # Number of Training Iterations" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Training" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2022-05-26T11:29:48.028579Z", "start_time": "2022-05-26T11:29:44.486693Z" }, "scrolled": true }, "outputs": [], "source": [ "# Train\n", "loss_list = []\n", "network.train()\n", "\n", "for i in range(1, EPOCHS+1) :\n", " optimizer.zero_grad()\n", " output = network(x)\n", " loss = loss_f(output, y)\n", " loss.backward()\n", " optimizer.step()\n", " \n", " loss_list.append(loss.item())\n", " if not i % 100 :\n", " print('EPOCH : %6d/%6d | Loss : %8.7f ' %(i, EPOCHS, loss.item()))\n", " clear_output(wait=True)\n", " \n", "print('Training Finished.')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Results" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2022-05-26T12:11:43.329449Z", "start_time": "2022-05-26T12:11:42.620544Z" } }, "outputs": [], "source": [ "figure = plt.figure(figsize=(15,5))\n", "ax1 = figure.add_subplot(1,2,1)\n", "network.eval()\n", "xx = np.linspace(0, np.pi*2, 100).reshape(-1,1)\n", "yy = np.sin(xx)\n", "ax1.plot(x.cpu(), y.cpu(), 'bo', label='Data points')\n", "ax1.plot(xx, yy, color='r', linestyle='-', lw=2.5, label='Target Function')\n", "ax1.plot(xx, network(torch.FloatTensor(xx).to(device)).cpu().detach(),\n", " color='b', linestyle='--', lw=2.5, label='Trained Neural Network')\n", "ax1.legend(fontsize=13)\n", "ax1.set_title('Regression', fontsize=15)\n", "ax1.set_xlabel('x', fontsize=13)\n", "ax1.set_ylabel('y', fontsize=13)\n", "\n", "ax2 = figure.add_subplot(1,2,2,)\n", "ax2.plot(loss_list, label='Training Loss')\n", "ax2.set_yscale('log')\n", "ax2.set_title('Training Loss in log scale', fontsize=15)\n", "ax2.legend(fontsize=13)\n", "ax2.set_xlabel('epoch', fontsize=13)\n", "ax2.set_ylabel('loss', fontsize=13)\n", "plt.savefig('regression', dpi=100)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.2" } }, "nbformat": 4, "nbformat_minor": 4 }