dqn/solution/Deep_Q_Network_Solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Q-Network (DQN)\n",
    "---\n",
    "In this notebook, you will implement a DQN agent with OpenAI Gym's LunarLander-v2 environment.\n",
    "\n",
    "### 1. Import the Necessary Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import random\n",
    "import torch\n",
    "import numpy as np\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Instantiate the Environment and Agent\n",
    "\n",
    "Initialize the environment in the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "State shape:  (8,)\n",
      "Number of actions:  4\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('LunarLander-v2')\n",
    "env.seed(0)\n",
    "print('State shape: ', env.observation_space.shape)\n",
    "print('Number of actions: ', env.action_space.n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please refer to the instructions in `Deep_Q_Network.ipynb` if you would like to write your own DQN agent.  Otherwise, run the code cell below to load the solution files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dqn_agent import Agent\n",
    "\n",
    "agent = Agent(state_size=8, action_size=4, seed=0)\n",
    "\n",
    "# watch an untrained agent\n",
    "state = env.reset()\n",
    "for j in range(200):\n",
    "    action = agent.act(state)\n",
    "    env.render()\n",
    "    state, reward, done, _ = env.step(action)\n",
    "    if done:\n",
    "        break \n",
    "        \n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Train the Agent with DQN\n",
    "\n",
    "Run the code cell below to train the agent from scratch.  You are welcome to amend the supplied values of the parameters in the function, to try to see if you can get better performance!\n",
    "\n",
    "Alternatively, you can skip to the next step below (**4. Watch a Smart Agent!**), to load the saved model weights from a pre-trained agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 100\tAverage Score: -225.00\n",
      "Episode 200\tAverage Score: -176.93\n",
      "Episode 300\tAverage Score: -87.32\n",
      "Episode 400\tAverage Score: -56.19\n",
      "Episode 500\tAverage Score: -66.67\n",
      "Episode 600\tAverage Score: -42.52\n",
      "Episode 700\tAverage Score: 36.50\n",
      "Episode 800\tAverage Score: 48.18\n",
      "Episode 900\tAverage Score: 155.64\n",
      "Episode 1000\tAverage Score: 191.26\n",
      "Episode 1100\tAverage Score: 185.16\n",
      "Episode 1200\tAverage Score: 180.23\n",
      "Episode 1300\tAverage Score: 184.97\n",
      "Episode 1400\tAverage Score: 188.67\n",
      "Episode 1500\tAverage Score: 181.42\n",
      "Episode 1600\tAverage Score: 199.10\n",
      "Episode 1604\tAverage Score: 200.17\n",
      "Environment solved in 1504 episodes!\tAverage Score: 200.17\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEKCAYAAAA8QgPpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJztnXeYFFXWh39nMkMaMkgaclQUBwQlg4Bi1hVdAyYwK+uqS1jDt+rKqqvurhEVI6IopgUkCworYcgZBhhgiEOOw6T7/VFV3dXVVd1V1VVd3dPnfZ55puvWrapT1V333HPOveeSEAIMwzAMY5UkrwVgGIZh4hNWIAzDMIwtWIEwDMMwtmAFwjAMw9iCFQjDMAxjC1YgDMMwjC1YgTAMwzC2YAXCMAzD2IIVCMMwDGOLFK8FcJPatWuL7Oxsr8VgGIaJK5YvX35ICFEnXL0KrUCys7ORm5vrtRgMwzBxBRHtNFOPXVgMwzCMLViBMAzDMLZgBcIwDMPYghUIwzAMYwtWIAzDMIwtWIEwDMMwtmAFwjAMw9iCFQjDMEwIFm49hJ2HT3stRkzCCoRhGCYEt3+0BL1fne+1GAGcKCpBebnwWgxWIEx8s7bgOErLyl2/zotTN+CXzQddvw7DhOPYmWJc8PwsvDlni9eisAJh4pdN+0/g6rcW4tVZm12/1ocLd+Duj5dZPs6s6+PtX/IwOXe35fNHwpHTxRj51Upkj5qGVbuPRfXajH0OnSoGAExYlO+tIGAFwsQxhSfPAQDW7znhsSQSawuOY8a6fb7taWv2ofer8/HLpvCWy6szN+Ppb9cElBWVlKHg6BnH5QSAJdsP49Jxc/HDqr0AgPlRsK4OnCjC8bMlrl/HDCO/WokHPl/utRg2kVxXp86V4vYPl/hKtxw4iWETlqKopCxqkrACYUyzYEsh9h8v8lqMIATc9QULYe78V7+1EA98scK3vWaP1KvftP+kres+PHEFevzjF9/1f167Dxv3hVeWR04XY8RnuTh+xt9Yl5ULlKl85kPHL0ZRid/1t27PCazYdRQAsK3wFD77Pd+WzKG45O9z0efVXxw/rx1+WLUXM9bvj+o1TxaV4Fxp5I37pKV+S3Vh3iHf52d/XIcFWwp932M0YAXCmGbYhKW46j8LvRYj6qgb3hIL8RZF7xBZu96JohIIITBXtlzKhaTEHpy4Alf86zfsO34W+48XIXvUNExZXhB0/Pu/bsOsDQcwcelOWQ6BFmOmo8WY6YbXnLPxAG54538AgGvfWoRnf1yvqziFEFi357i1G1Jx9ExsWCCR8trMzXhnfp6lY85/fhaufWtRyDr7jp9F9qhp+G1rIVqP/Rn/mLEpqM5HC3foHpucJP3QVu8+jhenbjDd8YkEViCMJQ6dOufYubYeOOnIj3xR3mFcNm6eAxLpU1Lml7H/PxeYPk4ZJSO/13hzzhZ8t0Jq8P+37ZBPMeUdPOU7Ju/gSVzw/Cx8k+tXDNsLT2HaWr9rrPvL83D5G5Icb/2ShzUFx3DwZBFmbzgAwK+4piwvwMmiEmzcZ80COnWuFADw87r9aDZ6Gs4Ul/r2fZNbgKv+sxDzNh0wfR4t/5671ZQcRSVlAdcOx+b9J3H0dHFA2d+nb8T7C7YFlKmt6JNFkkLbVngqoKMQjrd+ycMrMzbjz5NX48slu3TrLNl+GNNV3xsQ3hpdli9ZD3d8tBTFZeV4d/42lJcLfLJoB04UhVa+yUlSc/6PGZvw4cIduO9T95eyYAXCuMa8TQd8cQotv2w+iMvf+BXfr9xj+/wEf9d+z7GzQfvXFBxzxJwvLvVbHbuO+GMSvV/9Bd+ECHwr7VGSbIK8OWcrnpi8Gr9tLcQfP1iC9+SG7alvV/uOGfD6rwCA+Vv8MYnL3/gVj3y5MuDcJ4ukhnXHodO45q1F6PmPXzD8s1yUlJVj1S7Jdbat8DTGfr8OB074G8wl2w9j4hLjpR5mqdw6r8zYBCGAPUf9z3bbIUnZaRvC42dKcP/nuTh4UrrW8p1H0PG5mbqK5vXZ5kYPXTpuHto/OzOofN2e40Edj3s+WYZBb/6Ka94OtJDH/7odL/8c2IsfNmGp7/OIz5Zje+Ep9P/nAnR/eS6mrQls8MMxZUUBxny/1redPWoaHv9qJW58938YOn4xHpoouTTVijDv4Ck888O6IIX1+qzNeGxS4PcMAM3HTMfz/92AC56fFbLDlZoUaOruP+G+u5kVCANAGhqYPWoaFql8qpFQWlaOez7Jxa0fLNbdn3dAaog27HUvAH7NW4t8bhk9SuUe3iV/nxMy8Fis47YSQmDn4TN4ShP4VlMuv+yk8WHtOya92PmHpBFaeh4uq7Gmc7KSazX2ZyzNP+IrX7rjCDao4iZDxy/G2O/XGZ5nhCqwXCo3cIt3HEFpWTnenLMFpbI1VlTsf16lZeX49Pd8zFx/AJ/II4NW7JSU2D2f5JqK2+hxRLYmdhw6jRnr9uNscRkWbj2Eq/6zEH/9IfAe5snuvt1H/MrOqPNSqLKiVxccw175+zh48hwe/nIFhJB6/GsLAl11d30sKZ7DBlb4sTOSvD+u2ovlOwM7Lh/+5nc7DXh9AT5fvBNrCgJHvv17XniX2LlSfRfq2oLjAfcFABmpyWHPFymsQBgAwFrZr/3u/G1happD6ScpjWS4el7w2qwt+MeMTThw4hz2hWiwtXGPBz5fjnUmRn4pvUVNx9DX2CURoaxcBCkYAFixy5lhtftPFOHVmfaGORfIlsczP6zDT6v34s05W33+91PnyrBp/wmUlpWj5diffVZFrSrpAALjPgu2FAad26inX3jyHDo8OyOgce372nw88MVyXPGvX7FbHpU2UXYbrdp9DJOXBVqBR04X466Pl6LLS3N8ZdmjpuHHVXsghPApJgA4U1yGzQcCrakVu47i+f9uwNVvBVoz8zdL9zFpabDLSgiBC/82W/eeAOh2zM4Ul6GopAxPfrMaB01aC3purOvfWYSr31qINRqFl5bsfvNeoZe0ZZhQmHVvaV0NM9bvx5YD4eMKylFJGgWhjP75Onc3vo7y3A+7jP5ubcD2hEU7MGHRDlTNCGxCXpi6AZc0qxlQph0BBgAPf7kCzev0RLsG1XxlOw+fxqgpa3G6uCygx66Qf/gMXpy6wbf9/coC/Onr1UH1Or+g35A//tUqtK5XNaj8BdU5AeDd+dt9n7WxtVPnSvHarGAXnJ6VqmbJjiNBZWeKy3D/58uxYEuh6aG3L08PDqqvNOhspKW4r0A8s0CIqDER/UJEG4loPRE9LpfXJKLZRLRV/l9DLici+jcR5RHRGiLq7JXsTPyiflHNDo7SczuXhfBFL8s/gpZjpvsGHGgtkHjEyHWixGLU/Lwu0LrYceh0QBxJQYkLbNx3AtmjpqH3q/Px+/bDAIAUg4d2WuU6e2naRnPCqzDjop2z0R+30cbWOj4XHJMBjJ8PYDwMfNXuoz7rzKwlnrszWBEZUaEVCIBSAH8WQrQD0A3Aw0TUHsAoAHOFEK0AzJW3AeAKAK3kvxEA3o2+yBUXp0f8hTuf1aGtTpB38CTaPjMDP67aEyRDKHH05pn44xvB9d+bvw2l5QK58ogaPRdVReZUUWnAPX+7vACPfLkiqN4WOQ6WuzPYEvzOxOAKZUa2FV60oXTMcOnLxqMA31uwXbf87V9U7mKT71+HBtVNy1ShFYgQYp8QYoX8+SSAjQAaArgWwKdytU8BXCd/vhbAZ0JiMYAsImoQZbErPE60deXlwtJ8CTfYtD84RrFeDtjP2Wht1rWeMiwv9+/TzotQYgeKklm56xh6x8gEumhw8lwp9mp67nN1ZuOP/m4t3pq3FSt1FEi8YTRkGYDuXA4tZifDqgdIhCO9IisQNUSUDeAiAEsA1BNC7AMkJQOgrlytIQC1w7hALmMcxAlL5C9T1qCDgam/fu9xnP/cTN+IEbfmOp0+Z+xTVlwK6mHAfV6bj2d/1B+dpCdiuUpw7eRKJSirjGKasqIAOw+7k5IkFjlxtsRwspuW12ZtMWVtVHTMvgdHTpu3uhJCgRBRFQBTAIwUQoQa2qLXNw567EQ0gohyiSi3sDB49AfjRz023Ukvyzc6s6MVPlq4AyfPlWLB5uh/N+FcSZ/9rj8/Qs+HbWbS2bEKMuvaKlYtPAaY68Izi8YoLE8VCBGlQlIeE4UQ38nFBxTXlPxfebIFABqrDm8EYK/2nEKI8UKIHCFETp06ddwTPs6ZtmYf2j87E+v3Su6X1R5lY41EcakncDl17fcWbAtSGOEsEIaJlHCjuOxQoWMgJHUHPwKwUQjxumrXTwCGyZ+HAfhRVX6nPBqrG4DjiquLsY4yF0GZyKcMTTRq0N3KqxPJadWzwq0wVZ6DoHev437eFDTHQzcGoimLhcV9GEZNNBSIl/NALgNwB4C1RLRKLhsDYByAyUR0L4BdAP4g75sO4EoAeQDOALg7uuJWLITBLGmjBt1p/RHLI5PU1sU78/NQVhbeheVGD5Jh9Divegb2mshUkJ7i/kx0zxSIEGIhjEdP9tepLwA87KpQCUD2qGl4oHcL37adZtxI+cQ6ZqX9atkuDB3/O1Y8czlemaE/i1u7rgW7tJhoMXF4N/R9bX7YehXahcV4x3sLthkOGjR0Yak+vzB1I5qNNk4NboZopJoOBxmolElLd6OopNzn6jLDyK9Wha8UA9zfu7mr57+1a+PwlWKEpwa18VoEU1zRsX7AdpX04H6/dvY/AGSwAmHcwpenKUlbHro+IKWx0JaZxajR9oJwBtQbJrPGAsCsDeHTm8cCfxrQGvnjhvgapf5t64Y5whp92jh7Prfo26YO7rmsmddimGLskHYB23oKRJ0SRuGWrk1ck0mBFUiCojT92gbdyup+MWBEGKJWDsoqcGeLra0GVxHdUkqGVuXWBrSvBwBomFXJkfMbpSCJFb4a0Q2f3N0FE+7qgkpp7sQIHurTAuc3ND9j3CoZqf5mu2pGCoac3wCP9GupU4+z8TIuoW4bT6tm0RpaICbLwl4XgfETMwpr3/Gzltf1UN9Hm7/OwN5jZ/H0FOPU63okwsCqZCKsfnYgZj/RK6LzzBjZE+/c1tm3Kl7PVrWdEA8A8MqNF6CfQ5ZSTtMa6NOmbtj43Q0XBc9RHnJ++MQXd3ZviqcHt3Ws89GuQbWAYPjA9vUCZL+xcyO8fVtn1JazIEcbViAJis8CIWkWdtj6Ou9DJHEMK/3U3q/MD7muhxnsDPk1Wk8iHvjgzhxT9YiA6pmpyEyLbDxN2/rVcOX5DZAi+0RLdUauBR8TnBlXj/TUJLSqW8W0LHqNv0KyCQtp5TOX4/WhFwaV39wlfHznzu5NAehPNNVzM4Xj58d7ok7VdHx4Zw5WPzsQ41Xfa1pyEp65qr3lczoJK5AEYdzPm5A9appvW73YkbqhtKITIuljWTlWGSL7xeKdeHOOubiEtoOp12zEwyiyyibdLKnJgfdyueyaMmKjnCtMO5pMjxZ1KpuSAQCa1soEAPRtG34Sr1rJ6X0VijuspQXlAQD39TQeKKD9zt+5zZ/Uu2p6CqY+2gM1KqcBAKY91sO3L3/cEPRubX5isvIeNa/tf3Z1q6bjWU2D/8ndXQK226jSzfdo6bfiBrSvh+qZqb7tDX8bhPV/G6SrEH99qi+mPtojqNwNWIEkCB/8pp8R1HRKc50m344BosRc7Fgvf/1hHd6cY249bXOyxD7atUQAYFCHYOWgV++mixsZnlfJzaWXM+zaC88L2NambG9ZtwrqV8vQPW/jmplY8czlGB6iEQekBrlxzUzfdhPVZ4WXru+IpWP7o8N51U13ODo2rGYpu8FlqkZ67f8NQkdV7CKU4rqjW1NMvO+SoHLlZ6100F64rqNvX2oyoYXmnOpBB1eeXx+vD+3k2/783q6G189MS0GqQaqSJrUyA+7DTViBJCqaH7q/WP9V1XVh2bBBiuSAtpnev9E6EnaIB2tDFx2xz+gMBtALXr/2h04Brp/2KhfKr0/1xT2XNcNDfVsEHad1Z/3n1osCtmeN7IXHB7TybWfXCmz8a1ZOM/28G1SXFNHL158ftO/aCxuiblVpf6gOh9ILb1GnMqY+2hM1MtOC6tTITEXresEKIZSYoUYMvnBdxwDlo6BIqawXU6+aPzaRnEToFSI29M5tF6PDedXx0bAcTL6/e1z8ZlmBJAjB+Z2kbW0aaiuBYzsWiNFSplqOnSlG39fm4xrNsqJmefTLlWHrxMH7qduE6SkQI99+z1Z+t8t3D13q+9ykViaevbq9bi/2UdWInoHt6+GS5rVQu4q/UU5SXeuWLo0x/6m+Ie9h2dgBhvv++2gPTHmwO7J0Gn3196P9rb13u9/1NPn+bgH76lfPwMyRgYMCbu3aBLP+1Dv4GiHk1vt9/GVw25DxJcWSUixCtWWYkpxkSin0b1cPXXXmdcQirEASFOWF9HpehpESUhTbpv3hl47VQ7uSnN57Gwf6Q7fB0RuObKRAxlzZ1vfZ7LDO87Iq+Rpo5fJLxxgrgXCEajNrV0nHxU1rGnw/wYVVM1LQqEYlDO7oHxGVrJ3MBKCNJkBvJEOoBl3PLfhgnxaG8aVrOp3ne8bj77gY9/VohmaqGEhWpVTd4+IZViAJgradPigHzoPeEZctEMYaenpB77HrNaKA1Ov95O4ueOn6jrr7jdB+t0k25nfUrSq5b/QaYi1KHXWDq3fYY/1aYeFf+gWURdIRCGmBmDj+hWs7+AYOqGlepwr+elX7AAUVLzPfreBlMkXGQ7YVSsuJatsFt2Mg4SgtK8eKXcd8vnErFJeWY1HeIfTVmTMQS9ZGl+waWJZvbl6LWT94qAl8ZmeHf/NAd98aEkYTTa0w78k+KC4tN7UmvFJH7WpVK55Qv7RIXJEhYyAmzntH92xUq5SKx02ksqmsM4M83mELJEFRJg8GzUQ3nEioty6442Lh33O34ub3f8fyMMucKuuaq/nn7M24+5Nl+Pv04HWvdV0kHgVB7u3hH6F0cdMaIeualVBxYTWvUxnzn+xjS64u2TXRqXFW4PUjeERV0lOkgLqJuwjnYvS5XEO4uuz8HENZR07/PmKpE+MUrEASBK1iKDEx0Sv8OZ2fSLjlgGQZhVv7+bmf1geV7TwkDU3VW+3wqW+tzUJ3EzOT2RS0jdhPj1wW8pzVK6Uiu7b5eRtGhPtqrXz1ZKKV0WuszbbfRvU2/G0QRqpGi3mNGVdeJHw9ohu+eaC7q9fQwgok0QlyYemj78JyHuUd+3LJrpD19OQJ9X5uLzwdsH38bIlnPUIr4QTtPV3QKEu33lUXhE+zYQV/ypnIz2XmFOEmetpxl2ampWBQBylp5OAO+s/HyTY9nIRuG7yXNK+FLtnRHb1V8ZxyjCWCYugGXUvdXFgxGkQ309jc/3mublbTaKAOSIez4vTaHO0x+eOG4Pdth/HO/G1OiKe5vl+COU/0xn55ISMrjaGTPW+rlkq7BtWQP26I8fmi2I2Ih3kdVmELJMEJWpHQysERKBCr65GYwcqxG/fZGx7sBFZuMVI3jl309FrLulXQQ54IZ6Xz4IQCcauzUgHb9KjCCiTBCbZA9Ovp9ZTdGIUVLaSGw5vWw+zoIm3dUDh9Jz65nHBhOSicrqsrAiFZf0QGu7ASHO3LbRgD0SuLQH+48eIqDcmR0+ETBALe9T6TA1xYoeuaFVGxJJ26Jd+yxYbXM38ut59zZFZr5MIN7lgfN+c0wpMVcJ5HOFiBMLaJxP4wnBkcSRMoH3roVPg07ElEKCqxtsCUU1hrfE1aIA430gPb18eV59fHmCvb6e6PvgvLOKgfkQKxf6iP9JRkvHJTp/AVKyCsQJhAjILoDq8H4uQ57HDkdDF+23rIk2snW2jxTMdAbMpiRKW0ZLxz28WOnMvUKKwwN1pHntWulygxIhcW+7AighVIglNUEpjt1rA5j9IwXgcMkJgnYBRWmLpeBdGdxAkL5P7eLdCgeqWgVPOA6t7tLC8Qyw8uDmAFkuCM+X5twLalBaUiGYXlm1mceC+wlQZVr3cdq8OnjXDiK05NTsKNBuubJN4vKHbgUVgVnO9WFCDv4KmIz6O7oJQDNojWhWW2MdBzfcWLMopkImGsYUY+IkLPVrUx/g5nXGJ2ZGDcgS2QCs4Tk1dbqm8lmWIk+iORX/pIXTqxNHzarDX0+b3Bq/cx8Q9bIEwA0VoTPZFxOydS4kEB/2KNVhbXdI8n2AJhAjDOxmu+biTEixsqEgJuMd4CGhpi6uuK0Uf5/cOX4fhZc3OT4g22QOKYyct2m5rzYAXjZIoVaya6l6gnErqREj+aOKX/zsvKQHatTDx/TQfLx8aUEtOhSnoKGmZVMtw//8k++N+ofob7YxlWIDb54Nft6PrSHM+uX3D0DJ6esgb3f77cMxncaPwiaQu8mlNiFXZhBZOekoz5T/U1vfhVRSK7dmWcF0LBxDKsQGzy0vSNvmVhvWDFrmMAgIMni4L2/bqlEB/+tt3Wea1l43VgFFbEZ4g/kiN862JJT7IutE6PlrW9FsExWIHEITPX78djk1YCAMrLg/ffOWEpXpwWvCpfJOjPRLd/PsNUJgnQIEW6zkUoYki3RJ14ufcv7qs4I9JYgcQhardVKCvg6Oliy+eOVu/W6DoJoD8spTIxjzdPLhasoUT4zcQqPAorzgkVh7jvs1zL5zOcB6I3kdDDxsPhaSlRhWMgoXn95k5YuiP0ksZq4uV7r4iwAolzykK04jsOnTbcZxndXFjOv7oJN4zXBrHUYLrxdd3QuRFu6KyftiSkLM6LwoSBXVhxjtaFVVJWbrjP3PncqaslAfSEIUkW1gOJdeJdfiYyWIHEOVoX1vS1+3yf7bzb5VZGYdk4f9A5NCdJBL2izoXl1IJSTGxZZolC3CkQIhpMRJuJKI+IRnktj9doG/zSMv+2k71Dt9YDSUSsxEBi/QnHgiUZAyIkLHGlQIgoGcDbAK4A0B7ArUTU3lup3GX4Z7n4/Pd8w/3lGhPkz9/4kye63cBHcnZXRHP4nO/d3tnZE8pYbXRjoZFmGD3iSoEA6AogTwixXQhRDOArANd6LJOrzN5wAM/8uN5wf7kAikvLseXAyaB9dhpp4zXRozQKK4Yay8EdG7hyXqsr6G158YqAbbb8mFgh3hRIQwC7VdsFclnCcupcKR74YjkGvvEr9h0/G7Dv5LlSy+czTKaoW26/IUvkXnVADMRE/dRIp64zjEvE2y9Tr9kJeAeJaAQR5RJRbmFhYZTEih6lZcFTz+dtOggArmT8PH6mBA9PXKF7bley8Zrtnbtw7SEXuGNxaAmYic7WhGPws4w+8aZACgA0Vm03ArBXXUEIMV4IkSOEyKlTp05UhXOTORsOAAAelVOYuIX6JTx4oggTFu3AtLX78NHCHcF1nbieg5og0nOlWFkqMALMXOWCRtVdl4NhIiXeFMgyAK2IqBkRpQG4BcBPHssUFZRZ5T+v229YJzf/aMTXUZrgbYWn0PXvczFhkaQ4nMiF9eOqPZEJ5zLRmiFu5jL/uuUiqa7LslQkEmESaqwRVwpECFEK4BEAMwFsBDBZCGEcYU4wSnTcW1ZRlMK+Y1KW35NFxnEU/cC6sVZ5YWpwgkety8rLNkDv2m4ZJU8ObG2qXjynbKkIvP3Hzpjy4KVeixGzxF0qEyHEdADTvZZDQQjhSs+nvFzgsa+suaucXJ8jLSWwb2F2FNaUFcZWRqwvSqVngdSqko5Ch9P2pyQn+X4z8eq2j4XvrXvzWjhTbH2giBWiFReLV+LKAkkEPv89HzkvzsaRM8WYumZf2PraYyNFaRi0CkSvvdCbtb5uz/GAbXWbrK5/4IS/UZ6xbj/y5bxdXjohohECqZyWjCrp5vtt7JQxZtKIbvjxkR6+bQ6iRx9WIBHi9G/2mR/X49CpYlsNR/7hMxFfX7kfMwHl4tJgl1my5jjlfNmjpuHoGf9ILqVXLwR8w5AjJdLvwur8jHDM+3Nv3+fH+7cCAFzSvJalc8Rqk+j0s4oEjn14ByuQCHHrBffqpSg4ehZCCFON8dnisqAyu734Yjl+E0kgO1IFkqTzNlx/kf1pRs3rVEHb+lUB+BVymcbPqHYFpWutPhXtG1SzLYcb2HVh/T66H357uq/D0jBewQokQrRmc3m5wNyNB+LanP562e4g95Te3ZwpLsOxM8VYsMU/3yZJR4OEehbqBnXn4dP4One3YV01epMkZ6w3HqFmBr0Je6MGt43onArKc1Gea83KaQH/Af1nrDzNnx65zD8jPX5/WmhQvRIa18x09JyKu7VetQxHz8uEhxWIw3z6ez7u/TQXP63eG7ZuKDq/MNsZgWywctexkOuMKJwpKcPwz3IxbMJSnCiS3FN6FoS21x1wDpUVM0FnrokeyrWcRs8C0FOIVrg5R5q2lJWZCsCvQIbmNMZrf+iEuy5t5q8sP6ag+BOkwLteuUITuVG2s46GHWLJhdUwqxLeHHoh3r39YtvnaFCdlY8d4m4UVqyhbRr3HJXSiRw8EThyRwiBm977HcN7NsfgjvWjJJ09UlMoyGrQsyKKSsqw5cApAECZnAV4lo4VUBpCgZxWWRKnzgW7xPQo0Ym9OIFRA71s7AAM/ywXq3YfCyhvWisTO3XiThc0qo6RA6SYxz09muH6ixpi4/4TAPzKNCmJcNPFjXwTRBW+eaA7GmZVwjn5Hs0aG3WqpmPHy1earB05sTAKS811EbgaF4/uj8z0ZAelSRzYAokQo456sWZORkmZwPKdR/HIlysCysvLBZblm1u+c3vhKVsyWuWLxbtw47u/B5Tp3WZZufAplj3HJMW5rTB4FcRQFsgplQKZsqLAlHzfr3RnQmJasn4jUqdqOirrNDBGffCnB7VFv7b1fNs1Kqf51kHXPgutwdYluybOy6pkXuiAc5EHsbPYsUTsUr96BqplpHotRlzCCsQlXp25OWBb6bFp3+8PftuOP7z3O7JHTQt7ziOnix2TzyrbdRRDabnACXmi4VX/WYhNci9bDRHw4rTgCYQKSyysfa0Q6nyREMpFpOeysRLwz0iVFFBRSWDHoldrf7odvV592ORvnhNb0sQLnRpVx9CcxuErxjjswooQs6a8Yqloe4h5BwN/UMV0AAAfWUlEQVStCiJjq8bJiYJWWauZ3wEAZRora4eOkikpE5i0dJdrcjlJarK13rSR/tArz0yTFMjZkkA3nTpwf9slTU1dN54HaDAS6vkr8QxbIBFi9V3Wti3axiZUE2a03KxXaGMboXrw8cDNXRpjcAfz8Skjd5FeqWKB6A19Vnju6nhcGy3+XViMfeL7jY8j/BaIv2zV7mNByRFDuUViTYFoXUkxJp4lbs5phGoZqXjvDv2RPHpfi9E3padYFAskFPE5IS6Ov3QmYliBhMFM3GGrajVAozZAafzVvvTr3l4UkKxQyqtlfJ1vc80Fmb1iYd4hr0WwTbh4xis3XRA0y97oGL3impXTMHJAK3x6TxfbMipU5YAvEyOwAgnBuj3H0fmF2Rj4xgJMXaM/r+O/q/fi8jd+9Q1fNY5fSDu0PvDAOqF7od+5NPrIKT75X77XItgmXO+/QfVKePG6jppj9OvqKRYiwsgBrdGyblXbMiqMv/NijL2yXcTncYZ4tJoYp2AFEoJN+yXLYsuBU3jkS/3MuOv3SiOPth4MHmK7cpd/fQ61XjGaZFhWLvh19Agz3iPtnEIjC8TtpIwNqlfC8F7N3b0Iw5iAFUgIzLQDPteUTuXr3/kfNstKSKgGLE03yLJbVh7ahcW4h5nHbnbtEidiGVUzpAGSlzSzlnyRYaIJD+MNgZl2QFEgSm9Ue0zhyXNoU7+qqeG+ZUJEbVU8JhD1c29Tryo2q+JaPkxaIE58hbWrpGPOE719KUoYJhZhBRICMw3BDnkdi7JygbHfr8Xxs4F5mkrKJdNDPeLV6LyJ5MIa1KEeZq4/EL5ilFB/J9Mf76k710KrMKzEQOzQsm4VR87jBnWrSrmjGte0N2ueqRiwCysEZhLGLco7DACYteEAJi7ZFbQIVGmZwJHTxTh4sshXZtTASC6siqtCMlKln9v9vZrj/TtyHDnnbZc0ceQ86u8kOYmQopOZN3gOjzcxkFhgQLu6+PjuLri/VwuvRWE8hBWIBWas24+5G631mkvLytH5hdkY/OZv/sJQFkgFbnwe6tMSQOgJh/+65UJL57SbN0rhwsZZputq1wsx+qqi5YZcPLq/Z2trEBH6tqkbNLSZSSxYgRhwtrgsINU4IK2cd++nuZi9IViJHDWYL1Kik39kmkEQ/fS50oB5IbHA7D/1CirLHzfE1rlK5NQneutuKFzRsQEWjepn+px222rtWtdmzqO1SL1uO+tXz3B8bQ2GsQIrEAO6j5uLMd+v1d03/LPcoLJdR/SXkz1nMO9DL0PtC1M3WJDQGcIpg5Z1q+CRvi19218OvwQA8NofOhkeM95gNve9PZphyPkNMKx7tuGxKUmEWqpFlsIRqrd/QaPqYY8XmkEQoQhKO2PowuJeOZMYsAIx4NgZZxYteurbNbrlekHafceLdGpGj1+e7IM5T/TyLX4ESI3kk4Pa+LYvbVEbAHDthefpnuPze7uiSrp/bMYd3ZpiyoPdsfb5gcjKTMPbt3VG9UzjmdRJSWTJqkhJInzzQHdMfTQ4OZ3avfJYv5a46WL/YkvKHkWPm7EmtArD6BjWH0yiYHoUFhH1ANBKCPExEdUBUEUIYW4JOSYIvUG96ane6PP3br8Ydaqmo1ntygCAVc8ODEov365BNZxUrQRo5IYikK9RvqxlLbygmb1tBjM9+JXPXI5/z9uK27s19SUq1JKR4i8XkKymb5cHpoPRDsMOLVfgttEgC1YgTKJgSoEQ0XMAcgC0AfAxgFQAXwC4zD3RKjbndFbVUzd4VmlWu7JvSLFVzKyQ+PPjPQ33PTWojW/9kyTyZ+k1u+xpVmZqgMWXbKIFrlE5Dc9d3SFknRQT6dl9nkQbMRCjY9iFxSQKZru81wO4BsBpABBC7AUQeVKfBOa3LYVBZZEsE9q/bd1IxAni3ds6458h4hyAtFIfADzctyUubloDgOTm8bXJYdpRJdbxzf3dA8qTkgj544Zg+98jW6JV7XIyylFWRV5p0MyKdNr7MXJheR1cZ5hoYdaFVSyEEEQkAICIKrsoU0Lw4MQVQWWLt1tfnc8trji/Qdg6s//Uy2c5KG1mEpl3Cylteg2DoHlShC2xmaNv7doEA9vXx7BLsy2fz9iFxRqESQzMWiCTieh9AFlENBzAHAAfuCcWEw9kZaYhu3ZgX0LdeKaEUQDCl+I+PHOe6G1ZvlDtuCJnchJheK/mphbDCh6FZVDPrIAME+eYskCEEK8R0eUATkCKgzwrhJjtqmSMJR7t1wofLvR+TAMR0KNlbdzZvSkeVg3/DX0MYeqjPQyHQmt55cYLQu6vWTnNpfXjA1WDkVuMYyBMohBWgRBRMoCZQogBAFhpxCihhsZGm9TkJPzt2vCjr2pUTsPRMyVIIqBjw+ro2NB43obSJjerXRk3d2kc8rw/P94TKUmEJ79ZbUnucJjVC6w/mEQhrN0uhCgDcIaIws/KYqJKK02yvYHt63kkiT0+v/cSvHhdR2Rlhp84qLTJevNntGSmJaNWlfSAskgGKGhlCAdbIEyiYDaIXgRgLRHNhjwSCwCEEI+5IhUTxKP9WuI/8/ICyl6/+UJc/dZCjyTSx8q66A2zKuH2bk1D1hnesxkuaVbLt4xrTnZN0+cPFcz+y+A2OHOuFJdbULpmg+NGgwIYpqJhVoFMk/8Yj4j1Pq1bne6xQ9r7Ps8Y2dM32TG0LJIw6hn1WsXWqEYmPror8vXJ1WRlpqLjedUDZuIzTEXGbBD9UyJKA9BaLtoshHAm1wcTlk6Ns9ixDqBt/WqW6v/fNR2wYHMhDjsUUA/3DQhhbvIiw1QUTA3jJaI+ALYCeBvAOwC2EFFwmlbGFc5vWC3mLZBYQnlWVTNScU+PZgD0U8dYPm+YL0GIxFkQjGEA8y6sfwIYKITYDABE1BrAJAD6aVcrANm1MpF/2Nyw0mgQbk4FwEaK24RVINERg2FiBrMTCVMV5QEAQogtkPJhVViqmxgZFE2UnjQTHr2G3kpw3zasQZgEw6wCySWij4ioj/z3AYDldi9KRK8S0SYiWkNE3xNRlmrfaCLKI6LNRDRIVT5YLssjolF2r22aqLQ45hACqJyeghs6N7R1vHbxJDcxM8w2XgmXHFKA05gwiYVZBfIggPUAHgPwOIANAB6I4LqzAXQUQlwAYAuA0QBARO0B3AKgA4DBAN4homR5MuPbAK4A0B7ArXJd19BZ78lzzGa31TKoQ30M7xlowfRrWxejrmjrhFgA/LLF4GNzjHBzSU6ds7eaZG3NnBWGiRfMxkBSAPxLCPE64JudbvtXL4SYpdpcDOAm+fO1AL4SQpwDsIOI8gB0lfflCSG2y9f/Sq7r2hJ+Tkw8cxq7nVu9wyY4PIQ1llArWicNgnMlwSn4g69tnZkje6Lw1DkbRzKMt5i1QOYCqKTargQpoaIT3APgZ/lzQwC7VfsK5DKjcteIJU+M0gjabQuj4lWJcc+NEx2ColJpeeIMeeEvvWWJ7VCrSrrlIcoMEwuYtUAyhBCnlA0hxCkiygx1ABHNAaC3UtFYIcSPcp2xAEoBTFQO06kvoK/odN9eIhoBYAQANGnSJJSIIYlJF1aYRtowvTgophSi26ifk123nx6KBVK7SjoKjp7VdVkpSoZhEgGzCuQ0EXUWQqwAACLKAXA21AFy8kVDiGgYgKsA9Bf+yGsBAHWmvEYA9sqfjcq11x0PYDwA5OTk2G42YykYHKkoRO7HJp69qj3GfL8WnRplha/sBfIDeLx/q4AZ6lYoKpGUQ52qxgpkUd5h2yIyTLxhVoGMBPANEe2F9CqeB2Co3YsS0WAAfwHQWwihnmzxE4Avieh1+RqtACyFZJm0IqJmAPZACrT/0e714xW7vek+bepg+c6jDksTSMeG1fHTIz1cvYYdtFbbny5vrV/RBL1a1wEADOuejZW7VtkOmjNMRSGkAiGiLgB2CyGWEVFbAPcDuAHADACRLD7xFqQg/Gx52ONiIcQDQoj1RDQZUnC8FMDDcjZgENEjAGYCSAYwQQixPoLrh6U8hiyQSKhZOQ2ZaSkJ5cLSw4nbb16nCvLHDUHhSSngfbKIs/kwiU04C+R9AIorqjuAMQAeBXAhJDfRTQbHhUQIYbjSkBDiJQAv6ZRPBzDdzvXsUFEa3BiPbbtCYAzEeapmSK9NSVkF+ZEwjE3CKZBkIYSyUPdQAOOFEFMATCGiVe6K5i0VxQJRGtNYHJYcr2SkJnstAsPEBOGG8SYTkaJk+gOYp9pXoXNWV5zmNhFtEPd5/ur2+G8MxnwYJpqEUwKTACwgokOQRl39BgBE1BLAcZdl85YKokF8FkgFuR8z6A02cHpU3V2XcW4yhgmpQIQQLxHRXAANAMxSDbdNghQLqbDEogtLbx5IrcppKCkzniGd6PYHp6ZiGPcI64YSQizWKdvijjixQ+ypD32Wjh3g613rNZaJ2IB6lo2XYRIMs6lMEo54aXCSkwgpycZfo5MzseMRt+//wT4t8M8/dHL1GgwTq7ACMSAWXVh28MdAKsb9mCGaKvMvg9vixosbRfGKDBM7sAIxIJrt7YyRPUPuTzaxGqERiW1/+Ekc9ckw0YMViAP0aFk7ouPb1Ksacn9ECoQq/jodWtSLOkU7BtS2fujvkmEqEqxADLDiwoq0kQq3il1qiBgHY45oWZQT77skOhdimBiAWyYDrDQ4bsdLUkxaIDwKy3tq8eqCTALBCsQAK0rBiZxI8/7c23BfqFFW4dCbSBiJSyweqNh3xzCxQ4VORxIJVlabc2JluqQQpkJqREH0wGOfGtQGwy7Ntn0+hmEYBbZADCi1oBRKQ8wEN0soV5MjFogcRq+akYIq6RW736A7kTChhhEwTHRgBWKAFavCCRdWSAskWdpnJ56hPSTR3DvhBigwDGMfViAGlJabtyqs1LVDp8b2l4n1DeNNoA64ntJIpPtnmGjBCsQAKxZIqWyB/OuWC21fL8kgznH3Zdnokl3T9nmD2tIE65En1t0yTHRhBWKAlRhIsRwDSU8xv9DQU4PaBGwbxcmzKqWZPqceymm5A84wjNOwAtGhvFxYcnkoFkh6qvnHWS0jMJBtlPTP6TkmidYjTzCDi2GiCisQHaxYHwB863GkRzBaysgCCZQkdGuop4QSMQbCMEx0YAWig9V5HYoCSUuJ4HEa6QZVyz+gXV3bp21euzIA4LysDMvnYBiG0YMViA5WR1UpFkskOauMhvGqVVn/dvWw8pnLLZ1XOe29PZph0vBu6Ne2nk0J45tESmfPMNGCFYgOVi0QRYG4kXZd2+6lJFu7huLWSkoidG9Ry4Zk8U235tI9D+xQ32NJGKbiUbGnJNvEagxEIdRkwCA0dY0tEGGqnsnLJBztGlRD/rghXovBMBUStkB0yKqUivlP9gm7ToeWSBpro2O1usyqAmEYhnELViA6pCQnIbt2ZVSrZM1Ai6RxN0q50alRdU29UCcJLqpTldOLMwzjDuzCCoHR3AzD+hEYB9rwyaUtauHVP3RCw6xKtq9x16XZeKx/K/tCMQzDhIAtECeQ3UxOL7OhVR7SNcxf5NauTVCzcmQz2RmGYYxgBeIo9jWINmxvpCfUCiQjzMx3DpcwDOMmrEAcxNIgLM22drhu3zb6kwbVVk4ri0F+hmEYJ2EF4gDKUNtIguiV0wITMd7bo5luPSvrW7ABwjCMm7ACCYXFFjiSBlu96uD/RvXjhZAYhol5eBSWg1ht898Y2gm1qwQPsz1PJ3geDXninSY1M7HryBmvxWCYhIEViAMI3ygsay329Rc1ckGaxOW7hy7FjkOnvRaDYRIGViAeEal1cEXH4NxOOsncI7tInFG7SrquRccwjDuwAnEQK0qhawTL1K55fiAqpZpf/ZBhGMYNPA2iE9GTRCSIqLa8TUT0byLKI6I1RNRZVXcYEW2V/4ZFRT6L9c26sPLHDYloCG61jFRTqeMTLQbCMEx08UyBEFFjAJcD2KUqvgJAK/lvBIB35bo1ATwH4BIAXQE8R0Q1oiqwCfQa7CVj+kdfEIZhmCjgpQXyBoCnETgJ+1oAnwmJxQCyiKgBgEEAZgshjgghjgKYDWBw1CU2QLkBPQukrofJDNkAYRjGTTyJgRDRNQD2CCFWa+Y7NASwW7VdIJcZlUeFt/54ER75cmXYenoNtpX5HK/f3An5PIooLpnzRC9Otc8kHK4pECKaA0BvGbixAMYAGKh3mE6ZCFGud90RkNxfaNKkiSlZw2E6IWGE7ccNnSMb1qtVVukcaI8aLetyWhkm8XBNgQghBuiVE9H5AJoBUKyPRgBWEFFXSJZFY1X1RgD2yuV9NOXzDa47HsB4AMjJyYnKQtjKetux0gP927UdUKtyum42X4ZhGKeIegxECLFWCFFXCJEthMiGpBw6CyH2A/gJwJ3yaKxuAI4LIfYBmAlgIBHVkIPnA+WyqGB2XZDYUB9A9UqpGHJBA6/FYBimghNrubCmA9gOIA/ABwAeAgAhxBEALwBYJv/9TS6LCgICM0b2xK1dQ7vEvM5fpSRkTDMxxJdhGCZSPJ9IKFshymcB4GGDehMATIiSWAACh+W2rV8N5zesjkk69fyjsKIhlTFjhrRDg+qVMLCDXuiJYRjGWTxXIBUJq0vgOk21jFQ8PoCXsGUYJjqwryMEd12aDQBoLc8aD+uhUu1//46Lw7q8GIZh4hm2QEIwuGMD5I8bYrq+2oU1qEN9DGJXEsMwFRi2QBzE6yA6wzBMNGELxALXXdgQP67ag8XbAweA+dcDAe7v1RxNa1XWPf7tP3ZGjcxUt8VkGIaJCqxALFApLRlvDL0Q3V+ep7ufQBh9ZTvD43luBsMwFQl2YdkkS8eSYA8WwzCJBCsQm+gnToy6GAzDMJ7BCsRBvJ4HwjAME01YgdhEb8QVWyAMwyQSrEAcJFay8TIMw0QDViA20Y2BRF0KhmEY72AFYhERYoURNkAYhkkkeB6ITdTKYtLwbpiyooBnojMMk1CwAnGA7i1qoXuLWl6LwTAME1XYhWUbtjYYhklsWIEwDMMwtmAFYhMOdzAMk+iwArEJ6w+GYRIdViAMwzCMLViBWERxXaUm86NjGCax4VbQIvWrZeDBPi3w6T1dvRaFYRjGU3geiEWICH8Z3BYi1JR0hmGYBIAtEIZhGMYWrEBswmlLGIZJdFiBMAzDMLZgBcIwDMPYghUIwzAMYwtWIAzDMIwtWIEwDMMwtmAFwjAMw9iCFQjDMAxjC1YgDMMwjC1YgTAMwzC2YAXCMAzD2IIVCMMwDGMLzxQIET1KRJuJaD0RvaIqH01EefK+QarywXJZHhGN8kZqhmEYRsGTdO5E1BfAtQAuEEKcI6K6cnl7ALcA6ADgPABziKi1fNjbAC4HUABgGRH9JITYEH3pGYZhGMC79UAeBDBOCHEOAIQQB+XyawF8JZfvIKI8AMrKTXlCiO0AQERfyXVZgTAMw3iEVy6s1gB6EtESIlpARF3k8oYAdqvqFchlRuUMwzCMR7hmgRDRHAD1dXaNla9bA0A3AF0ATCai5gD0FtkQ0Fd0uksCEtEIACMAoEmTJtYFZxiGYUzhmgIRQgww2kdEDwL4Tkjrwi4lonIAtSFZFo1VVRsB2Ct/NirXXnc8gPEAkJOT4+q6swPa1cONndkQYhgmMfEqBvIDgH4A5stB8jQAhwD8BOBLInodUhC9FYClkCyTVkTUDMAeSIH2P3ohuJoPh+V4LQLDMIxneKVAJgCYQETrABQDGCZbI+uJaDKk4HgpgIeFEGUAQESPAJgJIBnABCHEem9Et87LN5yPNvWrei0GwzCMo5DUbldMcnJyRG5urtdiMAzDxBVEtFwIEdbFwjPRGYZhGFuwAmEYhmFswQqEYRiGsQUrEIZhGMYWrEAYhmEYW7ACYRiGYWzBCoRhGIaxBSsQhmEYxhYVeiIhERUC2BnBKWpDSrESa7Bc1mC5rMFymScWZQIil6upEKJOuEoVWoFEChHlmpmNGW1YLmuwXNZgucwTizIB0ZOLXVgMwzCMLViBMAzDMLZgBRKa8V4LYADLZQ2Wyxosl3liUSYgSnJxDIRhGIaxBVsgDMMwjC1YgehARIOJaDMR5RHRqChfuzER/UJEG4loPRE9LpfXJKLZRLRV/l9DLici+rcs6xoi6uyyfMlEtJKIpsrbzYhoiSzX10SUJpeny9t58v5sF2XKIqJviWiT/Ny6x8LzIqI/yd/hOiKaREQZXjwvIppARAflBdyUMsvPh4iGyfW3EtEwl+R6Vf4e1xDR90SUpdo3WpZrMxENUpU7+r7qyaXa9yQRCSKqLW97+rzk8kfl+19PRK+oyt1/XkII/lP9QVrxcBuA5pCW2l0NoH0Ur98AQGf5c1UAWwC0B/AKgFFy+SgA/5A/XwngZ0jL/nYDsMRl+Z4A8CWAqfL2ZAC3yJ/fA/Cg/PkhAO/Jn28B8LWLMn0K4D75cxqALK+fF4CGAHYAqKR6Tnd58bwA9ALQGcA6VZml5wOgJoDt8v8a8ucaLsg1EECK/PkfKrnay+9iOoBm8jua7Mb7qieXXN4Y0qqoOwHUjpHn1RfAHADp8nbdaD4vV17oeP4D0B3ATNX2aACjPZTnRwCXA9gMoIFc1gDAZvnz+wBuVdX31XNBlkYA5kJaz36q/NIcUr3wvmcnv2jd5c8pcj1yQaZqkBpq0pR7+rwgKZDdcgOSIj+vQV49LwDZmobH0vMBcCuA91XlAfWckkuz73oAE+XPAe+h8rzcel/15ALwLYBOAPLhVyCePi9IHZIBOvWi8rzYhRWM8uIrFMhlUUd2Y1wEYAmAekKIfQAg/68rV4umvG8CeBpAubxdC8AxIUSpzrV9csn7j8v1naY5gEIAH8uutQ+JqDI8fl5CiD0AXgOwC8A+SPe/HN4/LwWrz8eL9+IeSL17z+UiomsA7BFCrNbs8vp5tQbQU3Z7LiCiLtGUixVIMKRTFvWhakRUBcAUACOFECdCVdUpc1xeIroKwEEhxHKT147Wc0yBZNa/K4S4CMBpSC4ZI6L1vGoAuBaS++A8AJUBXBHi2jHxu4OxHFGVj4jGAigFMNFruYgoE8BYAM/q7fZKLpkUSC6ybgCeAjCZiChacrECCaYAkq9ToRGAvdEUgIhSISmPiUKI7+TiA0TUQN7fAMBBuTxa8l4G4BoiygfwFSQ31psAsogoRefaPrnk/dUBHHFBrgIABUKIJfL2t5AUitfPawCAHUKIQiFECYDvAFwK75+XgtXnE7X3Qg44XwXgNiH7WTyWqwWkjsBq+fffCMAKIqrvsVyQr/OdkFgKyTtQO1pysQIJZhmAVvJomTRIAc2fonVxuffwEYCNQojXVbt+AqCM5BgGKTailN8pjwbpBuC44ppwEiHEaCFEIyFENqRnMk8IcRuAXwDcZCCXIu9Ncn3He2BCiP0AdhNRG7moP4AN8Ph5QXJddSOiTPk7VeTy9HmpsPp8ZgIYSEQ1ZOtqoFzmKEQ0GMBfAFwjhDijkfcWkkarNQPQCsBSROF9FUKsFULUFUJky7//AkgDXfbD4+cF4AdInTkQUWtIgfFDiNbzijSoUxH/II2s2AJptMLYKF+7BySTcg2AVfLflZD84XMBbJX/15TrE4C3ZVnXAsiJgox94B+F1Vz+YeYB+Ab+0SAZ8naevL+5i/JcCCBXfmY/QDLpPX9eAP4PwCYA6wB8DmlETNSfF4BJkOIwJZAav3vtPB9IMYk8+e9ul+TKg+SjV37776nqj5Xl2gzgClW5o++rnlya/fnwB9G9fl5pAL6Qf2MrAPSL5vPimegMwzCMLdiFxTAMw9iCFQjDMAxjC1YgDMMwjC1YgTAMwzC2YAXCMAzD2IIVCMMYQERlRLRK9RcycykRPUBEdzpw3Xwl26vF4wYR0fPy3IPpkcrBMOFICV+FYRKWs0KIC81WFkK856YwJugJaaJiLwCLPJaFSQBYgTCMReR0Fl9DSqUNAH8UQuQR0fMATgkhXiOixwA8ACmf0wYhxC1EVBPABEiTCc8AGCGEWENEtSBNEqsDaRIhqa51O4DHIE0YWwLgISFEmUaeoZCyqjaHlH+rHoATRHSJEOIaN54BwwDswmKYUFTSuLCGqvadEEJ0BfAWpJxgWkYBuEgIcQEkRQJIM9NXymVjAHwmlz8HYKGQkkH+BKAJABBROwBDAVwmW0JlAG7TXkgI8TX860ScD2lW8kWsPBi3YQuEYYwJ5cKapPr/hs7+NQAmEtEPkNKrAFKamhsBQAgxj4hqEVF1SC6nG+TyaUR0VK7fH8DFAJZJ6bRQCf6kh1paQUpNAQCZQoiTJu6PYSKCFQjD2EMYfFYYAkkxXAPgGSLqgNCptPXOQQA+FUKMDiUIEeVCysCaQkQbADQgolUAHhVC/Bb6NhjGPuzCYhh7DFX9/129g4iSADQWQvwCaQGuLABVAPwK2QVFRH0AHBLSWi/q8isgJYMEpCSHNxFRXXlfTSJqqhVECJEDYBqk+McrkBLkXcjKg3EbtkAYxphKck9eYYYQQhnKm05ESyB1wm7VHJcM4AvZPUUA3hBCHJOD7B8T0RpIQXQlnfr/AZhERCsALICUCh5CiA1E9FcAs2SlVALgYUhrcmvpDCnY/hCA13X2M4zjcDZehrGIPAorRwhxyGtZGMZL2IXFMAzD2IItEIZhGMYWbIEwDMMwtmAFwjAMw9iCFQjDMAxjC1YgDMMwjC1YgTAMwzC2YAXCMAzD2OL/AcJhDOvEW2mgAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x101b1dc18>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def dqn(n_episodes=2000, max_t=1000, eps_start=1.0, eps_end=0.01, eps_decay=0.995):\n",
    "    \"\"\"Deep Q-Learning.\n",
    "    \n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode\n",
    "        eps_start (float): starting value of epsilon, for epsilon-greedy action selection\n",
    "        eps_end (float): minimum value of epsilon\n",
    "        eps_decay (float): multiplicative factor (per episode) for decreasing epsilon\n",
    "    \"\"\"\n",
    "    scores = []                        # list containing scores from each episode\n",
    "    scores_window = deque(maxlen=100)  # last 100 scores\n",
    "    eps = eps_start                    # initialize epsilon\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        state = env.reset()\n",
    "        score = 0\n",
    "        for t in range(max_t):\n",
    "            action = agent.act(state, eps)\n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "            agent.step(state, action, reward, next_state, done)\n",
    "            state = next_state\n",
    "            score += reward\n",
    "            if done:\n",
    "                break \n",
    "        scores_window.append(score)       # save most recent score\n",
    "        scores.append(score)              # save most recent score\n",
    "        eps = max(eps_end, eps_decay*eps) # decrease epsilon\n",
    "        print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)), end=\"\")\n",
    "        if i_episode % 100 == 0:\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))\n",
    "        if np.mean(scores_window)>=200.0:\n",
    "            print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode-100, np.mean(scores_window)))\n",
    "            torch.save(agent.qnetwork_local.state_dict(), 'checkpoint.pth')\n",
    "            break\n",
    "    return scores\n",
    "\n",
    "scores = dqn()\n",
    "\n",
    "# plot the scores\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(len(scores)), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Watch a Smart Agent!\n",
    "\n",
    "In the next code cell, you will load the trained weights from file to watch a smart agent!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the weights from file\n",
    "agent.qnetwork_local.load_state_dict(torch.load('checkpoint.pth'))\n",
    "\n",
    "for i in range(3):\n",
    "    state = env.reset()\n",
    "    for j in range(200):\n",
    "        action = agent.act(state)\n",
    "        env.render()\n",
    "        state, reward, done, _ = env.step(action)\n",
    "        if done:\n",
    "            break \n",
    "            \n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Explore\n",
    "\n",
    "In this exercise, you have implemented a DQN agent and demonstrated how to use it to solve an OpenAI Gym environment.  To continue your learning, you are encouraged to complete any (or all!) of the following tasks:\n",
    "- Amend the various hyperparameters and network architecture to see if you can get your agent to solve the environment faster.  Once you build intuition for the hyperparameters that work well with this environment, try solving a different OpenAI Gym task with discrete actions!\n",
    "- You may like to implement some improvements such as prioritized experience replay, Double DQN, or Dueling DQN! \n",
    "- Write a blog post explaining the intuition behind the DQN algorithm and demonstrating how to use it to solve an RL environment of your choosing.  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}