dqn_sb3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/dqn_sb3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cgDiMlHXXN7V"
   },
   "source": [
    "# DQN and Double DQN with Stable-Baselines3\n",
    "\n",
    "Stable-Baselines3: https://github.com/DLR-RM/stable-baselines3\n",
    "\n",
    "Documentation: https://stable-baselines.readthedocs.io/en/master/\n",
    "\n",
    "RL Baselines3 zoo: https://github.com/DLR-RM/rl-baselines3-zoo\n",
    "\n",
    "Double Q-Learning: https://paperswithcode.com/method/double-q-learning\n",
    "\n",
    "\n",
    "[RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo) is a training framework for Reinforcement Learning (RL), using Stable Baselines3.\n",
    "\n",
    "It provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.\n",
    "\n",
    "## Introduction\n",
    "\n",
    "In this notebook, we will study DQN using Stable-Baselines3 and then see how to reduce value overestimation with double DQN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "StmMaKjrX6MC"
   },
   "source": [
    "## Installation\n",
    "\n",
    "We will install master version of SB3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for autoformatting\n",
    "# %load_ext jupyter_black"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "r7yLmacAXJ0F"
   },
   "outputs": [],
   "source": [
    "!apt-get install ffmpeg freeglut3-dev xvfb  # For visualization\n",
    "\n",
    "!pip install \"stable-baselines3[extra]>=2.0.0a4\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LwC8l-17YseR"
   },
   "source": [
    "Import DQN and evaluation helper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VYbeqK0tYenp"
   },
   "outputs": [],
   "source": [
    "import gymnasium as gym\n",
    "import numpy as np\n",
    "import torch as th\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from stable_baselines3 import DQN\n",
    "from stable_baselines3.common.evaluation import evaluate_policy"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "7xQwxJDgptxH"
   },
   "source": [
    "## The Mountain Car Problem\n",
    "\n",
    "In this environment, the agent must drive an underpowered car up a steep mountain road. For any given state the agent may choose to accelerate to the left, right or cease any acceleration.\n",
    "\n",
    "Source: https://github.com/Farama-Foundation/Gymnasium/blob/main/gymnasium/envs/classic_control/mountain_car.py\n",
    "\n",
    "![mountaincar.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAlgAAAGKCAYAAADQeD9lAAAAiHpUWHRSYXcgcHJvZmlsZSB0eXBlIGV4aWYAAHjaVY7dCYBADIPfbwpH6LW9/owjouAGjm9L1cPvIQ2hhLT9Oo+2JB2w8VATF4GAnR3XMAYFAXSEnje0eC71cDjjRlhG3BR4PvKTvwwSk0NZVYZssmG0405IGJpF2Qo5w2fJMKga+ue2fgNqxQ1ExSxUK+4OSAAACgZpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADw/eHBhY2tldCBiZWdpbj0i77u/IiBpZD0iVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkIj8+Cjx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IlhNUCBDb3JlIDQuNC4wLUV4aXYyIj4KIDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+CiAgPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIKICAgIHhtbG5zOmV4aWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20vZXhpZi8xLjAvIgogICAgeG1sbnM6dGlmZj0iaHR0cDovL25zLmFkb2JlLmNvbS90aWZmLzEuMC8iCiAgIGV4aWY6UGl4ZWxYRGltZW5zaW9uPSI2MDAiCiAgIGV4aWY6UGl4ZWxZRGltZW5zaW9uPSIzOTQiCiAgIHRpZmY6SW1hZ2VXaWR0aD0iNjAwIgogICB0aWZmOkltYWdlSGVpZ2h0PSIzOTQiCiAgIHRpZmY6T3JpZW50YXRpb249IjEiLz4KIDwvcmRmOlJERj4KPC94OnhtcG1ldGE+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAKPD94cGFja2V0IGVuZD0idyI/Pl0v6iEAAAAEc0JJVAgICAh8CGSIAAALfklEQVR42u3dXZLaOBiGUdzVO8oCgQVmTZqrnk5R/kFGsvRJ59wmPaHBjZ957ZAlpZRuAAAU8+UpAAAQWAAAAgsAQGABACCwAAAEFgCAwAIAQGABAAgsAACBBQCAwAIAEFgAAAILAEBgAQA0tizL7e/fZYjv5dvLCQD0ZC2y/vxJAgsAYOboElgAgOgSWAAAfUeXwAIApomuq2JLYAEAQ2p5uVBgAQBiSmABAGKqbwILABBThS0ppeTlAwCaR8my3EbJEv9UDgCAwAIAEFgAAAILAACBBQAgsAAABBYAAAILAEBgAQAILAAABBYAgMACABBYAAACCwAAgQUAILAAAAQWAAACCwBAYAEACCwAAAQWAIDAAgAQWAAACCwAAIEFACCwAAAEFgAAAgsAQGABAAgsAAAEFgCAwAIAEFgAAAgsAIDrfHsKAIDIlmWp9t9OKQksAEBIXfFnHoWXwAIABNWHj+01uAQWACCqCj9ugQUAhA6qs/dJ1Xx8AgsACBFWNUMq9888evwCCwDoMqpaBNXZx+YSIQDQbVj1HFU5j1tgAQDNoipqUB0RWACAsBJYAEDEsBo9qgQWACCsBBYAECmsZowqgQUACCuBBQD0HFfCSmABAMJKYAEAwkpgAQBDxpWwElgAQKGwElcCCwAoGFfCSmABAMKquS9PAQCIK3FVlgULAISVsCrMggUA4kpcFWbBAoDJ40pYCSwAoFBYiSuBBQAUjCthVZd7sABAXCGwAABx1TeXCAFAWFGYBQsAxBUCCwAQVwILABBXU3EPFgAIKwqzYAGAuEJgAQDiSmABAOJqKu7BAoDAcSWs+mTBAgBxhcACAHElrgQWACCuBBYAIK44z03uABAgrMRVLBYsABBXCCwAEFcILABAXE3FPVgA0CFhFZsFCwDEFYVZsACgIR/DMCYLFgCIKwQWAIgrBBYAIK4EFgAgrhBYACCuEFgAIK4QWACAuEJgAYC4QmABgLhCYAEA4gqBBQAgsAAgFusVAgsAxBUCCwDEFQILAMQVAgsAEFcILAAQVwgsABBXCCwAQFwhsABAXCGwAEBcIbAAABBYAHCS9QqBBQDiCoEFAOIKgQUA4goEFgCIKwQWAIgrBBYAiCsEFgAAAgsAarFeIbAAQFwhsABAXCGwAEBcgcACAHGFwAIAGCvcb7dbUuwATHkStF5RydfWQQYA4go+DCyRBYC4gkKB9XpQiSwAxBV8GFieAgDEFVQILCsWAEDhwBJZAIzOekWTwBJZAIgrqBBYIgsAcQUVAgsAxBVUCCwrFgBA4cASWQBEZ72iy8ASWQCIK6gQWCILAHEFFQILAMQVVAgsKxYAQOHAElkA9M56RcjAElkAiCuoEFgiCwBxBRUCCwCACoFlxQKgB9YrhgoskQWAuIIKgSWyABBXUCGwAEBcQYXAsmIBABQOLJEFwFWsV0wVWCILAHEFFQJLZAEgrqBCYAEAUCGwrFgAlGS9QmCJLADEFdS7RCiyABBXCCwAAPoPLCsWAGdYrxBYIgsAcQXXBpbIAkBcIbAAAIgRWFYsAPZYrxBYIgsAcQV9BJbIAkBcIbAAQFxBjMCyYgEAAktkAVCY9QqBJbIAEFcQI7AAEFfiCoFVgRULABBYIguAk6xXCCyRBYC4gtiBBYC4AoFVgRULABBYIguAA9YrBJbIAkBcwZiBBQAgsCqwYgHEZr1CYIksAMQVzBFYIgtAXIHAAgAgRmBZsQBisF5BoMASWQDiCgSWyAIQVyCwAACYNrCsWAB9sV7BAIElsgDEFQgskQUgrkBgAQAwbWBZsQDasF7BwIElsgDEFQgskQUgrkBgAQAwbWBZsQDqsl7BhIElsgDEFQgskQUgrkBgAQAwbWBZsQDKsF6BwBJZAOIKBBYA4goEVjBWLABAYIksgOasVyCwRBaAuAKBBYC4AoE1CCsWACCwRBbAZaxXILBEFoC4AoEFACCwBmfFAlh//7NegcASWQDiCgSWyAIQVyCwAAAQWLmsWMBsrFcgsEQWgLgCgSWyAMQVCCwAAARWLVYsYFTWKxBYIgtAXIHAElkA4goQWADiChBYV7NiAQACS2QBbL5fWa9AYIksAHEFAgsAcQUIrCxWLABAYIksYELWKxBYIgtAXIHAQmQB4goQWAAAAisKKxbQC+sVCCyRBSCuQGAhsgBxBQgsAACBFZ0VC7ia9QoElsgCEFcgsBBZgLgCBBaAuAIE1qisWACAwBJZQADWKxBYiCxAXIHA8hSILEBcAQILAEBgzcqKBZxlvQKBhcgCxBUgsEQWIK4AgSWyAHEFCCwAAIFFdVYsYIv1CgQWIgsQV4DAElmAuAIElsgCxBUgsAAAEFjNWbFgXtYrEFiILEBcAQJLZAHiChBYIktkgbgCBBYA4goQWN2zYgGAwEJkAW+wXoHAQmQB4goQWCILEFeAwEJkgbgCBBYA4goQWMOwYgGAwEJkwdSsV4DAElmAuAIElsgSWSCuAIGFyAJxBQgsRBYgrgCBBQAgsPicFQvas14BAktkAeIKEFiILBBXgMBCZIG4AhBYIgsQV4DAQmSBuAIEFiILxBUgsBBZgLgCBBaAuAIEFtVZsQBAYCGyoCvWK0BgIbJAXAECC5EF4goQWIgsEFcAAguRBeIKEFiILBBXgMBCZIG4AhBYiCwQV4DAQmSBuAIEFiILAsVVSklcAQILkQWl4gpAYCGyoHBcWa4AgcUlkfXvCUdkIa4ABBYXnJRAXAEILE5wuRBxBSCwEFkgrgCBhcgCcQUILESWyEJcAQgsRBbiSlwBAguRBeIKEFiILJFFP2ElrgCBxVCRJbRoHVdHxymAwCJcZG2d5IgfLr0HtLgCevXtKeBsZL2e3JZlcXILHlTRH6/jD+jmPSp5R8JJzmv3ZlT3+tgdc0BvLFh85OfE9u8Jz5I1RlBF+T4ca4DAYujQElmCSlwBCCwuiCwnQUFV63tzXAFdv2+5BwsnQ1GVE9GOJ4BjFiyqnIT9DcMxgkqsAwgsRJagElcAAovxI+v1ROm+LEElrIAZ+CR3LgstIRE/rmo/RnEFCCwQWdWfo9kDU1wBAgtORpbQmtvWMSCuAIEFGZFlzWLvdd86RgBCvb/5HCx6O7l6Tq51v993f/35fFZ5nbz+wMgsWDTjkuF7z0nLuHr39+SGlbgCBBZUDgqXDNvICadSkeWSICCw4OLQWjsZC632cVUisqxWgMCCjiLr5+RMX2GW85ps/V5xBYzMJ7nTZWSt/TM7s5yU1/6Zoa3Q2bJ2Y/q7X3s2hIUVwC8LFl2H1rsn7Zm0uDE9x96lXXEFCCzoILLcm3U+nFpE1l5YiStAYEFnobV1Mp8ptK6+MT03rKxWAL/cg0WoyFo7iY94f9brfVgtL/kdhVVuGAPMwIJFyNDKPdnP7H6//3+JLqV0ezweRcLK5UAAgcVgkeWyYRuPx8NqBSCwEFpshVKu5/NptQIQWAit39CKGFu1gyUnsrY+V0tYAQgsJgitPVatc5G1FlfCCmDfkrxLMuKB/UZIRTr0P7kxPedrH4/H7qfAR3veAAQWNAqtKNFwNrKOvm6k5whAYEGHsdX7j0NuZO39/tFWPoCe+KBRprH3YaVr0RE9LtbiyloFcNH/1FuwmPbgz7zhvacflaMl6+fXI3+PAAILJoutnmLkJ6aOPgRUVAEILAgTWq1iJdJjBRBYQJGA+TRsavz5ftwBBBZMEVw1+fEGaMPfIoSTsdJjcAkqAIEFQwXX1dElpgAEFkwbXWcjTEABCCygYIQBENuXpwAAQGABAAgsAACBBQCAwAIAEFgAAAILAACBBQAgsAAABBYAAAILAEBgAQAILAAAgQUAgMACABBYAAACCwAAgQUAILAAAAQWAAACCwBAYAEACCwAAAQWAIDAAgAQWAAAAgsAAIEFACCwAAAEFgAAmf4DVd1BO270hYQAAAAASUVORK5CYII=)\n",
    "\n",
    "\n",
    "    Observation:\n",
    "        Type: Box(2)\n",
    "        Num    Observation               Min            Max\n",
    "        0      Car Position              -1.2           0.6\n",
    "        1      Car Velocity              -0.07          0.07\n",
    "    Actions:\n",
    "        Type: Discrete(3)\n",
    "        Num    Action\n",
    "        0      Accelerate to the Left\n",
    "        1      Don't accelerate\n",
    "        2      Accelerate to the Right\n",
    "        Note: This does not affect the amount of velocity affected by the\n",
    "        gravitational pull acting on the car.\n",
    "    Reward:\n",
    "         Reward of 0 is awarded if the agent reached the flag (position = 0.5)\n",
    "         on top of the mountain.\n",
    "         Reward of -1 is awarded if the position of the agent is less than 0.5.\n",
    "    Starting State:\n",
    "         The position of the car is assigned a uniform random value in\n",
    "         [-0.6 , -0.4].\n",
    "         The starting velocity of the car is always assigned to 0.\n",
    "    Episode Termination:\n",
    "         The car position is more than 0.5\n",
    "         Episode length is greater than 200\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "og59Z62aZJna"
   },
   "source": [
    "Create the environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tVY05GIhZEMM"
   },
   "outputs": [],
   "source": [
    "env = gym.make(\"MountainCar-v0\", render_mode=\"rgb_array\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IEF4P0DAZMAN"
   },
   "source": [
    "Create the model with tuned hyperparameters from the RL Zoo\n",
    "\n",
    "```yaml\n",
    "MountainCar-v0:\n",
    "  n_timesteps: !!float 1.2e5\n",
    "  policy: 'MlpPolicy'\n",
    "  learning_rate: !!float 4e-3\n",
    "  batch_size: 128\n",
    "  buffer_size: 10000\n",
    "  learning_starts: 1000\n",
    "  gamma: 0.98\n",
    "  target_update_interval: 600\n",
    "  train_freq: 16\n",
    "  gradient_steps: 8\n",
    "  exploration_fraction: 0.2\n",
    "  exploration_final_eps: 0.07\n",
    "  policy_kwargs: \"dict(net_arch=[256, 256])\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fbEcqWhqgDmH"
   },
   "outputs": [],
   "source": [
    "tensorboard_log = \"data/tb/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4-1scts3Y1c7",
    "outputId": "d8001eb5-3546-4d37-dd8b-32e53c3437c6"
   },
   "outputs": [],
   "source": [
    "dqn_model = DQN(\n",
    "    \"MlpPolicy\",\n",
    "    env,\n",
    "    verbose=1,\n",
    "    train_freq=16,\n",
    "    gradient_steps=8,\n",
    "    gamma=0.99,\n",
    "    exploration_fraction=0.2,\n",
    "    exploration_final_eps=0.07,\n",
    "    target_update_interval=600,\n",
    "    learning_starts=1000,\n",
    "    buffer_size=10000,\n",
    "    batch_size=128,\n",
    "    learning_rate=4e-3,\n",
    "    policy_kwargs=dict(net_arch=[256, 256]),\n",
    "    tensorboard_log=tensorboard_log,\n",
    "    seed=2,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eNoFwsCPZQuz"
   },
   "source": [
    "Evaluate the agent before training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Qn0C7RHyZTHA",
    "outputId": "61236eac-6bf0-4eee-b749-251bbd6f9d0c"
   },
   "outputs": [],
   "source": [
    "mean_reward, std_reward = evaluate_policy(\n",
    "    dqn_model,\n",
    "    dqn_model.get_env(),\n",
    "    deterministic=True,\n",
    "    n_eval_episodes=20,\n",
    ")\n",
    "\n",
    "print(f\"mean_reward:{mean_reward:.2f} +/- {std_reward:.2f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kM0m1NyAgVhR"
   },
   "outputs": [],
   "source": [
    "# Optional: Monitor training in tensorboard\n",
    "# %load_ext tensorboard\n",
    "# %tensorboard --logdir $tensorboard_log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dUoPUfQ4ZexC"
   },
   "source": [
    "We will first train the agent until convergence and then analyse the learned q-value function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9wKP0gKjZgWZ"
   },
   "outputs": [],
   "source": [
    "dqn_model.learn(int(1.2e5), log_interval=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tI8xh463Zi7M"
   },
   "source": [
    "Evaluate after training, the mean episodic reward should have improved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "z952YogYZ8yD",
    "outputId": "5b8f02b6-45cc-4bf0-a47a-54ec0dfce3c2"
   },
   "outputs": [],
   "source": [
    "mean_reward, std_reward = evaluate_policy(dqn_model, dqn_model.get_env(), deterministic=True, n_eval_episodes=20)\n",
    "\n",
    "print(f\"mean_reward:{mean_reward:.2f} +/- {std_reward:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xVm9QPNVwKXN"
   },
   "source": [
    "### Prepare video recording"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MPyfQxD5z26J"
   },
   "outputs": [],
   "source": [
    "# Set up fake display; otherwise rendering will fail\n",
    "import os\n",
    "os.system(\"Xvfb :1 -screen 0 1024x768x24 &\")\n",
    "os.environ['DISPLAY'] = ':1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "SLzXxO8VMD6N"
   },
   "outputs": [],
   "source": [
    "import base64\n",
    "from pathlib import Path\n",
    "\n",
    "from IPython import display as ipythondisplay\n",
    "\n",
    "\n",
    "def show_videos(video_path=\"\", prefix=\"\"):\n",
    "    \"\"\"\n",
    "    Taken from https://github.com/eleurent/highway-env\n",
    "\n",
    "    :param video_path: (str) Path to the folder containing videos\n",
    "    :param prefix: (str) Filter the video, showing only the only starting with this prefix\n",
    "    \"\"\"\n",
    "    html = []\n",
    "    for mp4 in Path(video_path).glob(\"{}*.mp4\".format(prefix)):\n",
    "        video_b64 = base64.b64encode(mp4.read_bytes())\n",
    "        html.append(\n",
    "            \"\"\"<video alt=\"{}\" autoplay \n",
    "                    loop controls style=\"height: 400px;\">\n",
    "                    <source src=\"data:video/mp4;base64,{}\" type=\"video/mp4\" />\n",
    "                </video>\"\"\".format(\n",
    "                mp4, video_b64.decode(\"ascii\")\n",
    "            )\n",
    "        )\n",
    "    ipythondisplay.display(ipythondisplay.HTML(data=\"<br>\".join(html)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LTRNUfulOGaF"
   },
   "source": [
    "We will record a video using the [VecVideoRecorder](https://stable-baselines.readthedocs.io/en/master/guide/vec_envs.html#vecvideorecorder) wrapper, you will learn about those wrapper in the next notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Trag9dQpOIhx"
   },
   "outputs": [],
   "source": [
    "from stable_baselines3.common.vec_env import VecVideoRecorder, DummyVecEnv\n",
    "\n",
    "\n",
    "def record_video(\n",
    "    env_id,\n",
    "    model,\n",
    "    video_length=500,\n",
    "    prefix=\"\",\n",
    "    video_folder=\"videos/\",\n",
    "):\n",
    "    \"\"\"\n",
    "    :param env_id: (str)\n",
    "    :param model: (RL model)\n",
    "    :param video_length: (int)\n",
    "    :param prefix: (str)\n",
    "    :param video_folder: (str)\n",
    "    \"\"\"\n",
    "    eval_env = DummyVecEnv([lambda: gym.make(env_id, render_mode=\"rgb_array\")])\n",
    "    # Start the video at step=0 and record 500 steps\n",
    "    eval_env = VecVideoRecorder(\n",
    "        eval_env,\n",
    "        video_folder=video_folder,\n",
    "        record_video_trigger=lambda step: step == 0,\n",
    "        video_length=video_length,\n",
    "        name_prefix=prefix,\n",
    "    )\n",
    "\n",
    "    obs = eval_env.reset()\n",
    "    for _ in range(video_length):\n",
    "        action, _ = model.predict(obs, deterministic=False)\n",
    "        obs, _, _, _ = eval_env.step(action)\n",
    "\n",
    "    # Close the video recorder\n",
    "    eval_env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qFiOqKE3aDzI"
   },
   "source": [
    "## Visualize trained agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "MvVGX13xaGbf",
    "outputId": "61b5d454-7c8f-441e-c3cd-57158043b5d6"
   },
   "outputs": [],
   "source": [
    "record_video(\"MountainCar-v0\", dqn_model, video_length=500, prefix=\"dqn-mountaincar\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tHaYvk8uaK70"
   },
   "outputs": [],
   "source": [
    "show_videos(\"videos\", prefix=\"dqn\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TY00dtf4aRLQ"
   },
   "source": [
    "## Visualize Q-values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cP3YH2x7rieM"
   },
   "source": [
    "### Exercise (5 minutes): Retrieve q-values\n",
    "\n",
    "The function will be used to retrieve the learned q-values for a given state (`observation` in the code).\n",
    "\n",
    "The q-network from SB3 DQN can be accessed via `model.q_net` and is a PyTorch module (you can therefore call `.forward()` on it).\n",
    "\n",
    "You need to convert the observation to a PyTorch tensor and then convert the resulting q-values to numpy array.\n",
    "\n",
    "Note: It is recommended to use `with th.no_grad():` context to save computation and memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QiCp8OpCbKZW"
   },
   "outputs": [],
   "source": [
    "def get_q_values(model: DQN, obs: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    Retrieve Q-values for a given observation.\n",
    "\n",
    "    :param model: a DQN model\n",
    "    :param obs: a single observation\n",
    "    :return: the associated q-values for the given observation\n",
    "    \"\"\"\n",
    "    assert model.get_env().observation_space.contains(obs), f\"Invalid observation of shape {obs.shape}: {obs}\"\n",
    "    ### YOUR CODE HERE\n",
    "    # Retrieve q-values for the given observation and convert them to numpy\n",
    "    \n",
    "    ### END OF YOUR CODE\n",
    "    assert isinstance(q_values, np.ndarray), \"The returned q_values is not a numpy array\"\n",
    "    assert q_values.shape == (3,), f\"Wrong shape: (3,) was expected but got {q_values.shape}\"\n",
    "\n",
    "    return q_values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jcudYVkdbB0e"
   },
   "source": [
    "### Q-values for the initial state\n",
    "\n",
    "Let's reset the environment to start a new episode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "izFixcWgaVe3"
   },
   "outputs": [],
   "source": [
    "obs, _ = env.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6ouWnFaut0KB"
   },
   "source": [
    "we plot the rendered environment to visualize it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 266
    },
    "id": "NF1L_1Gfal-g",
    "outputId": "268ac65c-aad7-4626-f7a2-9556d079b60e"
   },
   "outputs": [],
   "source": [
    "plt.axis('off')\n",
    "plt.imshow(env.render())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rtKIGyJ-bEQO"
   },
   "source": [
    "### Exercise (5 minutes): predict taken action according to q-values\n",
    "\n",
    "Using the `get_q_values()` function, retrieve the q-values for the initial observation, print them for each action (\"left\", \"nothing\", \"right\") and print the action that the greedy (deterministic) policy would follow (i.e., the action with the highest q-value for that state)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BPegCYbSuY-f"
   },
   "outputs": [],
   "source": [
    "action_str = [\"Left\", \"Nothing\", \"Right\"]  # action=0 -> go left, action=1 -> do nothing, action=2 -> go right"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bv90MTHvaqbV"
   },
   "outputs": [],
   "source": [
    "### YOUR CODE HERE\n",
    "# Retrieve q-values for the initial state\n",
    "# You should use `get_q_values()`\n",
    "\n",
    "### END OF YOUR CODE\n",
    "\n",
    "print(f\"Q-value of the initial state left={q_value_left:.2f} nothing={q_value_nothing:.2f} right={q_value_right:.2f}\")\n",
    "\n",
    "### YOUR CODE HERE\n",
    "# Compute the action taken in the initilal state according to q-values \n",
    "# when following a greedy strategy\n",
    "\n",
    "\n",
    "## END of your code here\n",
    "\n",
    "print(f\"Action taken by the greedy policy in the initial state: {action_str[action]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ov0UwoBzcaDv"
   },
   "source": [
    "The q-value of the initial state corresponds to how much (discounted) reward the agent expects to get in this episode.\n",
    "\n",
    "We will compare the estimated q-value to the discounted return of the episode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dhhF-GJccVne"
   },
   "outputs": [],
   "source": [
    "initial_q_value = q_values.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JueeSE1xcQpK"
   },
   "source": [
    "## Step until the end of the episode\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "V_OYobAab8SF"
   },
   "outputs": [],
   "source": [
    "episode_rewards = []\n",
    "done = False\n",
    "i = 0\n",
    "\n",
    "while not done:\n",
    "    i += 1\n",
    "\n",
    "    # Display current state\n",
    "    plt.imshow(env.render(mode=\"rgb_array\"))\n",
    "    plt.show()\n",
    "\n",
    "    # Retrieve q-value\n",
    "    q_values = get_q_values(dqn_model, obs)\n",
    "\n",
    "    # Take greedy-action\n",
    "    action, _ = dqn_model.predict(obs, deterministic=True)\n",
    "\n",
    "    print(f\"Q-value of the current state left={q_values[0]:.2f} nothing={q_values[1]:.2f} right={q_values[2]:.2f}\")\n",
    "    print(f\"Action: {action_str[action]}\")\n",
    "\n",
    "    obs, reward, terminated, truncated, info = env.step(action)\n",
    "    \n",
    "    done = terminated or truncated\n",
    "\n",
    "    episode_rewards.append(reward)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YBEomET1wkjN"
   },
   "source": [
    "### Exercise (3 minutes): compare estimated initial q-value with actual discounted return\n",
    "\n",
    "Compute the discounted return (sum of discounted reward) of the episode and compare it to the initial estimated q-value.\n",
    "\n",
    "Note: You will need to use the discount factor `dqn_model.gamma`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Om4NNW2VdnM9"
   },
   "outputs": [],
   "source": [
    "sum_discounted_rewards = 0\n",
    "\n",
    "### YOUR CODE HERE\n",
    "# Compute the sum of discounted reward for the last episode\n",
    "# using `episode_rewards` list and `dqn_model.gamma` discount factor\n",
    "\n",
    "### END OF YOUR CODE\n",
    "\n",
    "print(f\"Sum discounted rewards: {sum_discounted_rewards:.2f}, initial q-value {initial_q_value:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZloxyPKPg9HX"
   },
   "source": [
    "## Exercise (30 minutes): Double DQN\n",
    "\n",
    "In DQN, the same network is responsible for selecting and estimating the best next action (in the TD-target) and that may lead to over-estimation (the action which q-value is over-estimated will be chosen more often and this slow down training).\n",
    "\n",
    "To reduce over-estimation, double q-learning (and then double DQN) was proposed. It decouples the action selection from the value estimation.\n",
    "\n",
    "Concretely, in DQN, the target q-value is defined as:\n",
    "\n",
    "$$Y^{DQN}_{t} = r_{t+1} + \\gamma{Q}\\left(s_{t+1}, \\arg\\max_{a}Q\\left(s_{t+1}, a; \\mathbb{\\theta}_{target}\\right); \\mathbb{\\theta}_{target}\\right)$$\n",
    "\n",
    "where the target network `q_net_target` with parameters $\\mathbb{\\theta}_{target}$ is used for both action selection and estimation, and can therefore be rewritten:\n",
    "\n",
    "$$Y^{DQN}_{t} = r_{t+1} + \\gamma \\max_{a}{Q}\\left(s_{t+1}, a; \\mathbb{\\theta}_{target}\\right)$$\n",
    "\n",
    "Double DQN uses the online network `q_net` with parameters $\\mathbb{\\theta}_{online}$ to select the action and the target network `q_net_target` to estimate the associated q-values:\n",
    "\n",
    "$$Y^{DoubleDQN}_{t} = r_{t+1} + \\gamma{Q}\\left(s_{t+1}, \\arg\\max_{a}Q\\left(s_{t+1}, a; \\mathbb{\\theta}_{online}\\right); \\mathbb{\\theta}_{target}\\right)$$\n",
    "\n",
    "\n",
    "The goal in this exercise is for you to write the update method for `DoubleDQN`.\n",
    "\n",
    "You will need to:\n",
    "\n",
    "1. Sample replay buffer data using `self.replay_buffer.sample(batch_size)`\n",
    "\n",
    "2. Compute the Double DQN target q-value using the next observations `replay_data.next_observation`, the online network `self.q_net`, the target network `self.q_net_target`, the rewards `replay_data.rewards` and the termination signals `replay_data.dones`. Be careful with the shape of each object ;)\n",
    "\n",
    "3. Compute the current q-value estimates using the online network `self.q_net`, the current observations `replay_data.observations` and the buffer actions `replay_data.actions`\n",
    "\n",
    "4. Compute the loss to train the q-network using L2 or Huber loss (`F.smooth_l1_loss`)\n",
    "\n",
    "\n",
    "Link: https://paperswithcode.com/method/double-q-learning\n",
    "\n",
    "Paper: https://arxiv.org/abs/1509.06461\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4227ILqjg8b4"
   },
   "outputs": [],
   "source": [
    "from torch.nn import functional as F\n",
    "\n",
    "class DoubleDQN(DQN):\n",
    "    def train(self, gradient_steps: int, batch_size: int = 100) -> None:\n",
    "        # Switch to train mode (this affects batch norm / dropout)\n",
    "        self.policy.set_training_mode(True)\n",
    "        # Update learning rate according to schedule\n",
    "        self._update_learning_rate(self.policy.optimizer)\n",
    "\n",
    "        losses = []\n",
    "        for _ in range(gradient_steps):\n",
    "            ### YOUR CODE HERE\n",
    "            # Sample replay buffer\n",
    "            replay_data = ...\n",
    "\n",
    "            # Do not backpropagate gradient to the target network\n",
    "            with th.no_grad():\n",
    "                # Compute the next Q-values using the target network\n",
    "                next_q_values = ...\n",
    "                # Decouple action selection from value estimation\n",
    "                # Compute q-values for the next observation using the online q net\n",
    "                next_q_values_online = ...\n",
    "                # Select action with online network\n",
    "                next_actions_online = ...\n",
    "                # Estimate the q-values for the selected actions using target q network\n",
    "                next_q_values = ...\n",
    "               \n",
    "                # 1-step TD target\n",
    "                target_q_values = ...\n",
    "\n",
    "            # Get current Q-values estimates\n",
    "            current_q_values = ...\n",
    "\n",
    "            # Retrieve the q-values for the actions from the replay buffer\n",
    "            current_q_values = th.gather(current_q_values, dim=1, index=replay_data.actions.long())\n",
    "\n",
    "            # Check the shape\n",
    "            assert current_q_values.shape == target_q_values.shape\n",
    "\n",
    "            # Compute loss (L2 or Huber loss)\n",
    "            loss = ...\n",
    "\n",
    "            ### END OF YOUR CODE\n",
    "            \n",
    "            losses.append(loss.item())\n",
    "\n",
    "            # Optimize the q-network\n",
    "            self.policy.optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            # Clip gradient norm\n",
    "            th.nn.utils.clip_grad_norm_(self.policy.parameters(), self.max_grad_norm)\n",
    "            self.policy.optimizer.step()\n",
    "\n",
    "        # Increase update counter\n",
    "        self._n_updates += gradient_steps\n",
    "\n",
    "        self.logger.record(\"train/n_updates\", self._n_updates, exclude=\"tensorboard\")\n",
    "        self.logger.record(\"train/loss\", np.mean(losses))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dG3fpWWOg_AS"
   },
   "source": [
    "## Monitoring Evolution of the Estimated q-value\n",
    "\n",
    "\n",
    "Here we create a SB3 callback to over-estimate initial q-values and then monitor their evolution over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BLbQ9RhUpMOl"
   },
   "outputs": [],
   "source": [
    "from torch.nn import functional as F\n",
    "\n",
    "from stable_baselines3.common.callbacks import BaseCallback\n",
    "\n",
    "\n",
    "class MonitorQValueCallback(BaseCallback):\n",
    "    \"\"\"\n",
    "    Callback to monitor the evolution of the q-value\n",
    "    for the initial state.\n",
    "    It allows to artificially over-estimate a q-value for initial states.\n",
    "\n",
    "    \"\"\"\n",
    "    def __init__(self, sample_interval: int = 2500):\n",
    "        super().__init__()\n",
    "        self.timesteps = []\n",
    "        self.max_q_values = []\n",
    "        self.sample_interval = sample_interval\n",
    "        n_samples = 512\n",
    "        env = gym.make(\"MountainCar-v0\")\n",
    "        # Sample initial states that will be used to monitor the estimated q-value\n",
    "        self.start_obs = np.array([env.reset() for _ in range(n_samples)])\n",
    "    \n",
    "    def _on_training_start(self) -> None:\n",
    "        # Create overestimation\n",
    "        obs = th.tensor(self.start_obs, device=self.model.device).float()\n",
    "        # Over-estimate going left q-value for the initial states\n",
    "        target_q_values = th.ones((len(obs), 1), device=self.model.device).float() * 100\n",
    "\n",
    "        for _ in range(100):\n",
    "            # Get current Q-values estimates\n",
    "            current_q_values = self.model.q_net(obs)\n",
    "\n",
    "            # Over-estimate going left\n",
    "            current_q_values = th.gather(current_q_values, dim=1, index=th.zeros((len(obs), 1), device=self.model.device).long())\n",
    "\n",
    "            loss = F.mse_loss(current_q_values, target_q_values)\n",
    "\n",
    "            # Optimize the policy\n",
    "            self.model.policy.optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            self.model.policy.optimizer.step()\n",
    "\n",
    "    def _on_step(self) -> bool:\n",
    "        # Sample q-values\n",
    "        if self.n_calls % self.sample_interval == 0:\n",
    "            # Monitor estimated q-values using current model\n",
    "            obs = th.tensor(self.start_obs, device=self.model.device).float()\n",
    "            with th.no_grad():\n",
    "                q_values = self.model.q_net(obs).cpu().numpy()\n",
    "\n",
    "            self.logger.record(\"train/max_q_value\", float(q_values.max()))\n",
    "            self.timesteps.append(self.num_timesteps)\n",
    "            self.max_q_values.append(q_values.max())\n",
    "        return True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "36dtW1c4xQUG"
   },
   "source": [
    "## Evolution of the q-value with initial over-estimation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j-XTmT6SxdOa"
   },
   "source": [
    "### DQN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZIo1EDx2xcwZ",
    "outputId": "aac8b11a-e4d6-4278-945e-f48070289f1d"
   },
   "outputs": [],
   "source": [
    "dqn_model = DQN(\n",
    "    \"MlpPolicy\",\n",
    "    \"MountainCar-v0\",\n",
    "    verbose=1,\n",
    "    train_freq=16,\n",
    "    gradient_steps=8,\n",
    "    gamma=0.99,\n",
    "    exploration_fraction=0.2,\n",
    "    exploration_final_eps=0.07,\n",
    "    target_update_interval=5000,\n",
    "    learning_starts=1000,\n",
    "    buffer_size=25000,\n",
    "    batch_size=128,\n",
    "    learning_rate=4e-3,\n",
    "    policy_kwargs=dict(net_arch=[256, 256]),\n",
    "    tensorboard_log=tensorboard_log,\n",
    "    seed=102,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Vd62HoWsxfBJ"
   },
   "source": [
    "Define the callback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wcULdR48xhH5"
   },
   "outputs": [],
   "source": [
    "monitor_dqn_value_cb = MonitorQValueCallback()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5Z_cFIapxhR7"
   },
   "outputs": [],
   "source": [
    "dqn_model.learn(total_timesteps=int(4e4), callback=monitor_dqn_value_cb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NxAGOezBx3GS"
   },
   "source": [
    "### Double DQN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xYNcgKsSegj0"
   },
   "outputs": [],
   "source": [
    "double_q = DoubleDQN(\"MlpPolicy\",\n",
    "            \"MountainCar-v0\",\n",
    "            verbose=1,\n",
    "            train_freq=16,\n",
    "            gradient_steps=8,\n",
    "            gamma=0.99,\n",
    "            exploration_fraction=0.2,\n",
    "            exploration_final_eps=0.07,\n",
    "            target_update_interval=5000,\n",
    "            learning_starts=1000,\n",
    "            buffer_size=25000,\n",
    "            batch_size=128,\n",
    "            learning_rate=4e-3,\n",
    "            policy_kwargs=dict(net_arch=[256, 256]),\n",
    "            tensorboard_log=tensorboard_log,\n",
    "            seed=102)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JWhQTCXQyBJZ"
   },
   "outputs": [],
   "source": [
    "monitor_double_q_value_cb = MonitorQValueCallback()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mvveywQUkEpW"
   },
   "outputs": [],
   "source": [
    "double_q.learn(int(4e4), log_interval=10, callback=monitor_double_q_value_cb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rPqcCnbnyIR5"
   },
   "source": [
    "### Evolution of the max q-value for start states over time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jNeKzUoaa6_K"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(6, 3), dpi=200)\n",
    "plt.title(\"Evolution of max q-value for start states over time\")\n",
    "plt.plot(monitor_dqn_value_cb.timesteps, monitor_dqn_value_cb.max_q_values, label=\"DQN\")\n",
    "plt.plot(monitor_double_q_value_cb.timesteps, monitor_double_q_value_cb.max_q_values, label=\"Double DQN\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "WqRixViAv6gT"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "xVm9QPNVwKXN"
   ],
   "name": "dqn_sb3.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}