8-6-natural-language-processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7a7f37e9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Natural Language Processing\n",
    "\n",
    "&copy; 李浩东 haodongli@zju.edu.cn\n",
    "\n",
    "- Word Embedding (word2vec)\n",
    "- Word Embedding with Global Vectors (GloVe)\n",
    "- Subword Embedding\n",
    "- Bidirectional Encoder Representations from Transformers (BERT)\n",
    "- Pretraining BERT\n",
    "\n",
    "---\n",
    "\n",
    "- Sentiment Analysis: Using RNN & CNN\n",
    "- Natural Language Inference: Fine-Tuning BERT\n",
    "- Chat with ChatGPT\n",
    "- nanoGPT: The simplest, fastest repository for training/finetuning medium-sized GPTs\n",
    "\n",
    "![gpt-4-motif.svg](./images/8/GPT-4-cover.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6fdfa69",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Word Embedding (word2vec)\n",
    "\n",
    "### One-Hot Vectors Are a Bad Choice\n",
    "\n",
    "- Although one-hot word vectors are easy to construct, they are usually not a good choice.\n",
    "- A main reason is that one-hot word vectors cannot accurately express the similarity between different words, such as the *cosine similarity* that we often use. \n",
    "- For vectors $\\mathbf{x}, \\mathbf{y} \\in \\mathbb{R}^d$, their cosine similarity is the cosine of the angle between them:\n",
    "\n",
    "\n",
    "$$\\frac{\\mathbf{x}^\\top \\mathbf{y}}{\\|\\mathbf{x}\\| \\|\\mathbf{y}\\|} \\in [-1, 1].$$\n",
    "\n",
    "\n",
    "- Since the cosine similarity between one-hot vectors of any two different words is 0, one-hot vectors cannot encode similarities among words.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccdfff28",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Self-Supervised word2vec\n",
    "\n",
    "- The [word2vec](https://code.google.com/archive/p/word2vec/) tool was proposed to address the above issue.\n",
    "- It maps each word to a fixed-length vector, and  these vectors can better express the similarity and analogy relationship among different words.\n",
    "    - skip-gram\n",
    "    - ontinuous bag of words (CBOW)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d28d031",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The Skip-Gram Model\n",
    "\n",
    "![skip-gram.svg](./images/8/skip-gram.svg)\n",
    "\n",
    "- The *skip-gram* model assumes that a word can be used to generate its surrounding words in a text sequence.\n",
    "- Take the text sequence \"the\", \"man\", \"loves\", \"his\", \"son\" as an example.\n",
    "- Let's choose \"loves\" as the *center word* and set the context window size to 2. given the center word \"loves\", the skip-gram model considers the conditional probability for generating the *context words*: \"the\", \"man\", \"his\", and \"son\", which are no more than 2 words away from the center word:\n",
    "\n",
    "$$P(\\textrm{\"the\"},\\textrm{\"man\"},\\textrm{\"his\"},\\textrm{\"son\"}\\mid\\textrm{\"loves\"}).$$\n",
    "\n",
    "- Assume that the context words are independently generated given the center word (i.e., conditional independence). In this case, the above conditional probability can be rewritten as\n",
    "\n",
    "$$P(\\textrm{\"the\"}\\mid\\textrm{\"loves\"})\\cdot P(\\textrm{\"man\"}\\mid\\textrm{\"loves\"})\\cdot P(\\textrm{\"his\"}\\mid\\textrm{\"loves\"})\\cdot P(\\textrm{\"son\"}\\mid\\textrm{\"loves\"}).$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12af2f6a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- In the skip-gram model, each word has two $d$-dimensional-vector representations for calculating conditional probabilities.\n",
    "- More concretely, for any word with index $i$ in the dictionary, denote by $\\mathbf{v}_i\\in\\mathbb{R}^d$ and $\\mathbf{u}_i\\in\\mathbb{R}^d$ its two vectors when used as a *center* word and a *context* word, respectively. The conditional probability of generating any context word $w_o$ (with index $o$ in the dictionary) given the center word $w_c$ (with index $c$ in the dictionary) can be modeled by a softmax operation on vector dot products:\n",
    "\n",
    "\n",
    "$$P(w_o \\mid w_c) = \\frac{\\text{exp}(\\mathbf{u}_o^\\top \\mathbf{v}_c)}{ \\sum_{i \\in \\mathcal{V}} \\text{exp}(\\mathbf{u}_i^\\top \\mathbf{v}_c)},$$\n",
    "\n",
    "- where the vocabulary index set $\\mathcal{V} = \\{0, 1, \\ldots, |\\mathcal{V}|-1\\}$.\n",
    "- Given a text sequence of length $T$, where the word at time step $t$ is denoted as $w^{(t)}$. Assume that context words are independently generated given any center word. \n",
    "- For context window size $m$, the likelihood function of the skip-gram model is the probability of generating all context words given any center word:\n",
    "\n",
    "\n",
    "$$ \\prod_{t=1}^{T} \\prod_{-m \\leq j \\leq m,\\ j \\neq 0} P(w^{(t+j)} \\mid w^{(t)}),$$\n",
    "\n",
    "- where any time step that is less than $1$ or greater than $T$ can be omitted."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edeff878",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Training\n",
    "\n",
    "- The skip-gram model parameters are the center word vector and context word vector for each word in the vocabulary.\n",
    "- In training, we learn the model parameters by maximizing the likelihood function (i.e., maximum likelihood estimation). This is equivalent to minimizing the following loss function:\n",
    "\n",
    "$$ - \\sum_{t=1}^{T} \\sum_{-m \\leq j \\leq m,\\ j \\neq 0} \\text{log}\\, P(w^{(t+j)} \\mid w^{(t)}).$$\n",
    "\n",
    "- When using stochastic gradient descent to minimize the loss, in each iteration we can randomly sample a shorter subsequence to calculate the (stochastic) gradient for this subsequence to update the model parameters.\n",
    "- To calculate this (stochastic) gradient, we need to obtain the gradients of the log conditional probability with respect to the center word vector and the context word vector."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f96f61ad",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- The log conditional probability involving any pair of the center word $w_c$ and the context word $w_o$ is\n",
    "\n",
    "\n",
    "$$\\log P(w_o \\mid w_c) =\\mathbf{u}_o^\\top \\mathbf{v}_c - \\log\\left(\\sum_{i \\in \\mathcal{V}} \\text{exp}(\\mathbf{u}_i^\\top \\mathbf{v}_c)\\right).$$\n",
    "\n",
    "- Through differentiation, we can obtain its gradient with respect to the center word vector $\\mathbf{v}_c$ as\n",
    "\n",
    "$$\\begin{aligned}\\frac{\\partial \\text{log}\\, P(w_o \\mid w_c)}{\\partial \\mathbf{v}_c}&= \\mathbf{u}_o - \\frac{\\sum_{j \\in \\mathcal{V}} \\exp(\\mathbf{u}_j^\\top \\mathbf{v}_c)\\mathbf{u}_j}{\\sum_{i \\in \\mathcal{V}} \\exp(\\mathbf{u}_i^\\top \\mathbf{v}_c)}\\\\&= \\mathbf{u}_o - \\sum_{j \\in \\mathcal{V}} \\left(\\frac{\\text{exp}(\\mathbf{u}_j^\\top \\mathbf{v}_c)}{ \\sum_{i \\in \\mathcal{V}} \\text{exp}(\\mathbf{u}_i^\\top \\mathbf{v}_c)}\\right) \\mathbf{u}_j\\\\&= \\mathbf{u}_o - \\sum_{j \\in \\mathcal{V}} P(w_j \\mid w_c) \\mathbf{u}_j.\\end{aligned}$$\n",
    "\n",
    "- The gradients for the other word vectors can be obtained in the same way.\n",
    "- After training, for any word with index $i$ in the dictionary, we obtain both word vectors $\\mathbf{v}_i$ (as the center word) and $\\mathbf{u}_i$ (as the context word).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6beb6e5f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The Continuous Bag of Words (CBOW) Model\n",
    "\n",
    "![cbow.svg](./images/8/cbow.svg)\n",
    "\n",
    "- The *continuous bag of words* (CBOW) model is similar to the skip-gram model. The major difference from the skip-gram model is that the continuous bag of words model assumes that a center word is generated based on its surrounding context words in the text sequence.\n",
    "\n",
    "$$P(\\textrm{\"loves\"}\\mid\\textrm{\"the\"},\\textrm{\"man\"},\\textrm{\"his\"},\\textrm{\"son\"}).$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "522095be",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Since there are multiple context words in the continuous bag of words model, these context word vectors are averaged in the calculation of the conditional probability.\n",
    "- Specifically, for any word with index $i$ in the dictionary, denote by $\\mathbf{v}_i\\in\\mathbb{R}^d$ and $\\mathbf{u}_i\\in\\mathbb{R}^d$ its two vectors when used as a *context* word and a *center* word (meanings are switched in the skip-gram model), respectively.\n",
    "- The conditional probability of generating any center word $w_c$ (with index $c$ in the dictionary) given its surrounding context words $w_{o_1}, \\ldots, w_{o_{2m}}$ (with index $o_1, \\ldots, o_{2m}$ in the dictionary) can be modeled by\n",
    "\n",
    "$$P(w_c \\mid w_{o_1}, \\ldots, w_{o_{2m}}) = \\frac{\\text{exp}\\left(\\frac{1}{2m}\\mathbf{u}_c^\\top (\\mathbf{v}_{o_1} + \\ldots + \\mathbf{v}_{o_{2m}}) \\right)}{ \\sum_{i \\in \\mathcal{V}} \\text{exp}\\left(\\frac{1}{2m}\\mathbf{u}_i^\\top (\\mathbf{v}_{o_1} + \\ldots + \\mathbf{v}_{o_{2m}}) \\right)}.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a23e3acf",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- For brevity, let $\\mathcal{W}_o= \\{w_{o_1}, \\ldots, w_{o_{2m}}\\}$ and $\\bar{\\mathbf{v}}_o = \\left(\\mathbf{v}_{o_1} + \\ldots + \\mathbf{v}_{o_{2m}} \\right)/(2m)$. Then it can be simplified as\n",
    "\n",
    "$$P(w_c \\mid \\mathcal{W}_o) = \\frac{\\exp\\left(\\mathbf{u}_c^\\top \\bar{\\mathbf{v}}_o\\right)}{\\sum_{i \\in \\mathcal{V}} \\exp\\left(\\mathbf{u}_i^\\top \\bar{\\mathbf{v}}_o\\right)}.$$\n",
    "\n",
    "- Given a text sequence of length $T$, where the word at time step $t$ is denoted as $w^{(t)}$.\n",
    "- For context window size $m$, the likelihood function of the continuous bag of words model is the probability of generating all center words given their context words:\n",
    "\n",
    "$$ \\prod_{t=1}^{T}  P(w^{(t)} \\mid  w^{(t-m)}, \\ldots, w^{(t-1)}, w^{(t+1)}, \\ldots, w^{(t+m)}).$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "716021bd",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Training\n",
    "\n",
    "- Training continuous bag of words models is almost the same as training skip-gram models. The maximum likelihood estimation of the continuous bag of words model is equivalent to minimizing the following loss function:\n",
    "\n",
    "\n",
    "\n",
    "$$  -\\sum_{t=1}^T  \\text{log}\\, P(w^{(t)} \\mid  w^{(t-m)}, \\ldots, w^{(t-1)}, w^{(t+1)}, \\ldots, w^{(t+m)}).$$\n",
    "\n",
    "- Notice that\n",
    "\n",
    "$$\\log\\,P(w_c \\mid \\mathcal{W}_o) = \\mathbf{u}_c^\\top \\bar{\\mathbf{v}}_o - \\log\\,\\left(\\sum_{i \\in \\mathcal{V}} \\exp\\left(\\mathbf{u}_i^\\top \\bar{\\mathbf{v}}_o\\right)\\right).$$\n",
    "\n",
    "- Through differentiation, we can obtain its gradient with respect to any context word vector $\\mathbf{v}_{o_i}$($i = 1, \\ldots, 2m$) as\n",
    "\n",
    "\n",
    "$$\\frac{\\partial \\log\\, P(w_c \\mid \\mathcal{W}_o)}{\\partial \\mathbf{v}_{o_i}} = \\frac{1}{2m} \\left(\\mathbf{u}_c - \\sum_{j \\in \\mathcal{V}} \\frac{\\exp(\\mathbf{u}_j^\\top \\bar{\\mathbf{v}}_o)\\mathbf{u}_j}{ \\sum_{i \\in \\mathcal{V}} \\text{exp}(\\mathbf{u}_i^\\top \\bar{\\mathbf{v}}_o)} \\right) = \\frac{1}{2m}\\left(\\mathbf{u}_c - \\sum_{j \\in \\mathcal{V}} P(w_j \\mid \\mathcal{W}_o) \\mathbf{u}_j \\right).$$\n",
    "\n",
    "- The gradients for the other word vectors can be obtained in the same way. Unlike the skip-gram model, the continuous bag of words model typically uses context word vectors as the word representations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c38fdc2a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Approximate Training\n",
    "\n",
    "- The main idea of the skip-gram model is using softmax operations to calculate the conditional probability of generating a context word $w_o$ based on the given center word $w_c$.\n",
    "- Due to the nature of the softmax operation, since a context word may be anyone in the dictionary $\\mathcal{V}$, the opposite of corresponding logarithmic loss contains the summation of items as many as the entire size of the vocabulary.\n",
    "- Unfortunately, the computational cost for such gradients that sum over a large dictionary (often with hundreds of thousands or millions of words) is huge!\n",
    "- In order to reduce the aforementioned computational complexity, this section will introduce two approximate training methods: *negative sampling* and *hierarchical softmax*."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c89eb2de",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Negative Sampling\n",
    "\n",
    "- Negative sampling modifies the original objective function. Given the context window of a center word $w_c$, the fact that any (context) word $w_o$ comes from this context window is considered as an event with the probability modeled by\n",
    "\n",
    "\n",
    "$$P(D=1\\mid w_c, w_o) = \\sigma(\\mathbf{u}_o^\\top \\mathbf{v}_c),$$\n",
    "\n",
    "- where $\\sigma$ uses the definition of the sigmoid activation function:\n",
    "\n",
    "$$\\sigma(x) = \\frac{1}{1+\\exp(-x)}.$$\n",
    "\n",
    "- Let's begin by maximizing the joint probability of all such events in text sequences to train word embeddings. Specifically, given a text sequence of length $T$, denote by $w^{(t)}$ the word at time step $t$ and let the context window size be $m$, consider maximizing the joint probability\n",
    "\n",
    "\n",
    "$$ \\prod_{t=1}^{T} \\prod_{-m \\leq j \\leq m,\\ j \\neq 0} P(D=1\\mid w^{(t)}, w^{(t+j)}).$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1de0b95e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$ \\prod_{t=1}^{T} \\prod_{-m \\leq j \\leq m,\\ j \\neq 0} P(D=1\\mid w^{(t)}, w^{(t+j)}).$$\n",
    "\n",
    "- The joint probability in is maximized to 1 only if all the word vectors are equal to infinity.\n",
    "- Of course, such results are meaningless. To make the objective function more meaningful, *negative sampling* adds negative examples sampled from a predefined distribution.\n",
    "- Denote by $S$ the event that a context word $w_o$ comes from the context window of a center word $w_c$. For this event involving $w_o$, from a predefined distribution $P(w)$ sample $K$ *noise words* that are not from this context window.\n",
    "- Denote by $N_k$ the event that a noise word $w_k$ ($k=1, \\ldots, K$) does not come from the context window of $w_c$. Assume that these events involving both the positive example and negative examples $S, N_1, \\ldots, N_K$ are mutually independent. Negative sampling rewrites the joint probability (involving only positive examples) as\n",
    "\n",
    "$$ \\prod_{t=1}^{T} \\prod_{-m \\leq j \\leq m,\\ j \\neq 0} P(w^{(t+j)} \\mid w^{(t)}),$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "073476ed",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$ \\prod_{t=1}^{T} \\prod_{-m \\leq j \\leq m,\\ j \\neq 0} P(w^{(t+j)} \\mid w^{(t)}),$$\n",
    "\n",
    "- where the conditional probability is approximated through events $S, N_1, \\ldots, N_K$:\n",
    "\n",
    "$$ P(w^{(t+j)} \\mid w^{(t)}) =P(D=1\\mid w^{(t)}, w^{(t+j)})\\prod_{k=1,\\ w_k \\sim P(w)}^K P(D=0\\mid w^{(t)}, w_k).$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd7ef1d9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Denote by $i_t$ and $h_k$ the indices of a word $w^{(t)}$ at time step $t$ of a text sequence and a noise word $w_k$, respectively. The logarithmic loss with respect to the conditional probabilities is\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "-\\log P(w^{(t+j)} \\mid w^{(t)})\n",
    "=& -\\log P(D=1\\mid w^{(t)}, w^{(t+j)}) - \\sum_{k=1,\\ w_k \\sim P(w)}^K \\log P(D=0\\mid w^{(t)}, w_k)\\\\\n",
    "=&-  \\log\\, \\sigma\\left(\\mathbf{u}_{i_{t+j}}^\\top \\mathbf{v}_{i_t}\\right) - \\sum_{k=1,\\ w_k \\sim P(w)}^K \\log\\left(1-\\sigma\\left(\\mathbf{u}_{h_k}^\\top \\mathbf{v}_{i_t}\\right)\\right)\\\\\n",
    "=&-  \\log\\, \\sigma\\left(\\mathbf{u}_{i_{t+j}}^\\top \\mathbf{v}_{i_t}\\right) - \\sum_{k=1,\\ w_k \\sim P(w)}^K \\log\\sigma\\left(-\\mathbf{u}_{h_k}^\\top \\mathbf{v}_{i_t}\\right).\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "- We can see that now the computational cost for gradients at each training step has nothing to do with the dictionary size, but linearly depends on $K$. When setting the hyperparameter $K$ to a smaller value, the computational cost for gradients at each training step with negative sampling is smaller."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd5ecc79",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Discussion\n",
    "\n",
    "You: What is negative sampling, for approximating the Word Embedding (word2vec) calculation?\n",
    "\n",
    "---\n",
    "\n",
    "ChatGPT: Negative sampling is a method for approximating the calculation of Word Embedding, specifically in the case of the popular word2vec algorithm. It aims to address the high computational cost associated with the original version of the algorithm, which requires computing the softmax function for each word in the vocabulary. \n",
    "\n",
    "In negative sampling, instead of considering all words in the vocabulary, **a small number (typically 5-20)** of \"negative\" words are sampled at random for each training example. These negative words are assumed to not appear in the context of the target word, and their embeddings are adjusted so as to minimize the likelihood of them appearing in the context. This is done in addition to optimizing the embeddings of the actual \"positive\" context words. \n",
    "\n",
    "By using negative sampling, the computational complexity of the word2vec algorithm is greatly reduced, while still producing word embeddings that capture meaningful semantic relationships between words.\n",
    "\n",
    "---\n",
    "\n",
    "You: Then the conditional probabilities of generating all the words in dictionary V is not equal to 1, is that sensible?\n",
    "\n",
    "---\n",
    "\n",
    "ChatGPT: When using negative sampling for word embedding approximations, **the conditional probabilities of generating all the words in the dictionary V may not sum to exactly 1**. This is because negative sampling only considers a subset of the vocabulary as \"negative\" samples, and the probabilities of these negative samples are adjusted to minimize the likelihood that they appear in the context. Therefore, the original true probabilities of generating these \"negative\" words may not be preserved in the final model.\n",
    "\n",
    "However, this does not necessarily mean that the model is not sensible or effective. Negative sampling is a common technique used in word embedding algorithms, and it has been shown to produce high-quality word embeddings that capture meaningful semantic relationships between words. The slight deviation from a probabilistic interpretation can be seen as an approximation that still yields desirable results with reduced computational complexity.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef202af7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Hierarchical Softmax\n",
    "\n",
    "- As an alternative approximate training method, *hierarchical softmax* uses the binary tree, a data structure illustrated below, where each leaf node of the tree represents a word in dictionary $\\mathcal{V}$.\n",
    "\n",
    "![hi-softmax.svg](./images/8/hi-softmax.svg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e03d9608",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Denote by $L(w)$ the number of nodes (including both ends) on the path from the root node to the leaf node representing word $w$ in the binary tree.\n",
    "- Let $n(w,j)$ be the $j^\\mathrm{th}$ node on this path, with its context word vector being $\\mathbf{u}_{n(w, j)}$. For example, $L(w_3) = 4$.\n",
    "- Hierarchical softmax approximates the conditional probability as\n",
    "\n",
    "$$P(w_o \\mid w_c) = \\prod_{j=1}^{L(w_o)-1} \\sigma\\left( [\\![  n(w_o, j+1) = \\text{leftChild}(n(w_o, j)) ]\\!] \\cdot \\mathbf{u}_{n(w_o, j)}^\\top \\mathbf{v}_c\\right),$$\n",
    "\n",
    "- where $\\text{leftChild}(n)$ is the left child node of node $n$: if $x$ is true, $[\\![x]\\!] = 1$; otherwise $[\\![x]\\!] = -1$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec1e1e3c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- To illustrate, let's calculate the conditional probability of generating word $w_3$ given word $w_c$ in :numref:`fig_hi_softmax`. This requires dot products between the word vector $\\mathbf{v}_c$ of $w_c$ and non-leaf node vectors on the path from the root to $w_3$, which is traversed left, right, then left:\n",
    "\n",
    "\n",
    "$$P(w_3 \\mid w_c) = \\sigma(\\mathbf{u}_{n(w_3, 1)}^\\top \\mathbf{v}_c) \\cdot \\sigma(-\\mathbf{u}_{n(w_3, 2)}^\\top \\mathbf{v}_c) \\cdot \\sigma(\\mathbf{u}_{n(w_3, 3)}^\\top \\mathbf{v}_c).$$\n",
    "\n",
    "- Since $\\sigma(x)+\\sigma(-x) = 1$, it holds that the conditional probabilities of generating all the words in dictionary $\\mathcal{V}$ based on any word $w_c$ sum up to one: \n",
    "\n",
    "$$\\sum_{w \\in \\mathcal{V}} P(w \\mid w_c) = 1.$$\n",
    "\n",
    "- Fortunately, since $L(w_o)-1$ is on the order of $\\mathcal{O}(\\text{log}_2|\\mathcal{V}|)$ due to the binary tree structure, when the dictionary size $\\mathcal{V}$ is huge, the computational cost for  each training step using hierarchical softmax is significantly reduced compared with that without approximate training."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6189187f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The Dataset for Pretraining Word Embeddings\n",
    "\n",
    "- Now that we know the technical details of the word2vec models and approximate training methods, let’s walk through their implementations.\n",
    "- In this section, we begin with the dataset for pretraining the word embedding model: the original format of the data will be transformed into minibatches that can be iterated over during training.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e4eeb51e",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import collections\n",
    "import math\n",
    "import os\n",
    "import random\n",
    "import torch\n",
    "from torch import nn\n",
    "from d2l import torch as d2l"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e937194",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- The dataset that we use here is [Penn Tree Bank (PTB)]( https://catalog.ldc.upenn.edu/LDC99T42). This corpus is sampled from Wall Street Journal articles, split into training, validation, and test sets.\n",
    "- In the original format, each line of the text file represents a sentence of words that are separated by spaces.\n",
    "- Here we treat each word as a token.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1aeedcfa",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'# sentences: 42069'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#@save\n",
    "d2l.DATA_HUB['ptb'] = (d2l.DATA_URL + 'ptb.zip',\n",
    "                       '319d85e578af0cdc590547f26231e4e31cdf1e42')\n",
    "\n",
    "#@save\n",
    "def read_ptb():\n",
    "    \"\"\"Load the PTB dataset into a list of text lines.\"\"\"\n",
    "    data_dir = d2l.download_extract('ptb')\n",
    "    # Read the training set\n",
    "    with open(os.path.join(data_dir, 'ptb.train.txt')) as f:\n",
    "        raw_text = f.read()\n",
    "    return [line.split() for line in raw_text.split('\\n')]\n",
    "\n",
    "sentences = read_ptb()\n",
    "f'# sentences: {len(sentences)}'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "499d2413",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- After reading the training set, we build a vocabulary for the corpus, where any word that appears less than 10 times is replaced by the “\\<unk\\>” token. Note that the original dataset also contains “\\<unk\\>” tokens that represent rare (unknown) words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5a45e1e6",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'vocab size: 6719'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab = d2l.Vocab(sentences, min_freq=10)\n",
    "f'vocab size: {len(vocab)}'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8a70460",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Subsampling\n",
    "\n",
    "- Text data typically have high-frequency words such as “the”, “a”, and “in”: they may even occur billions of times in very large corpora. However, these words often co-occur with many different words in context windows, providing little useful signals. \n",
    "- For instance, consider the word “chip” in a context window: intuitively its co-occurrence with a low-frequency word “intel” is more useful in training than the co-occurrence with a high-frequency word “a”. Moreover, training with vast amounts of (high-frequency) words is slow. Thus, when training word embedding models, high-frequency words can be subsampled (Mikolov et al., 2013).\n",
    "- Specifically, each indexed word $w_i$ in the dataset will be discarded with probability\n",
    "\n",
    "\n",
    "$$ P(w_i) = \\max\\left(1 - \\sqrt{\\frac{t}{f(w_i)}}, 0\\right),$$\n",
    "\n",
    "- where $f(w_i)$ is the ratio of the number of words $w_i$ to the total number of words in the dataset, and the constant $t$ is a hyperparameter ($10^{-4}$ in the experiment). We can see that only when the relative frequency $f(w_i) > t$  can the (high-frequency) word $w_i$ be discarded,  and the higher the relative frequency of the word,  the greater the probability of being discarded.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cdff69f2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "def subsample(sentences, vocab):\n",
    "    \"\"\"Subsample high-frequency words.\"\"\"\n",
    "    # Exclude unknown tokens ('<unk>')\n",
    "    sentences = [[token for token in line if vocab[token] != vocab.unk]\n",
    "                 for line in sentences]\n",
    "    counter = collections.Counter([\n",
    "        token for line in sentences for token in line])\n",
    "    num_tokens = sum(counter.values())\n",
    "\n",
    "    # Return True if `token` is kept during subsampling\n",
    "    def keep(token):\n",
    "        return(random.uniform(0, 1) <\n",
    "               math.sqrt(1e-4 / counter[token] * num_tokens))\n",
    "\n",
    "    return ([[token for token in line if keep(token)] for line in sentences],\n",
    "            counter)\n",
    "\n",
    "subsampled, counter = subsample(sentences, vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "44ca3247",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"262.190625pt\" height=\"182.179846pt\" viewBox=\"0 0 262.190625 182.179846\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-03-21T14:35:49.073282</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 182.179846 \n",
       "L 262.190625 182.179846 \n",
       "L 262.190625 -0 \n",
       "L 0 -0 \n",
       "L 0 182.179846 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 59.690625 144.623596 \n",
       "L 254.990625 144.623596 \n",
       "L 254.990625 8.723596 \n",
       "L 59.690625 8.723596 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 68.567898 144.623596 \n",
       "L 75.814651 144.623596 \n",
       "L 75.814651 123.430769 \n",
       "L 68.567898 123.430769 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: #1f77b4\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 86.684781 144.623596 \n",
       "L 93.931534 144.623596 \n",
       "L 93.931534 85.866285 \n",
       "L 86.684781 85.866285 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: #1f77b4\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 104.801664 144.623596 \n",
       "L 112.048417 144.623596 \n",
       "L 112.048417 76.742412 \n",
       "L 104.801664 76.742412 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: #1f77b4\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 122.918547 144.623596 \n",
       "L 130.1653 144.623596 \n",
       "L 130.1653 96.57227 \n",
       "L 122.918547 96.57227 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: #1f77b4\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 141.03543 144.623596 \n",
       "L 148.282183 144.623596 \n",
       "L 148.282183 124.590629 \n",
       "L 141.03543 124.590629 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: #1f77b4\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 159.152313 144.623596 \n",
       "L 166.399067 144.623596 \n",
       "L 166.399067 137.819442 \n",
       "L 159.152313 137.819442 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: #1f77b4\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 177.269196 144.623596 \n",
       "L 184.51595 144.623596 \n",
       "L 184.51595 143.084243 \n",
       "L 177.269196 143.084243 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: #1f77b4\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 195.38608 144.623596 \n",
       "L 202.632833 144.623596 \n",
       "L 202.632833 144.233412 \n",
       "L 195.38608 144.233412 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: #1f77b4\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 213.502963 144.623596 \n",
       "L 220.749716 144.623596 \n",
       "L 220.749716 144.489971 \n",
       "L 213.502963 144.489971 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: #1f77b4\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 231.619846 144.623596 \n",
       "L 238.866599 144.623596 \n",
       "L 238.866599 144.548766 \n",
       "L 231.619846 144.548766 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: #1f77b4\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_13\">\n",
       "    <path d=\"M 75.814651 144.623596 \n",
       "L 83.061404 144.623596 \n",
       "L 83.061404 15.195024 \n",
       "L 75.814651 15.195024 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: url(#h5be171eedd)\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_14\">\n",
       "    <path d=\"M 93.931534 144.623596 \n",
       "L 101.178287 144.623596 \n",
       "L 101.178287 59.579697 \n",
       "L 93.931534 59.579697 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: url(#h5be171eedd)\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_15\">\n",
       "    <path d=\"M 112.048417 144.623596 \n",
       "L 119.29517 144.623596 \n",
       "L 119.29517 134.831601 \n",
       "L 112.048417 134.831601 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: url(#h5be171eedd)\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_16\">\n",
       "    <path d=\"M 130.1653 144.623596 \n",
       "L 137.412054 144.623596 \n",
       "L 137.412054 144.078408 \n",
       "L 130.1653 144.078408 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: url(#h5be171eedd)\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_17\">\n",
       "    <path d=\"M 148.282183 144.623596 \n",
       "L 155.528937 144.623596 \n",
       "L 155.528937 144.575491 \n",
       "L 148.282183 144.575491 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: url(#h5be171eedd)\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_18\">\n",
       "    <path d=\"M 166.399067 144.623596 \n",
       "L 173.64582 144.623596 \n",
       "L 173.64582 144.623596 \n",
       "L 166.399067 144.623596 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: url(#h5be171eedd)\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_19\">\n",
       "    <path d=\"M 184.51595 144.623596 \n",
       "L 191.762703 144.623596 \n",
       "L 191.762703 144.623596 \n",
       "L 184.51595 144.623596 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: url(#h5be171eedd)\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_20\">\n",
       "    <path d=\"M 202.632833 144.623596 \n",
       "L 209.879586 144.623596 \n",
       "L 209.879586 144.623596 \n",
       "L 202.632833 144.623596 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: url(#h5be171eedd)\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_21\">\n",
       "    <path d=\"M 220.749716 144.623596 \n",
       "L 227.996469 144.623596 \n",
       "L 227.996469 144.623596 \n",
       "L 220.749716 144.623596 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: url(#h5be171eedd)\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_22\">\n",
       "    <path d=\"M 238.866599 144.623596 \n",
       "L 246.113352 144.623596 \n",
       "L 246.113352 144.623596 \n",
       "L 238.866599 144.623596 \n",
       "z\n",
       "\" clip-path=\"url(#p75958a2569)\" style=\"fill: url(#h5be171eedd)\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"m9a874bdfd9\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m9a874bdfd9\" x=\"66.756209\" y=\"144.623596\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(63.574959 159.222033)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m9a874bdfd9\" x=\"110.943729\" y=\"144.623596\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(104.581229 159.222033)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m9a874bdfd9\" x=\"155.131249\" y=\"144.623596\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(148.768749 159.222033)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m9a874bdfd9\" x=\"199.318769\" y=\"144.623596\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 60 -->\n",
       "      <g transform=\"translate(192.956269 159.222033)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m9a874bdfd9\" x=\"243.506289\" y=\"144.623596\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 80 -->\n",
       "      <g transform=\"translate(237.143789 159.222033)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- # tokens per sentence -->\n",
       "     <g transform=\"translate(100.6125 172.900158)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-23\" d=\"M 3272 2816 \n",
       "L 2363 2816 \n",
       "L 2100 1772 \n",
       "L 3016 1772 \n",
       "L 3272 2816 \n",
       "z\n",
       "M 2803 4594 \n",
       "L 2478 3297 \n",
       "L 3391 3297 \n",
       "L 3719 4594 \n",
       "L 4219 4594 \n",
       "L 3897 3297 \n",
       "L 4872 3297 \n",
       "L 4872 2816 \n",
       "L 3775 2816 \n",
       "L 3519 1772 \n",
       "L 4513 1772 \n",
       "L 4513 1294 \n",
       "L 3397 1294 \n",
       "L 3072 0 \n",
       "L 2572 0 \n",
       "L 2894 1294 \n",
       "L 1978 1294 \n",
       "L 1656 0 \n",
       "L 1153 0 \n",
       "L 1478 1294 \n",
       "L 494 1294 \n",
       "L 494 1772 \n",
       "L 1594 1772 \n",
       "L 1856 2816 \n",
       "L 850 2816 \n",
       "L 850 3297 \n",
       "L 1978 3297 \n",
       "L 2297 4594 \n",
       "L 2803 4594 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6b\" d=\"M 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 1991 \n",
       "L 2875 3500 \n",
       "L 3609 3500 \n",
       "L 1753 1863 \n",
       "L 3688 0 \n",
       "L 2938 0 \n",
       "L 1159 1709 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-23\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"83.789062\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"115.576172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"154.785156\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6b\" x=\"215.966797\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"270.251953\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"331.775391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"395.154297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"447.253906\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"479.041016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"542.517578\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"604.041016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"645.154297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"676.941406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"729.041016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"790.564453\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"853.943359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"893.152344\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"954.675781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"1018.054688\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"1073.035156\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <defs>\n",
       "       <path id=\"md8881e9f0a\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#md8881e9f0a\" x=\"59.690625\" y=\"144.623596\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(46.328125 148.422814)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md8881e9f0a\" x=\"59.690625\" y=\"117.89872\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 5000 -->\n",
       "      <g transform=\"translate(27.240625 121.697939)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md8881e9f0a\" x=\"59.690625\" y=\"91.173845\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 10000 -->\n",
       "      <g transform=\"translate(20.878125 94.973064)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"254.492188\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md8881e9f0a\" x=\"59.690625\" y=\"64.448969\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 15000 -->\n",
       "      <g transform=\"translate(20.878125 68.248188)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"254.492188\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md8881e9f0a\" x=\"59.690625\" y=\"37.724094\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 20000 -->\n",
       "      <g transform=\"translate(20.878125 41.523313)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"254.492188\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md8881e9f0a\" x=\"59.690625\" y=\"10.999219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 25000 -->\n",
       "      <g transform=\"translate(20.878125 14.798437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"254.492188\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- count -->\n",
       "     <g transform=\"translate(14.798438 90.779846)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-75\" d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"54.980469\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-75\" x=\"116.162109\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"179.541016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"242.919922\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_23\">\n",
       "    <path d=\"M 59.690625 144.623596 \n",
       "L 59.690625 8.723596 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_24\">\n",
       "    <path d=\"M 254.990625 144.623596 \n",
       "L 254.990625 8.723596 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_25\">\n",
       "    <path d=\"M 59.690625 144.623596 \n",
       "L 254.990625 144.623596 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_26\">\n",
       "    <path d=\"M 59.690625 8.723596 \n",
       "L 254.990625 8.723596 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_27\">\n",
       "     <path d=\"M 155.389063 46.079846 \n",
       "L 247.990625 46.079846 \n",
       "Q 249.990625 46.079846 249.990625 44.079846 \n",
       "L 249.990625 15.723596 \n",
       "Q 249.990625 13.723596 247.990625 13.723596 \n",
       "L 155.389063 13.723596 \n",
       "Q 153.389063 13.723596 153.389063 15.723596 \n",
       "L 153.389063 44.079846 \n",
       "Q 153.389063 46.079846 155.389063 46.079846 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"patch_28\">\n",
       "     <path d=\"M 157.389063 25.322033 \n",
       "L 177.389063 25.322033 \n",
       "L 177.389063 18.322033 \n",
       "L 157.389063 18.322033 \n",
       "z\n",
       "\" style=\"fill: #1f77b4\"/>\n",
       "    </g>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- origin -->\n",
       "     <g transform=\"translate(185.389063 25.322033)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
       "Q 2906 2416 2648 2759 \n",
       "Q 2391 3103 1925 3103 \n",
       "Q 1463 3103 1205 2759 \n",
       "Q 947 2416 947 1791 \n",
       "Q 947 1169 1205 825 \n",
       "Q 1463 481 1925 481 \n",
       "Q 2391 481 2648 825 \n",
       "Q 2906 1169 2906 1791 \n",
       "z\n",
       "M 3481 434 \n",
       "Q 3481 -459 3084 -895 \n",
       "Q 2688 -1331 1869 -1331 \n",
       "Q 1566 -1331 1297 -1286 \n",
       "Q 1028 -1241 775 -1147 \n",
       "L 775 -588 \n",
       "Q 1028 -725 1275 -790 \n",
       "Q 1522 -856 1778 -856 \n",
       "Q 2344 -856 2625 -561 \n",
       "Q 2906 -266 2906 331 \n",
       "L 2906 616 \n",
       "Q 2728 306 2450 153 \n",
       "Q 2172 0 1784 0 \n",
       "Q 1141 0 747 490 \n",
       "Q 353 981 353 1791 \n",
       "Q 353 2603 747 3093 \n",
       "Q 1141 3584 1784 3584 \n",
       "Q 2172 3584 2450 3431 \n",
       "Q 2728 3278 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 434 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"61.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"102.294922\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-67\" x=\"130.078125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"193.554688\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"221.337891\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"patch_29\">\n",
       "     <path d=\"M 157.389063 40.000158 \n",
       "L 177.389063 40.000158 \n",
       "L 177.389063 33.000158 \n",
       "L 157.389063 33.000158 \n",
       "z\n",
       "\" style=\"fill: url(#h5be171eedd)\"/>\n",
       "    </g>\n",
       "    <g id=\"text_15\">\n",
       "     <!-- subsampled -->\n",
       "     <g transform=\"translate(185.389063 40.000158)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-62\" d=\"M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "M 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2969 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
       "L 2906 4863 \n",
       "L 3481 4863 \n",
       "L 3481 0 \n",
       "L 2906 0 \n",
       "L 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "z\n",
       "M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-73\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-75\" x=\"52.099609\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-62\" x=\"115.478516\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"178.955078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"231.054688\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"292.333984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"389.746094\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"453.222656\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"481.005859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-64\" x=\"542.529297\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p75958a2569\">\n",
       "   <rect x=\"59.690625\" y=\"8.723596\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       " <defs>\n",
       "  <pattern id=\"h5be171eedd\" patternUnits=\"userSpaceOnUse\" x=\"0\" y=\"0\" width=\"72\" height=\"72\">\n",
       "   <rect x=\"0\" y=\"0\" width=\"73\" height=\"73\" fill=\"#ff7f0e\"/>\n",
       "   <path d=\"M -36 36 \n",
       "L 36 -36 \n",
       "M -24 48 \n",
       "L 48 -24 \n",
       "M -12 60 \n",
       "L 60 -12 \n",
       "M 0 72 \n",
       "L 72 0 \n",
       "M 12 84 \n",
       "L 84 12 \n",
       "M 24 96 \n",
       "L 96 24 \n",
       "M 36 108 \n",
       "L 108 36 \n",
       "\" style=\"fill: #000000; stroke: #000000; stroke-width: 1.0; stroke-linecap: butt; stroke-linejoin: miter\"/>\n",
       "  </pattern>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "d2l.show_list_len_pair_hist(['origin', 'subsampled'], '# tokens per sentence',\n",
    "                            'count', sentences, subsampled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "bfc2eb33",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'# of \"the\": before=50770, after=2052'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def compare_counts(token):\n",
    "    return (f'# of \"{token}\": '\n",
    "            f'before={sum([l.count(token) for l in sentences])}, '\n",
    "            f'after={sum([l.count(token) for l in subsampled])}')\n",
    "\n",
    "compare_counts('the')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4d8b9a86",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'# of \"join\": before=45, after=45'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "compare_counts('join')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c94a3858",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[], [3228, 710, 4060], [3922, 1922, 4743]]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus = [vocab[line] for line in subsampled]\n",
    "corpus[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0c58948",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Extracting Center Words and Context Words\n",
    "\n",
    "- It uniformly samples an integer between 1 and `max_window_size` at random as the context window size.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f687a9d7",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "def get_centers_and_contexts(corpus, max_window_size):\n",
    "    \"\"\"Return center words and context words in skip-gram.\"\"\"\n",
    "    centers, contexts = [], []\n",
    "    for line in corpus:\n",
    "        # To form a \"center word--context word\" pair, each sentence needs to\n",
    "        # have at least 2 words\n",
    "        if len(line) < 2:\n",
    "            continue\n",
    "        centers += line\n",
    "        for i in range(len(line)):  # Context window centered at `i`\n",
    "            window_size = random.randint(1, max_window_size)\n",
    "            indices = list(range(max(0, i - window_size),\n",
    "                                 min(len(line), i + 1 + window_size)))\n",
    "            # Exclude the center word from the context words\n",
    "            indices.remove(i)\n",
    "            contexts.append([line[idx] for idx in indices])\n",
    "    return centers, contexts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "975c8743",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dataset [[0, 1, 2, 3, 4, 5, 6], [7, 8, 9]]\n",
      "center 0 has contexts [1]\n",
      "center 1 has contexts [0, 2, 3]\n",
      "center 2 has contexts [1, 3]\n",
      "center 3 has contexts [2, 4]\n",
      "center 4 has contexts [2, 3, 5, 6]\n",
      "center 5 has contexts [3, 4, 6]\n",
      "center 6 has contexts [4, 5]\n",
      "center 7 has contexts [8, 9]\n",
      "center 8 has contexts [7, 9]\n",
      "center 9 has contexts [8]\n"
     ]
    }
   ],
   "source": [
    "tiny_dataset = [list(range(7)), list(range(7, 10))]\n",
    "print('dataset', tiny_dataset)\n",
    "for center, context in zip(*get_centers_and_contexts(tiny_dataset, 2)):\n",
    "    print('center', center, 'has contexts', context)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9d6d341",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- When training on the PTB dataset, we set the maximum context window size to 5. The following extracts all the center words and their context words in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "0c56645f",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'# center-context pairs: 1502675'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_centers, all_contexts = get_centers_and_contexts(corpus, 5)\n",
    "f'# center-context pairs: {sum([len(contexts) for contexts in all_contexts])}'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54730ce1",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Negative Sampling\n",
    "\n",
    "- We use negative sampling for approximate training. To sample noise words according to a predefined distribution, we define the following `RandomGenerator` class, where the (possibly unnormalized) sampling distribution is passed via the argument `sampling_weights`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "59720742",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "class RandomGenerator:\n",
    "    \"\"\"Randomly draw among {1, ..., n} according to n sampling weights.\"\"\"\n",
    "    def __init__(self, sampling_weights):\n",
    "        # Exclude\n",
    "        self.population = list(range(1, len(sampling_weights) + 1))\n",
    "        self.sampling_weights = sampling_weights\n",
    "        self.candidates = []\n",
    "        self.i = 0\n",
    "\n",
    "    def draw(self):\n",
    "        if self.i == len(self.candidates):\n",
    "            # Cache `k` random sampling results\n",
    "            self.candidates = random.choices(\n",
    "                self.population, self.sampling_weights, k=10000)\n",
    "            self.i = 0\n",
    "        self.i += 1\n",
    "        return self.candidates[self.i - 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "57ed281f",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2, 1, 1, 2, 3, 3, 3, 3, 2]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generator = RandomGenerator([2, 3, 4])\n",
    "[generator.draw() for _ in range(10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3fba01f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- For a pair of center word and context word, we randomly sample `K` (5 in the experiment) noise words. According to the suggestions in the word2vec paper, the sampling probability $P(w)$ of a noise word $w$ is set to its relative frequency in the dictionary raised to the power of 0.75."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "776d0428",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "def get_negatives(all_contexts, vocab, counter, K):\n",
    "    \"\"\"Return noise words in negative sampling.\"\"\"\n",
    "    # Sampling weights for words with indices 1, 2, ... (index 0 is the\n",
    "    # excluded unknown token) in the vocabulary\n",
    "    sampling_weights = [counter[vocab.to_tokens(i)]**0.75\n",
    "                        for i in range(1, len(vocab))]\n",
    "    all_negatives, generator = [], RandomGenerator(sampling_weights)\n",
    "    for contexts in all_contexts:\n",
    "        negatives = []\n",
    "        while len(negatives) < len(contexts) * K:\n",
    "            neg = generator.draw()\n",
    "            # Noise words cannot be context words\n",
    "            if neg not in contexts:\n",
    "                negatives.append(neg)\n",
    "        all_negatives.append(negatives)\n",
    "    return all_negatives\n",
    "\n",
    "all_negatives = get_negatives(all_contexts, vocab, counter, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dd88d91",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Loading Training Examples in Minibatches\n",
    "\n",
    "- In a minibatch, the $i^\\mathrm{th}$ example includes a center word and its $n_i$ context words and $m_i$ noise words.  Due to varying context window sizes, $n_i+m_i$ varies for different $i$. Thus, for each example we concatenate its context words and noise words in  the `contexts_negatives` variable, and pad zeros until the concatenation length reaches $\\max_i n_i+m_i$ (`max_len`). To exclude paddings in the calculation of the loss, we define a mask variable `masks`. There is a one-to-one correspondence between elements in `masks` and elements in `contexts_negatives`, where zeros (otherwise ones) in `masks` correspond to paddings in `contexts_negatives`.\n",
    "\n",
    "- To distinguish between positive and negative examples, we separate context words from noise words in  `contexts_negatives` via a `labels` variable, similar to `masks`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "fb9a2a7a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "def batchify(data):\n",
    "    \"\"\"Return a minibatch of examples for skip-gram with negative sampling.\"\"\"\n",
    "    max_len = max(len(c) + len(n) for _, c, n in data)\n",
    "    centers, contexts_negatives, masks, labels = [], [], [], []\n",
    "    for center, context, negative in data:\n",
    "        cur_len = len(context) + len(negative)\n",
    "        centers += [center]\n",
    "        contexts_negatives += [context + negative + [0] * (max_len - cur_len)]\n",
    "        masks += [[1] * cur_len + [0] * (max_len - cur_len)]\n",
    "        labels += [[1] * len(context) + [0] * (max_len - len(context))]\n",
    "    return (torch.tensor(centers).reshape((-1, 1)), torch.tensor(\n",
    "        contexts_negatives), torch.tensor(masks), torch.tensor(labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8e3ce6ef",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "centers = tensor([[1],\n",
      "        [1]])\n",
      "contexts_negatives = tensor([[2, 2, 3, 3, 3, 3],\n",
      "        [2, 2, 2, 3, 3, 0]])\n",
      "masks = tensor([[1, 1, 1, 1, 1, 1],\n",
      "        [1, 1, 1, 1, 1, 0]])\n",
      "labels = tensor([[1, 1, 0, 0, 0, 0],\n",
      "        [1, 1, 1, 0, 0, 0]])\n"
     ]
    }
   ],
   "source": [
    "x_1 = (1, [2, 2], [3, 3, 3, 3])\n",
    "x_2 = (1, [2, 2, 2], [3, 3])\n",
    "batch = batchify((x_1, x_2))\n",
    "\n",
    "names = ['centers', 'contexts_negatives', 'masks', 'labels']\n",
    "for name, data in zip(names, batch):\n",
    "    print(name, '=', data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8597ffbb",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Putting It All Together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9d184a7f",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "def load_data_ptb(batch_size, max_window_size, num_noise_words):\n",
    "    \"\"\"Download the PTB dataset and then load it into memory.\"\"\"\n",
    "    num_workers = d2l.get_dataloader_workers()\n",
    "    sentences = read_ptb()\n",
    "    vocab = d2l.Vocab(sentences, min_freq=10)\n",
    "    subsampled, counter = subsample(sentences, vocab)\n",
    "    corpus = [vocab[line] for line in subsampled]\n",
    "    all_centers, all_contexts = get_centers_and_contexts(\n",
    "        corpus, max_window_size)\n",
    "    all_negatives = get_negatives(\n",
    "        all_contexts, vocab, counter, num_noise_words)\n",
    "\n",
    "    class PTBDataset(torch.utils.data.Dataset):\n",
    "        def __init__(self, centers, contexts, negatives):\n",
    "            assert len(centers) == len(contexts) == len(negatives)\n",
    "            self.centers = centers\n",
    "            self.contexts = contexts\n",
    "            self.negatives = negatives\n",
    "\n",
    "        def __getitem__(self, index):\n",
    "            return (self.centers[index], self.contexts[index],\n",
    "                    self.negatives[index])\n",
    "\n",
    "        def __len__(self):\n",
    "            return len(self.centers)\n",
    "\n",
    "    dataset = PTBDataset(all_centers, all_contexts, all_negatives)\n",
    "\n",
    "    data_iter = torch.utils.data.DataLoader(dataset, batch_size, shuffle=True,\n",
    "                                      collate_fn=batchify,\n",
    "                                      num_workers=num_workers)\n",
    "    return data_iter, vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7cbaf6fe",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "centers shape: torch.Size([512, 1])\n",
      "contexts_negatives shape: torch.Size([512, 60])\n",
      "masks shape: torch.Size([512, 60])\n",
      "labels shape: torch.Size([512, 60])\n"
     ]
    }
   ],
   "source": [
    "data_iter, vocab = load_data_ptb(512, 5, 5)\n",
    "for batch in data_iter:\n",
    "    for name, data in zip(names, batch):\n",
    "        print(name, 'shape:', data.shape)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc0d5d7c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Pretraining word2vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "beb6a168",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "batch_size, max_window_size, num_noise_words = 512, 5, 5\n",
    "data_iter, vocab = d2l.load_data_ptb(batch_size, max_window_size, num_noise_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04a64e4c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The Skip-Gram Model\n",
    "\n",
    "- We implement the skip-gram model by using embedding layers and batch matrix multiplications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "814d4b6a",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameter embedding_weight (torch.Size([20, 4]), dtype=torch.float32)\n"
     ]
    }
   ],
   "source": [
    "embed = nn.Embedding(num_embeddings=20, embedding_dim=4)\n",
    "print(f'Parameter embedding_weight ({embed.weight.shape}, '\n",
    "      f'dtype={embed.weight.dtype})')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "396289eb",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- The input of an embedding layer is the index of a token (word).\n",
    "- For any token index $i$, its vector representation can be obtained from the $i^\\mathrm{th}$ row of the weight matrix in the embedding layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "bcfdc60d",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[-1.2976, -0.1158, -0.3598, -0.6239],\n",
       "         [-0.6515, -1.2558,  0.9660,  0.7976],\n",
       "         [ 0.8370,  1.6535, -0.7562, -0.0753]],\n",
       "\n",
       "        [[ 0.8370,  1.6535, -0.7562, -0.0753],\n",
       "         [-1.1562, -1.2339,  1.1942,  0.2774],\n",
       "         [-0.1997,  0.9390,  1.0504,  0.5765]]], grad_fn=<EmbeddingBackward0>)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.tensor([[1, 2, 3], [3, 4, 5]])\n",
    "embed(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ec4a6f4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Defining the Forward Propagation\n",
    "\n",
    "- In the forward propagation, the input of the skip-gram model includes the center word indices `center` of shape (batch size, 1) and the concatenated context and noise word indices `contexts_and_negatives` of shape (batch size, `max_len`).\n",
    "- These two variables are first transformed from the token indices into vectors via the embedding layer, then their batch matrix multiplication returns an output of shape (batch size, 1, `max_len`).\n",
    "- Each element in the output is the dot product of a center word vector and a context or noise word vector.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "580a20e9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Another commonly used operation is to multiply batches of matrices with another. This comes in handy when we have minibatches of queries, keys, and values. More specifically, assume that \n",
    "\n",
    "$$\\mathbf{Q} = [\\mathbf{Q}_1, \\mathbf{Q}_2, \\ldots, \\mathbf{Q}_n]  \\in \\mathbb{R}^{n \\times a \\times b} \\\\\n",
    "    \\mathbf{K} = [\\mathbf{K}_1, \\mathbf{K}_2, \\ldots, \\mathbf{K}_n]  \\in \\mathbb{R}^{n \\times b \\times c} \n",
    "$$\n",
    "\n",
    "- Then the batch matrix multiplication (BMM) computes the element-wise product\n",
    "\n",
    "$$\\mathrm{BMM}(\\mathbf{Q}, \\mathbf{K}) = [\\mathbf{Q}_1 \\mathbf{K}_1, \\mathbf{Q}_2 \\mathbf{K}_2, \\ldots, \\mathbf{Q}_n \\mathbf{K}_n] \\in \\mathbb{R}^{n \\times a \\times c}.$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4e2e9899",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def skip_gram(center, contexts_and_negatives, embed_v, embed_u):\n",
    "    v = embed_v(center)\n",
    "    u = embed_u(contexts_and_negatives)\n",
    "    pred = torch.bmm(v, u.permute(0, 2, 1))\n",
    "    return pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "cafd73c2",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 1, 4])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skip_gram(torch.ones((2, 1), dtype=torch.long),\n",
    "          torch.ones((2, 4), dtype=torch.long), embed, embed).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81379ee6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Training\n",
    "\n",
    "#### Binary Cross-Entropy Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f0e7d013",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "class SigmoidBCELoss(nn.Module):\n",
    "    # Binary cross-entropy loss with masking\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "\n",
    "    def forward(self, inputs, target, mask=None):\n",
    "        out = nn.functional.binary_cross_entropy_with_logits(\n",
    "            inputs, target, weight=mask, reduction=\"none\")\n",
    "        return out.mean(dim=1)\n",
    "\n",
    "loss = SigmoidBCELoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "b445e2f2",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.9352, 1.8462])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred = torch.tensor([[1.1, -2.2, 3.3, -4.4]] * 2)\n",
    "label = torch.tensor([[1.0, 0.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0]])\n",
    "mask = torch.tensor([[1, 1, 1, 1], [1, 1, 0, 0]])\n",
    "loss(pred, label, mask) * mask.shape[1] / mask.sum(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "a512d4f5",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9352\n",
      "1.8462\n"
     ]
    }
   ],
   "source": [
    "def sigmd(x):\n",
    "    return -math.log(1 / (1 + math.exp(-x)))\n",
    "\n",
    "print(f'{(sigmd(1.1) + sigmd(2.2) + sigmd(-3.3) + sigmd(4.4)) / 4:.4f}')\n",
    "print(f'{(sigmd(-1.1) + sigmd(-2.2)) / 2:.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cde28f06",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Initializing Model Parameters\n",
    "\n",
    "- We define two embedding layers for all the words in the vocabulary when they are used as center words and context words, respectively. The word vector dimension `embed_size` is set to 100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "f4d25ed3",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "embed_size = 100\n",
    "net = nn.Sequential(nn.Embedding(num_embeddings=len(vocab), embedding_dim=embed_size),\n",
    "                    nn.Embedding(num_embeddings=len(vocab), embedding_dim=embed_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "618bb9e6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### The Training Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "19d9c5c2",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def train(net, data_iter, lr, num_epochs, device=d2l.try_gpu()):\n",
    "    def init_weights(module):\n",
    "        if type(module) == nn.Embedding:\n",
    "            nn.init.xavier_uniform_(module.weight)\n",
    "    net.apply(init_weights)\n",
    "    net = net.to(device)\n",
    "    optimizer = torch.optim.Adam(net.parameters(), lr=lr)\n",
    "    animator = d2l.Animator(xlabel='epoch', ylabel='loss',\n",
    "                            xlim=[1, num_epochs])\n",
    "    # Sum of normalized losses, no. of normalized losses\n",
    "    metric = d2l.Accumulator(2)\n",
    "    for epoch in range(num_epochs):\n",
    "        timer, num_batches = d2l.Timer(), len(data_iter)\n",
    "        for i, batch in enumerate(data_iter):\n",
    "            optimizer.zero_grad()\n",
    "            center, context_negative, mask, label = [\n",
    "                data.to(device) for data in batch]\n",
    "\n",
    "            pred = skip_gram(center, context_negative, net[0], net[1])\n",
    "            l = (loss(pred.reshape(label.shape).float(), label.float(), mask)\n",
    "                     / mask.sum(axis=1) * mask.shape[1])\n",
    "            l.sum().backward()\n",
    "            optimizer.step()\n",
    "            metric.add(l.sum(), l.numel())\n",
    "            if (i + 1) % (num_batches // 5) == 0 or i == num_batches - 1:\n",
    "                animator.add(epoch + (i + 1) / num_batches,\n",
    "                             (metric[0] / metric[1],))\n",
    "    print(f'loss {metric[0] / metric[1]:.3f}, '\n",
    "          f'{metric[1] / timer.stop():.1f} tokens/sec on {str(device)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "0908bfbb",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.366, 1616602.2 tokens/sec on cuda:0\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"259.00625pt\" height=\"180.65625pt\" viewBox=\"0 0 259.00625 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-03-21T14:36:32.186471</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 259.00625 180.65625 \n",
       "L 259.00625 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 50.14375 143.1 \n",
       "L 245.44375 143.1 \n",
       "L 245.44375 7.2 \n",
       "L 50.14375 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 71.84375 143.1 \n",
       "L 71.84375 7.2 \n",
       "\" clip-path=\"url(#pf75a5ab416)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"mf372e23e18\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf372e23e18\" x=\"71.84375\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(68.6625 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 115.24375 143.1 \n",
       "L 115.24375 7.2 \n",
       "\" clip-path=\"url(#pf75a5ab416)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf372e23e18\" x=\"115.24375\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(112.0625 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 158.64375 143.1 \n",
       "L 158.64375 7.2 \n",
       "\" clip-path=\"url(#pf75a5ab416)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf372e23e18\" x=\"158.64375\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 6 -->\n",
       "      <g transform=\"translate(155.4625 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 202.04375 143.1 \n",
       "L 202.04375 7.2 \n",
       "\" clip-path=\"url(#pf75a5ab416)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf372e23e18\" x=\"202.04375\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 8 -->\n",
       "      <g transform=\"translate(198.8625 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 245.44375 143.1 \n",
       "L 245.44375 7.2 \n",
       "\" clip-path=\"url(#pf75a5ab416)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf372e23e18\" x=\"245.44375\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(239.08125 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(132.565625 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 50.14375 120.191128 \n",
       "L 245.44375 120.191128 \n",
       "\" clip-path=\"url(#pf75a5ab416)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m73c9bb4795\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m73c9bb4795\" x=\"50.14375\" y=\"120.191128\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0.40 -->\n",
       "      <g transform=\"translate(20.878125 123.990347)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 50.14375 95.457902 \n",
       "L 245.44375 95.457902 \n",
       "\" clip-path=\"url(#pf75a5ab416)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m73c9bb4795\" x=\"50.14375\" y=\"95.457902\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.45 -->\n",
       "      <g transform=\"translate(20.878125 99.257121)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 50.14375 70.724677 \n",
       "L 245.44375 70.724677 \n",
       "\" clip-path=\"url(#pf75a5ab416)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m73c9bb4795\" x=\"50.14375\" y=\"70.724677\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.50 -->\n",
       "      <g transform=\"translate(20.878125 74.523895)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 50.14375 45.991451 \n",
       "L 245.44375 45.991451 \n",
       "\" clip-path=\"url(#pf75a5ab416)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m73c9bb4795\" x=\"50.14375\" y=\"45.991451\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.55 -->\n",
       "      <g transform=\"translate(20.878125 49.790669)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 50.14375 21.258225 \n",
       "L 245.44375 21.258225 \n",
       "\" clip-path=\"url(#pf75a5ab416)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m73c9bb4795\" x=\"50.14375\" y=\"21.258225\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.60 -->\n",
       "      <g transform=\"translate(20.878125 25.057444)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_12\">\n",
       "     <!-- loss -->\n",
       "     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_21\">\n",
       "    <path d=\"M 32.78375 13.377273 \n",
       "L 37.12375 51.38398 \n",
       "L 41.46375 66.749085 \n",
       "L 45.80375 75.04142 \n",
       "L 50.14375 80.31862 \n",
       "L 54.48375 84.294626 \n",
       "L 58.82375 87.351821 \n",
       "L 63.16375 89.840231 \n",
       "L 67.50375 91.975926 \n",
       "L 71.84375 93.825562 \n",
       "L 76.18375 95.819883 \n",
       "L 80.52375 97.606428 \n",
       "L 84.86375 99.244788 \n",
       "L 89.20375 100.728222 \n",
       "L 93.54375 102.069961 \n",
       "L 97.88375 103.783922 \n",
       "L 102.22375 105.308713 \n",
       "L 106.56375 106.691538 \n",
       "L 110.90375 107.945941 \n",
       "L 115.24375 109.118982 \n",
       "L 119.58375 110.63279 \n",
       "L 123.92375 111.993874 \n",
       "L 128.26375 113.230077 \n",
       "L 132.60375 114.353976 \n",
       "L 136.94375 115.379169 \n",
       "L 141.28375 116.722556 \n",
       "L 145.62375 117.922698 \n",
       "L 149.96375 119.021786 \n",
       "L 154.30375 120.004701 \n",
       "L 158.64375 120.898199 \n",
       "L 162.98375 122.066053 \n",
       "L 167.32375 123.115937 \n",
       "L 171.66375 124.076528 \n",
       "L 176.00375 124.943689 \n",
       "L 180.34375 125.728207 \n",
       "L 184.68375 126.744857 \n",
       "L 189.02375 127.662616 \n",
       "L 193.36375 128.502902 \n",
       "L 197.70375 129.258257 \n",
       "L 202.04375 129.950018 \n",
       "L 206.38375 130.842426 \n",
       "L 210.72375 131.64816 \n",
       "L 215.06375 132.389543 \n",
       "L 219.40375 133.056939 \n",
       "L 223.74375 133.654087 \n",
       "L 228.08375 134.437557 \n",
       "L 232.42375 135.152149 \n",
       "L 236.76375 135.80101 \n",
       "L 241.10375 136.383695 \n",
       "L 245.44375 136.922727 \n",
       "\" clip-path=\"url(#pf75a5ab416)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 50.14375 143.1 \n",
       "L 50.14375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 245.44375 143.1 \n",
       "L 245.44375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 50.14375 143.1 \n",
       "L 245.44375 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 50.14375 7.2 \n",
       "L 245.44375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pf75a5ab416\">\n",
       "   <rect x=\"50.14375\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr, num_epochs = 0.002, 10\n",
    "train(net, data_iter, lr, num_epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce60820f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Applying Word Embeddings\n",
    "\n",
    "- After training the word2vec model, we can use the cosine similarity of word vectors from the trained model to find words from the dictionary that are most semantically similar to an input word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "85e79001",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cosine sim=0.666: intel\n",
      "cosine sim=0.655: microprocessor\n",
      "cosine sim=0.579: chips\n"
     ]
    }
   ],
   "source": [
    "def get_similar_tokens(query_token, k, embed):\n",
    "    W = embed.weight.data\n",
    "    x = W[vocab[query_token]]\n",
    "    # Compute the cosine similarity. Add 1e-9 for numerical stability\n",
    "    cos = torch.mv(W, x) / torch.sqrt(torch.sum(W * W, dim=1) *\n",
    "                                      torch.sum(x * x) + 1e-9)\n",
    "    topk = torch.topk(cos, k=k+1)[1].cpu().numpy().astype('int32')\n",
    "    for i in topk[1:]:  # Remove the input words\n",
    "        print(f'cosine sim={float(cos[i]):.3f}: {vocab.to_tokens(i)}')\n",
    "\n",
    "get_similar_tokens('chip', 3, net[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a90e270d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Word Embedding with Global Vectors (GloVe)\n",
    "\n",
    "- Word-word co-occurrences within context windows may carry rich semantic information. For example, in a large corpus word \"solid\" is more likely to co-occur with \"ice\" than \"steam\", but word \"gas\" probably co-occurs with \"steam\" more frequently than \"ice\". \n",
    "- Besides, **global corpus statistics of such co-occurrences can be precomputed: this can lead to more efficient training**. To leverage statistical information in the entire corpus for word embedding, let's first revisit the skip-gram model, but interpreting it using global corpus statistics such as co-occurrence counts.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd2fd64e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Skip-Gram with Global Corpus Statistics\n",
    "\n",
    "- Denoting by $q_{ij}$ the conditional probability $P(w_j\\mid w_i)$ of word $w_j$ given word $w_i$ in the skip-gram model, we have\n",
    "\n",
    "$$q_{ij}=\\frac{\\exp(\\mathbf{u}_j^\\top \\mathbf{v}_i)}{ \\sum_{k \\in \\mathcal{V}} \\text{exp}(\\mathbf{u}_k^\\top \\mathbf{v}_i)},$$\n",
    "\n",
    "- where for any index $i$ vectors $\\mathbf{v}_i$ and $\\mathbf{u}_i$ represent word $w_i$ as the center word and context word, respectively, and $\\mathcal{V} = \\{0, 1, \\ldots, |\\mathcal{V}|-1\\}$ is the index set of the vocabulary.\n",
    "- Consider word $w_i$ that may occur multiple times in the corpus. In the entire corpus, all the context words wherever $w_i$ is taken as their center word form a *multiset* $\\mathcal{C}_i$ of word indices that *allows for multiple instances of the same element*. For any element, its number of instances is called its *multiplicity*.\n",
    "- To illustrate with an example, suppose that word $w_i$ occurs twice in the corpus and indices of the context words that take $w_i$ as their center word in the two context windows are $k, j, m, k$ and $k, l, k, j$. Thus, multiset $\\mathcal{C}_i = \\{j, j, k, k, k, k, l, m\\}$, where multiplicities of elements $j, k, l, m$ are 2, 4, 1, 1, respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "544925de",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Now let's denote the multiplicity of element $j$ in multiset $\\mathcal{C}_i$ as $x_{ij}$. This is the global co-occurrence count of word $w_j$ (as the context word) and word $w_i$ (as the center word) in the same context window in the entire corpus. \n",
    "- Using such global corpus statistics, the loss function of the skip-gram model is equivalent to\n",
    "\n",
    "$$-\\sum_{i\\in\\mathcal{V}}\\sum_{j\\in\\mathcal{V}} x_{ij} \\log\\,q_{ij}.$$\n",
    "\n",
    "- We further denote by $x_i$ the number of all the context words in the context windows where $w_i$ occurs as their center word, which is equivalent to $|\\mathcal{C}_i|$. Letting $p_{ij}$ be the conditional probability $x_{ij}/x_i$ for generating context word $w_j$ given center word $w_i$, the loss function can be rewritten as\n",
    "\n",
    "$$-\\sum_{i\\in\\mathcal{V}} x_i \\sum_{j\\in\\mathcal{V}} p_{ij} \\log\\,q_{ij}.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcf6b2a3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$-\\sum_{i\\in\\mathcal{V}} x_i \\sum_{j\\in\\mathcal{V}} p_{ij} \\log\\,q_{ij}.$$\n",
    "\n",
    "- $-\\sum_{j\\in\\mathcal{V}} p_{ij} \\log\\,q_{ij}$ calculates the cross-entropy of the conditional distribution $p_{ij}$ of global corpus statistics and the conditional distribution $q_{ij}$ of model predictions. This loss is also weighted by $x_i$ as explained above. Minimizing the loss function will allow the predicted conditional distribution to get close to the conditional distribution from the global corpus statistics.\n",
    "- Though being commonly used for measuring the distance between probability distributions, the cross-entropy loss function may not be a good choice here.\n",
    "    - On the one hand, the cost of properly normalizing $q_{ij}$ results in the sum over the entire vocabulary, which can be computationally expensive. \n",
    "    - On the other hand, a large number of rare events from a large corpus are often modeled by the cross-entropy loss to be assigned with too much weight."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bba0af2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The GloVe Model\n",
    "\n",
    "- In view of this, the *GloVe* model makes three changes to the skip-gram model based on squared loss.\n",
    "    - Use variables $p'_{ij}=x_{ij}$ and $q'_{ij}=\\exp(\\mathbf{u}_j^\\top \\mathbf{v}_i)$\n",
    "    that are not probability distributions\n",
    "    and take the logarithm of both, so the squared loss term is $\\left(\\log\\,p'_{ij} - \\log\\,q'_{ij}\\right)^2 = \\left(\\mathbf{u}_j^\\top \\mathbf{v}_i - \\log\\,x_{ij}\\right)^2$.\n",
    "    - Add two scalar model parameters for each word $w_i$: the center word bias $b_i$ and the context word bias $c_i$.\n",
    "    - Replace the weight of each loss term with the weight function $h(x_{ij})$, where $h(x)$ is increasing in the interval of $[0, 1]$.\n",
    "- Putting all things together, training GloVe is to minimize the following loss function:\n",
    "\n",
    "$$\\sum_{i\\in\\mathcal{V}} \\sum_{j\\in\\mathcal{V}} h(x_{ij}) \\left(\\mathbf{u}_j^\\top \\mathbf{v}_i + b_i + c_j - \\log\\,x_{ij}\\right)^2.$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "265fdca9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- For the weight function, a suggested choice is: $h(x) = (x/c) ^\\alpha$ (e.g $\\alpha = 0.75$) if $x < c$ (e.g., $c = 100$); otherwise $h(x) = 1$.\n",
    "- In this case, because $h(0)=0$, the squared loss term for any $x_{ij}=0$ can be omitted for computational efficiency. For example, when using minibatch stochastic gradient descent for training, at each iteration we randomly sample a minibatch of *non-zero* $x_{ij}$ to calculate gradients and update the model parameters.\n",
    "- Note that these non-zero $x_{ij}$ are precomputed global corpus statistics; thus, the model is called GloVe for *Global Vectors*.\n",
    "- It should be emphasized that if word $w_i$ appears in the context window of word $w_j$, then *vice versa*. Therefore, $x_{ij}=x_{ji}$. Unlike word2vec that fits the asymmetric conditional probability $p_{ij}$, GloVe fits the symmetric $\\log \\, x_{ij}$. Therefore, the center word vector and the context word vector of any word are mathematically equivalent in the GloVe model.\n",
    "- However in practice, owing to different initialization values, the same word may still get different values in these two vectors after training: GloVe sums them up as the output vector.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bca2a11d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Interpreting GloVe from the Ratio of Co-occurrence Probabilities\n",
    "\n",
    "\n",
    "- We can also interpret the GloVe model from another perspective.\n",
    "- Let $p_{ij} \\stackrel{\\mathrm{def}}{=} P(w_j \\mid w_i)$ be the conditional probability of generating the context word $w_j$ given $w_i$ as the center word in the corpus.\n",
    "\n",
    "|$$w_k=$$|solid|gas|water|fashion|\n",
    "|:-----------------|:-|:-|:-|:-|\n",
    "|$$p_1=P(w_k\\mid \\text{ice})$$|0.00019|0.000066|0.003|0.000017|\n",
    "|$$p_2=P(w_k\\mid\\text{steam})$$|0.000022|0.00078|0.0022|0.000018|\n",
    "|$$p_1/p_2$$|8.9|0.085|1.36|0.96|\n",
    "\n",
    "- For a word $w_k$ that is related to \"ice\" but unrelated to \"steam\", such as $w_k=\\text{solid}$, we expect a larger ratio of co-occurence probabilities, such as 8.9.\n",
    "- For a word $w_k$ that is related to \"steam\" but unrelated to \"ice\", such as $w_k=\\text{gas}$, we expect a smaller ratio of co-occurence probabilities, such as 0.085.\n",
    "- For a word $w_k$ that is related to both \"ice\" and \"steam\", such as $w_k=\\text{water}$, we expect a ratio of co-occurence probabilities that is close to 1, such as 1.36.\n",
    "- For a word $w_k$ that is unrelated to both \"ice\" and \"steam\", such as $w_k=\\text{fashion}$, we expect a ratio of co-occurence probabilities that is close to 1, such as 0.96.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4201479e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- It can be seen that the ratio of co-occurrence probabilities can intuitively express the relationship between words. Thus, we can design a function of three word vectors to fit this ratio. \n",
    "- For the ratio of co-occurrence probabilities ${p_{ij}}/{p_{ik}}$ with $w_i$ being the center word and $w_j$ and $w_k$ being the context words,\n",
    "we want to fit this ratio using some function $f$:\n",
    "\n",
    "$$f(\\mathbf{u}_j, \\mathbf{u}_k, {\\mathbf{v}}_i) \\approx \\frac{p_{ij}}{p_{ik}}.$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aeb11eab",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Among many possible designs for $f$, we only pick a reasonable choice in the following. Since the ratio of co-occurrence probabilities is a scalar, e require that $f$ be a scalar function, such as $f(\\mathbf{u}_j, \\mathbf{u}_k, {\\mathbf{v}}_i) = f\\left((\\mathbf{u}_j - \\mathbf{u}_k)^\\top {\\mathbf{v}}_i\\right)$.\n",
    "- Switching word indices $j$ and $k$, it must hold that $f(x)f(-x)=1$, so one possibility is $f(x)=\\exp(x)$, i.e.,\n",
    "\n",
    "$$f(\\mathbf{u}_j, \\mathbf{u}_k, {\\mathbf{v}}_i) = \\frac{\\exp\\left(\\mathbf{u}_j^\\top {\\mathbf{v}}_i\\right)}{\\exp\\left(\\mathbf{u}_k^\\top {\\mathbf{v}}_i\\right)} \\approx \\frac{p_{ij}}{p_{ik}}.$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d8cba4b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$$f(\\mathbf{u}_j, \\mathbf{u}_k, {\\mathbf{v}}_i) = \\frac{\\exp\\left(\\mathbf{u}_j^\\top {\\mathbf{v}}_i\\right)}{\\exp\\left(\\mathbf{u}_k^\\top {\\mathbf{v}}_i\\right)} \\approx \\frac{p_{ij}}{p_{ik}}.$$\n",
    "\n",
    "- Now let's pick $\\exp\\left(\\mathbf{u}_j^\\top {\\mathbf{v}}_i\\right) \\approx \\alpha p_{ij}$, where $\\alpha$ is a constant.\n",
    "- Since $p_{ij}=x_{ij}/x_i$, after taking the logarithm on both sides we get $\\mathbf{u}_j^\\top {\\mathbf{v}}_i \\approx \\log\\,\\alpha + \\log\\,x_{ij} - \\log\\,x_i$. We may use additional bias terms to fit $- \\log\\, \\alpha + \\log\\, x_i$, such as the center word bias $b_i$ and the context word bias $c_j$:\n",
    "\n",
    "$$\\mathbf{u}_j^\\top \\mathbf{v}_i + b_i + c_j \\approx \\log\\, x_{ij}.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "787926d3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Subword Embedding\n",
    "\n",
    "- In English, words such as \"helps\", \"helped\", and \"helping\" are inflected forms of the same word \"help\". The relationship between \"dog\" and \"dogs\" is the same as that between \"cat\" and \"cats\", and the relationship between \"boy\" and \"boyfriend\" is the same as that between \"girl\" and \"girlfriend\". \n",
    "- However, the internal structure of words was neither explored in word2vec nor in GloVe.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b5760cc",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The fastText Model\n",
    "\n",
    "- Recall how words are represented in word2vec. In both the skip-gram model and the continuous bag-of-words model, different inflected forms of the same word are directly represented by different vectors without shared parameters. To use morphological information, the *fastText* model proposed a *subword embedding* approach, where a subword is a character $n$-gram (Bojanowski et al., 2017).\n",
    "- Instead of learning word-level vector representations, fastText can be considered as the subword-level skip-gram, where each *center word* is represented by the sum of its subword vectors.\n",
    "\n",
    "[Bojanowski et al., 2017] Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5, 135–146."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fbdf824",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Let's illustrate how to obtain  subwords for each center word in fastText using the word \"where\". First, add special characters “&lt;” and “&gt;” at the beginning and end of the word to distinguish prefixes and suffixes from other subwords. Then, extract character $n$-grams from the word. For example, when $n=3$, we obtain all subwords of length 3: \"&lt;wh\", \"whe\", \"her\", \"ere\", \"re&gt;\", and the special subword \"&lt;where&gt;\".\n",
    "- In fastText, for any word $w$, denote by $\\mathcal{G}_w$ the union of all its subwords of length between 3 and 6 and its special subword. The vocabulary is the union of the subwords of all words. Letting $\\mathbf{z}_g$ be the vector of subword $g$ in the dictionary, the vector $\\mathbf{v}_w$ for word $w$ as a center word in the skip-gram model is the sum of its subword vectors:\n",
    "\n",
    "$$\\mathbf{v}_w = \\sum_{g\\in\\mathcal{G}_w} \\mathbf{z}_g.$$\n",
    "\n",
    "- The vocabulary in fastText is larger, resulting in more model parameters.\n",
    "- Thanks to shared parameters from subwords among words with similar structures, rare words and even out-of-vocabulary words may obtain better vector representations in fastText.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b2475e4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Byte Pair Encoding\n",
    "\n",
    "- In fastText, all the extracted subwords have to be of the specified lengths, such as $3$ to $6$, thus the vocabulary size cannot be predefined.\n",
    "- To allow for variable-length subwords in a fixed-size vocabulary, we can apply a compression algorithm called *byte pair encoding* (BPE) to extract subwords.\n",
    "- Byte pair encoding performs a statistical analysis of the training dataset to discover common symbols within a word, such as consecutive characters of arbitrary length. Starting from symbols of length 1, byte pair encoding iteratively merges the most frequent pair of consecutive symbols to produce new longer symbols. Note that for efficiency, pairs crossing word boundaries are not considered.\n",
    "- In the end, we can use such symbols as subwords to segment words. Byte pair encoding and its variants has been used for input representations in popular natural language processing pretraining models such as GPT-2 (Radford et al., 2019) and RoBERTa (Liu et al., 2019).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "e339a052",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import collections\n",
    "\n",
    "# a special end-of-word symbol '_', and a special unknown symbol '[UNK]'.\n",
    "symbols = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm',\n",
    "           'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z',\n",
    "           '_', '[UNK]']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f3bbf00",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Since we do not consider symbol pairs that cross boundaries of words, we only need a dictionary `raw_token_freqs` that maps words to their frequencies (number of occurrences) in a dataset.\n",
    "- Note that the special symbol `'_'` is appended to each word so that we can easily recover a word sequence (e.g., \"a taller man\") from a sequence of output symbols ( e.g., \"a_ tall er_ man\").\n",
    "- Since we start the merging process from a vocabulary of only single characters and special symbols, space is inserted between every pair of consecutive characters within each word (keys of the dictionary `token_freqs`). In other words, space is the delimiter between symbols within a word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "9aeb7a10",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'f a s t _': 4, 'f a s t e r _': 3, 't a l l _': 5, 't a l l e r _': 4}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_token_freqs = {'fast_': 4, 'faster_': 3, 'tall_': 5, 'taller_': 4}\n",
    "token_freqs = {}\n",
    "for token, freq in raw_token_freqs.items():\n",
    "    token_freqs[' '.join(list(token))] = raw_token_freqs[token]\n",
    "token_freqs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6cf37d2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- We define the following `get_max_freq_pair` function that returns the most frequent pair of consecutive symbols within a word, where words come from keys of the input dictionary `token_freqs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "ba61178e",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def get_max_freq_pair(token_freqs):\n",
    "    pairs = collections.defaultdict(int)\n",
    "    for token, freq in token_freqs.items():\n",
    "        symbols = token.split()\n",
    "        for i in range(len(symbols) - 1):\n",
    "            # Key of `pairs` is a tuple of two consecutive symbols\n",
    "            pairs[symbols[i], symbols[i + 1]] += freq\n",
    "    return max(pairs, key=pairs.get)  # Key of `pairs` with the max value"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d74b4aa",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- As a greedy approach based on frequency of consecutive symbols, byte pair encoding will use the following `merge_symbols` function to merge the most frequent pair of consecutive symbols to produce new symbols."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "45edd038",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def merge_symbols(max_freq_pair, token_freqs, symbols):\n",
    "    symbols.append(''.join(max_freq_pair))\n",
    "    new_token_freqs = dict()\n",
    "    for token, freq in token_freqs.items():\n",
    "        new_token = token.replace(' '.join(max_freq_pair),\n",
    "                                  ''.join(max_freq_pair))\n",
    "        new_token_freqs[new_token] = token_freqs[token]\n",
    "    return new_token_freqs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "694fff55",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Now we iteratively perform the byte pair encoding algorithm over the keys of the dictionary `token_freqs`. \n",
    "- In the first iteration, the most frequent pair of consecutive symbols are `'t'` and `'a'`, thus byte pair encoding merges them to produce a new symbol `'ta'`. \n",
    "- In the second iteration, byte pair encoding continues to merge `'ta'` and `'l'` to result in another new symbol `'tal'`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "d0c69278",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "merge #1: ('t', 'a')\n",
      "merge #2: ('ta', 'l')\n",
      "merge #3: ('tal', 'l')\n",
      "merge #4: ('f', 'a')\n",
      "merge #5: ('fa', 's')\n",
      "merge #6: ('fas', 't')\n",
      "merge #7: ('e', 'r')\n",
      "merge #8: ('er', '_')\n",
      "merge #9: ('tall', '_')\n",
      "merge #10: ('fast', '_')\n"
     ]
    }
   ],
   "source": [
    "num_merges = 10\n",
    "for i in range(num_merges):\n",
    "    max_freq_pair = get_max_freq_pair(token_freqs)\n",
    "    token_freqs = merge_symbols(max_freq_pair, token_freqs, symbols)\n",
    "    print(f'merge #{i + 1}:', max_freq_pair)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ea1277a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- After 10 iterations of byte pair encoding, we can see that list `symbols` now contains 10 more symbols that are iteratively merged from other symbols."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "2201dd66",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '_', '[UNK]', 'ta', 'tal', 'tall', 'fa', 'fas', 'fast', 'er', 'er_', 'tall_', 'fast_']\n"
     ]
    }
   ],
   "source": [
    "print(symbols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "e1447b51",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['fast_', 'fast er_', 'tall_', 'tall er_']\n"
     ]
    }
   ],
   "source": [
    "print(list(token_freqs.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95c0997c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Note that the result of byte pair encoding depends on the dataset being used. We can also use the subwords learned from one dataset to segment words of another dataset. As a greedy approach, the following `segment_BPE` function tries to break words into the longest possible subwords from the input argument `symbols`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "409be6b8",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def segment_BPE(tokens, symbols):\n",
    "    outputs = []\n",
    "    for token in tokens:\n",
    "        start, end = 0, len(token)\n",
    "        cur_output = []\n",
    "        # Segment token with the longest possible subwords from symbols\n",
    "        while start < len(token) and start < end:\n",
    "            if token[start: end] in symbols:\n",
    "                cur_output.append(token[start: end])\n",
    "                start = end\n",
    "                end = len(token)\n",
    "            else:\n",
    "                end -= 1\n",
    "        if start < len(token):\n",
    "            cur_output.append('[UNK]')\n",
    "        outputs.append(' '.join(cur_output))\n",
    "    return outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "db41c15c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['tall e s t _', 'fa t t er_']\n"
     ]
    }
   ],
   "source": [
    "tokens = ['tallest_', 'fatter_']\n",
    "print(segment_BPE(tokens, symbols))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "379bc3cf",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Word Similarity and Analogy\n",
    "\n",
    "- To demonstrate semantics of pretrained word vectors from large corpora in a straightforward way, let’s apply them in the word similarity and analogy tasks.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9a7c10c",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Loading Pretrained Word Vectors\n",
    "\n",
    "- Below lists pretrained GloVe embeddings of dimension 50, 100, and 300, which can be downloaded from the [GloVe website](https://nlp.stanford.edu/projects/glove/). The pretrained fastText embeddings are available in multiple languages. Here we consider one English version (300-dimensional \"wiki.en\") that can be downloaded from the\n",
    "[fastText website](https://fasttext.cc/).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "160727f3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "d2l.DATA_HUB['glove.6b.50d'] = (d2l.DATA_URL + 'glove.6B.50d.zip',\n",
    "                                '0b8703943ccdb6eb788e6f091b8946e82231bc4d')\n",
    "\n",
    "#@save\n",
    "d2l.DATA_HUB['glove.6b.100d'] = (d2l.DATA_URL + 'glove.6B.100d.zip',\n",
    "                                 'cd43bfb07e44e6f27cbcc7bc9ae3d80284fdaf5a')\n",
    "\n",
    "#@save\n",
    "d2l.DATA_HUB['glove.42b.300d'] = (d2l.DATA_URL + 'glove.42B.300d.zip',\n",
    "                                  'b5116e234e9eb9076672cfeabf5469f3eec904fa')\n",
    "\n",
    "#@save\n",
    "d2l.DATA_HUB['wiki.en'] = (d2l.DATA_URL + 'wiki.en.zip',\n",
    "                           'c1816da3821ae9f43899be655002f6c723e91b88')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34e6ebdb",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- To load these pretrained GloVe and fastText embeddings, we define the following `TokenEmbedding` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "38b69801",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "class TokenEmbedding:\n",
    "    \"\"\"Token Embedding.\"\"\"\n",
    "    def __init__(self, embedding_name):\n",
    "        self.idx_to_token, self.idx_to_vec = self._load_embedding(\n",
    "            embedding_name)\n",
    "        self.unknown_idx = 0\n",
    "        self.token_to_idx = {token: idx for idx, token in\n",
    "                             enumerate(self.idx_to_token)}\n",
    "\n",
    "    def _load_embedding(self, embedding_name):\n",
    "        idx_to_token, idx_to_vec = ['<unk>'], []\n",
    "        data_dir = d2l.download_extract(embedding_name)\n",
    "        # GloVe website: https://nlp.stanford.edu/projects/glove/\n",
    "        # fastText website: https://fasttext.cc/\n",
    "        with open(os.path.join(data_dir, 'vec.txt'), 'r') as f:\n",
    "            for line in f:\n",
    "                elems = line.rstrip().split(' ')\n",
    "                token, elems = elems[0], [float(elem) for elem in elems[1:]]\n",
    "                # Skip header information, such as the top row in fastText\n",
    "                if len(elems) > 1:\n",
    "                    idx_to_token.append(token)\n",
    "                    idx_to_vec.append(elems)\n",
    "        idx_to_vec = [[0] * len(idx_to_vec[0])] + idx_to_vec\n",
    "        return idx_to_token, torch.tensor(idx_to_vec)\n",
    "\n",
    "    def __getitem__(self, tokens):\n",
    "        indices = [self.token_to_idx.get(token, self.unknown_idx)\n",
    "                   for token in tokens]\n",
    "        vecs = self.idx_to_vec[torch.tensor(indices)]\n",
    "        return vecs\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.idx_to_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c957681",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Below we load the 50-dimensional GloVe embeddings (pretrained on a Wikipedia subset). When creating the `TokenEmbedding` instance, the specified embedding file has to be downloaded if it was not yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "1128b5a9",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "glove_6b50d = TokenEmbedding('glove.6b.50d')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "78e24e5f",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "400001"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(glove_6b50d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "46dfa521",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3367, 'beautiful')"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "glove_6b50d.token_to_idx['beautiful'], glove_6b50d.idx_to_token[3367]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d12b5c31",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Applying Pretrained Word Vectors\n",
    "\n",
    "#### Word Similarity\n",
    "\n",
    "- in order to find semantically similar words for an input word based on cosine similarities between word vectors, we implement the following knn ($k$-nearest neighbors) function.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "71e4555f",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def knn(W, x, k):\n",
    "    # Add 1e-9 for numerical stability\n",
    "    cos = torch.mv(W, x.reshape(-1,)) / (\n",
    "        torch.sqrt(torch.sum(W * W, axis=1) + 1e-9) *\n",
    "        torch.sqrt((x * x).sum()))\n",
    "    _, topk = torch.topk(cos, k=k)\n",
    "    return topk, [cos[int(i)] for i in topk]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "1ea6ee82",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def get_similar_tokens(query_token, k, embed):\n",
    "    topk, cos = knn(embed.idx_to_vec, embed[[query_token]], k + 1)\n",
    "    for i, c in zip(topk[1:], cos[1:]):  # Exclude the input word\n",
    "        print(f'cosine sim={float(c):.3f}: {embed.idx_to_token[int(i)]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "416928db",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cosine sim=0.856: chips\n",
      "cosine sim=0.749: intel\n",
      "cosine sim=0.749: electronics\n"
     ]
    }
   ],
   "source": [
    "get_similar_tokens('chip', 3, glove_6b50d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "da75f8cd",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cosine sim=0.839: babies\n",
      "cosine sim=0.800: boy\n",
      "cosine sim=0.792: girl\n"
     ]
    }
   ],
   "source": [
    "get_similar_tokens('baby', 3, glove_6b50d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "28a05605",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cosine sim=0.921: lovely\n",
      "cosine sim=0.893: gorgeous\n",
      "cosine sim=0.830: wonderful\n"
     ]
    }
   ],
   "source": [
    "get_similar_tokens('beautiful', 3, glove_6b50d)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e48145c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Word Analogy\n",
    "\n",
    "- Besides finding similar words, we can also apply word vectors to word analogy tasks. For example, “man”:“woman”::“son”:“daughter” is the form of a word analogy: “man” is to “woman” as “son” is to “daughter”.\n",
    "- Specifically, the word analogy completion task can be defined as: for a word analogy $a : b :: c : d$, given the first three words $a$, $b$ and $c$, find $d$. \n",
    "- Denote the vector of word $w$ by $\\text{vec}(w)$. To complete the analogy, we will find the word whose vector is most similar to the result of $\\text{vec}(c)+\\text{vec}(b)-\\text{vec}(a)$.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "24ccf66f",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def get_analogy(token_a, token_b, token_c, embed):\n",
    "    vecs = embed[[token_a, token_b, token_c]]\n",
    "    x = vecs[1] - vecs[0] + vecs[2]\n",
    "    topk, cos = knn(embed.idx_to_vec, x, 1)\n",
    "    return embed.idx_to_token[int(topk[0])]  # Remove unknown words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "20a1cb35",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'daughter'"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_analogy('man', 'woman', 'son', glove_6b50d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "48a42ce2",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'japan'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_analogy('beijing', 'china', 'tokyo', glove_6b50d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "07107983",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'biggest'"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_analogy('bad', 'worst', 'big', glove_6b50d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "287b365c",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'went'"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_analogy('do', 'did', 'go', glove_6b50d)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b1ccfd6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Bidirectional Encoder Representations from Transformers (BERT)\n",
    "\n",
    "\n",
    "- We have introduced several word embedding models for natural language understanding. For instance, word2vec and GloVe both assign the same pretrained vector to the same word regardless of the context of the word (if any).\n",
    "- Formally, a context-independent representation of any token $x$ is a function $f(x)$ that only takes $x$ as its input.\n",
    "\n",
    "### From Context-Independent to Context-Sensitive\n",
    "\n",
    "- Given the abundance of polysemy and complex semantics in natural languages, context-independent representations have obvious limitations.\n",
    "- For instance, the word \"crane\" in contexts \"a crane is flying\" and \"a crane driver came\" has completely different meanings; thus, the same word may be assigned different representations depending on contexts.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81f5eb21",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- This motivates the development of *context-sensitive* word representations, where representations of words depend on their contexts. Hence, a context-sensitive representation of token $x$ is a function $f(x, c(x))$ depending on both $x$ and its context $c(x)$.\n",
    "- Popular context-sensitive representations include \n",
    "    - TagLM (language-model-augmented sequence tagger) (Peters et al., 2017), \n",
    "    - CoVe (Context Vectors) (McCann et al., 2017), and \n",
    "    - ELMo (Embeddings from Language Models) (Peters et al., 2018).\n",
    "\n",
    "[Peters et al., 2017b] Peters, M., Ammar, W., Bhagavatula, C., & Power, R. (2017). Semi-supervised sequence tagging with bidirectional language models. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1756–1765).\n",
    "\n",
    "[McCann et al., 2017] McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017). Learned in translation: contextualized word vectors. Advances in Neural Information Processing Systems (pp. 6294–6305).\n",
    "\n",
    "[Peters et al., 2018] Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) (pp. 2227–2237)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ded85ede",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- For example, by taking the entire sequence as input, ELMo is a function that assigns a representation to each word from the input sequence. \n",
    "- Specifically, ELMo **combines all the intermediate layer representations from pretrained bidirectional LSTM as the output representation**. Then the ELMo representation will be added to a downstream task's existing supervised model as **additional features**, such as by concatenating ELMo representation and the original representation (e.g., GloVe) of tokens in the existing model. \n",
    "- On the one hand, all the weights in the pretrained bidirectional LSTM model are **frozen after ELMo representations are added**. \n",
    "- On the other hand, the existing supervised model **is specifically customized for a given task**. Leveraging different best models for different tasks at that time, adding ELMo improved the state of the art across six natural language processing tasks: sentiment analysis, natural language inference, semantic role labeling, coreference resolution, named entity recognition, and question answering."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d17ffb5",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### From Task-Specific to Task-Agnostic\n",
    "\n",
    "![gpt_paper.png](./images/8/gpt_paper.png)\n",
    "\n",
    "- Although ELMo has significantly improved solutions to a diverse set of natural language processing tasks, each solution still hinges on a task-specific architecture. However, it is practically non-trivial to craft a specific architecture for every natural language processing task. \n",
    "- The GPT (Generative Pre-Training) model represents an effort in designing a general task-agnostic model for context-sensitive representations (Radford et al., 2018). Built on a Transformer decoder, GPT pretrains a language model that will be used to represent text sequences.\n",
    "\n",
    "[Radford et al., 2018] Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training. OpenAI.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ef09fe4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- When applying GPT to a downstream task, the output of the language model will be fed into an added linear output layer to predict the label of the task. In sharp contrast to ELMo that freezes parameters of the pretrained model, GPT fine-tunes *all* the parameters in the pretrained Transformer decoder during supervised learning of the downstream task. \n",
    "- GPT was evaluated on twelve tasks of natural language inference, question answering, sentence similarity, and classification, and improved the state of the art in nine of them with minimal changes to the model architecture.\n",
    "- However, due to the autoregressive nature of language models, GPT only looks forward (left-to-right)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ab148af",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**You**: What is the autoregressive nature of language models?\n",
    "\n",
    "**ChatGPT**: The autoregressive nature of language models means that they predict the probability distribution of a given sequence of words by conditioning every prediction on the preceding words of the sequence. In other words, an autoregressive language model generates a sequence of tokens (words) one at a time, where the probability distribution of each subsequent word is conditioned on the previous words in the sequence. \n",
    "\n",
    "For example, if the model is generating the sentence \"The cat is sitting on the \\_\\_\\_,\" it will first predict the probability distribution of the next token (word) based on the context of the preceding words \"The cat is sitting on the\". The probability distribution will be conditioned on the distribution of all the previous tokens in the sequence. \n",
    "\n",
    "Therefore, autoregressive language models, such as GPT-2 and GPT-3, can produce coherent and contextually relevant text due to their ability to use the context of preceding words to inform the next word prediction.\n",
    "\n",
    "---\n",
    "\n",
    "**You**: So it that possible for language models to predict the distribution based on both preceding and succeeding sequence?\n",
    "\n",
    "**ChatGPT**: Yes, it is possible for language models to predict the probability distribution of a given sequence based on both the preceding and succeeding sequence in the text. Such language models are called bidirectional language models.\n",
    "\n",
    "Bidirectional language models use a technique called bidirectional encoding, which allows them to take into account the context of both preceding (left context) and succeeding (right context) words when generating text. Rather than processing text from left to right or right to left, bidirectional language models process the text in both directions simultaneously, using two separate layers of hidden units in the neural network.\n",
    "\n",
    "For example, a bidirectional language model trained on a sentence such as \"The cat is sitting on the \\_\\_\\_\" would consider the probabilities of the next word based on not only the preceding words, but also the succeeding words in the sentence. This allows it to better capture the context of the sentence and generate more coherent and relevant text.\n",
    "\n",
    "Bidirectional language models have been shown to be effective in a wide range of natural language processing tasks, such as named entity recognition, sentiment analysis, and machine translation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22e3ad70",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### BERT: Combining the Best of Both Worlds\n",
    "\n",
    "![elmo-gpt-bert.svg](./images/8/elmo-gpt-bert.svg)\n",
    "\n",
    "- As we have seen, ELMo encodes context bidirectionally but uses task-specific architectures; while GPT is task-agnostic but encodes context left-to-right. Combining the best of both worlds, BERT (Bidirectional Encoder Representations from Transformers) **encodes context bidirectionally and requires minimal architecture changes** for a wide range of natural language processing tasks (Devlin et al., 2018).\n",
    "\n",
    "\n",
    "[Devlin et al., 2018] Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "801bb786",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "![bert_paper.png](./images/8/bert_paper.png)\n",
    "\n",
    "- Using a pretrained Transformer encoder, BERT is able to represent any token based on its bidirectional context. During supervised learning of downstream tasks, BERT is similar to GPT in two aspects. \n",
    "    - First, BERT representations will be fed into an added output layer, with minimal changes to the model architecture depending on nature of tasks, such as predicting for every token vs. predicting for the entire sequence. \n",
    "    - Second, all the parameters of the pretrained Transformer encoder are fine-tuned, while the additional output layer will be trained from scratch.\n",
    "- BERT further improved the state of the art on eleven natural language processing tasks under broad categories of (i) single text classification (e.g., sentiment analysis), (ii) text pair classification (e.g., natural language inference), (iii) question answering, (iv) text tagging (e.g., named entity recognition).\n",
    "- Simple but powerful."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e34d0706",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Input Representation\n",
    "\n",
    "- The BERT input sequence unambiguously represents both single text and text pairs. \n",
    "    - In the former, the BERT input sequence is the concatenation of the special classification token “\\<cls\\>”, tokens of a text sequence, and the special separation token “\\<sep\\>”. \n",
    "    - In the latter, the BERT input sequence is the concatenation of “\\<cls\\>”, tokens of the first text sequence, “\\<sep\\>”, tokens of the second text sequence, and “\\<sep\\>”.\n",
    "- To distinguish text pairs, the learned segment embeddings $\\mathbf{e}_A$ and $\\mathbf{e}_B$ are added to the token embeddings of the first sequence and the second sequence, respectively. For single text inputs, only $\\mathbf{e}_A$ is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "e9e3f3cd",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "def get_tokens_and_segments(tokens_a, tokens_b=None):\n",
    "    \"\"\"Get tokens of the BERT input sequence and their segment IDs.\"\"\"\n",
    "    tokens = ['<cls>'] + tokens_a + ['<sep>']\n",
    "    # 0 and 1 are marking segment A and B, respectively\n",
    "    segments = [0] * (len(tokens_a) + 2)\n",
    "    if tokens_b is not None:\n",
    "        tokens += tokens_b + ['<sep>']\n",
    "        segments += [1] * (len(tokens_b) + 1)\n",
    "    return tokens, segments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b262fe8c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "![bert-input.svg](./images/8/bert-input.svg)\n",
    "\n",
    "- BERT chooses the Transformer encoder as its bidirectional architecture. \n",
    "- Common in the Transformer encoder, positional embeddings are added at every position of the BERT input sequence. \n",
    "- However, different from the original Transformer encoder, BERT uses *learnable* positional embeddings. \n",
    "- To sum up, the figure below shows that the embeddings of the BERT input sequence are the sum of the token embeddings, segment embeddings, and positional embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "7c8881ca",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "class BERTEncoder(nn.Module):\n",
    "    \"\"\"BERT encoder.\"\"\"\n",
    "    def __init__(self, vocab_size, num_hiddens, ffn_num_hiddens, num_heads,\n",
    "                 num_blks, dropout, max_len=1000, **kwargs):\n",
    "        super(BERTEncoder, self).__init__(**kwargs)\n",
    "        self.token_embedding = nn.Embedding(vocab_size, num_hiddens)\n",
    "        self.segment_embedding = nn.Embedding(2, num_hiddens)\n",
    "        self.blks = nn.Sequential()\n",
    "        for i in range(num_blks):\n",
    "            self.blks.add_module(f\"{i}\", d2l.TransformerEncoderBlock(\n",
    "                num_hiddens, ffn_num_hiddens, num_heads, dropout, True))\n",
    "        # In BERT, positional embeddings are learnable, thus we create a\n",
    "        # parameter of positional embeddings that are long enough\n",
    "        self.pos_embedding = nn.Parameter(torch.randn(1, max_len, num_hiddens))\n",
    "\n",
    "    def forward(self, tokens, segments, valid_lens):\n",
    "        # Shape of `X` remains unchanged in the following code snippet:\n",
    "        # (batch size, max sequence length, `num_hiddens`)\n",
    "        X = self.token_embedding(tokens) + self.segment_embedding(segments)\n",
    "        X = X + self.pos_embedding[:, :X.shape[1], :]\n",
    "        for blk in self.blks:\n",
    "            X = blk(X, valid_lens)\n",
    "        return X"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d3e65ec",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- We define `tokens` to be 2 BERT input sequences of length 8, where each token is an index of the vocabulary.\n",
    "- The forward inference of `BERTEncoder` with the input `tokens` returns the encoded result where each token is represented by a vector whose length is predefined by the hyperparameter `num_hiddens`.\n",
    "- This hyperparameter is usually referred to as the *hidden size* (number of hidden units) of the Transformer encoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "ded6185d",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/haodong/Applications/Miniconda/Miniconda/envs/dl/lib/python3.9/site-packages/torch/nn/modules/lazy.py:178: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.\n",
      "  warnings.warn('Lazy modules are a new feature under heavy development '\n"
     ]
    }
   ],
   "source": [
    "vocab_size, num_hiddens, ffn_num_hiddens, num_heads = 10000, 768, 1024, 4\n",
    "ffn_num_input, num_blks, dropout = 768, 2, 0.2\n",
    "encoder = BERTEncoder(vocab_size, num_hiddens, ffn_num_hiddens, num_heads, num_blks, dropout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "97440335",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 8, 768])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens = torch.randint(0, vocab_size, (2, 8))\n",
    "segments = torch.tensor([[0, 0, 0, 0, 1, 1, 1, 1], [0, 0, 0, 1, 1, 1, 1, 1]])\n",
    "encoded_X = encoder(tokens, segments, None)\n",
    "encoded_X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48f333b1",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Pretraining Tasks\n",
    "\n",
    "- Next, we will use these encoded representations to compute the loss function for pretraining BERT. \n",
    "    - masked language modeling \n",
    "    - next sentence prediction.\n",
    "\n",
    "#### Masked Language Modeling\n",
    "\n",
    "- To encode context bidirectionally for representing each token, BERT randomly masks tokens and uses tokens from the bidirectional context to predict the masked tokens in a self-supervised fashion. This task is referred to as a *masked language model*.\n",
    "- In this pretraining task, 15% of tokens will be selected at random as the masked tokens for prediction.\n",
    "    - a special “\\<mask\\>” token for 80% of the time (e.g., “this movie is great” becomes “this movie is \\<mask\\>”);\n",
    "    - a random token for 10% of the time (e.g., “this movie is great” becomes “this movie is drink”);\n",
    "    - the unchanged label token for 10% of the time (e.g., “this movie is great” becomes “this movie is great”).\n",
    "- Note that for 10% of 15% time a random token is inserted. This occasional noise encourages BERT to be less biased towards the masked token (especially when the label token remains unchanged) in its bidirectional context encoding."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "904c3adc",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- We implement the following `MaskLM` class to predict masked tokens in the masked language model task of BERT pretraining. The prediction uses a one-hidden-layer MLP (`self.mlp`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "c4184a9c",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "class MaskLM(nn.Module):\n",
    "    \"\"\"The masked language model task of BERT.\"\"\"\n",
    "    def __init__(self, vocab_size, num_hiddens, **kwargs):\n",
    "        super(MaskLM, self).__init__(**kwargs)\n",
    "        self.mlp = nn.Sequential(nn.LazyLinear(num_hiddens),\n",
    "                                 nn.ReLU(),\n",
    "                                 nn.LayerNorm(num_hiddens),\n",
    "                                 nn.LazyLinear(vocab_size))\n",
    "\n",
    "    def forward(self, X, pred_positions):\n",
    "        num_pred_positions = pred_positions.shape[1]\n",
    "        pred_positions = pred_positions.reshape(-1)\n",
    "        batch_size = X.shape[0]\n",
    "        batch_idx = torch.arange(0, batch_size)\n",
    "        # Suppose that `batch_size` = 2, `num_pred_positions` = 3, then\n",
    "        # `batch_idx` is `torch.tensor([0, 0, 0, 1, 1, 1])`\n",
    "        batch_idx = torch.repeat_interleave(batch_idx, num_pred_positions)\n",
    "        masked_X = X[batch_idx, pred_positions]\n",
    "        masked_X = masked_X.reshape((batch_size, num_pred_positions, -1))\n",
    "        mlm_Y_hat = self.mlp(masked_X)\n",
    "        return mlm_Y_hat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "436cd1a6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Recall that `encoded_X` from the forward inference of `BERTEncoder` represents 2 BERT input sequences. We define `mlm_positions` as the 3 indices to predict in either BERT input sequence of `encoded_X`. The forward inference of `mlm` returns prediction results `mlm_Y_hat` at all the masked positions `mlm_positions` of `encoded_X`. For each prediction, the size of the result is equal to the vocabulary size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "38a39395",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3, 10000])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mlm = MaskLM(vocab_size, num_hiddens)\n",
    "mlm_positions = torch.tensor([[1, 5, 2], [6, 1, 5]])\n",
    "mlm_Y_hat = mlm(encoded_X, mlm_positions)\n",
    "mlm_Y_hat.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "cf526ec1",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([6])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mlm_Y = torch.tensor([[7, 8, 9], [10, 20, 30]])\n",
    "# With the ground truth labels mlm_Y of the predicted tokens mlm_Y_hat under masks, \n",
    "# we can calculate the cross-entropy loss of the masked language model task in BERT pretraining.\n",
    "loss = nn.CrossEntropyLoss(reduction='none')\n",
    "mlm_l = loss(mlm_Y_hat.reshape((-1, vocab_size)), mlm_Y.reshape(-1))\n",
    "mlm_l.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0510c5ff",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Next Sentence Prediction\n",
    "\n",
    "- Although masked language modeling is able to encode bidirectional context for representing words, it does not explicitly model the **logical relationship** between text pairs.\n",
    "- To help understand the relationship between two text sequences, BERT considers a binary classification task, *next sentence prediction*, in its pretraining. When generating sentence pairs for pretraining, for half of the time they are indeed consecutive sentences with the label “True”; while for the other half of the time the second sentence is randomly sampled from the corpus with the label “False”.\n",
    "- The following `NextSentencePred` class uses a one-hidden-layer MLP to predict whether the second sentence is the next sentence of the first in the BERT input sequence. Due to self-attention in the Transformer encoder, the BERT representation of the special token “\\<cls\\>” encodes both the two sentences from the input. Hence, the output layer (`self.output`) of the MLP classifier takes X as input, where X is the output of the MLP hidden layer whose input is the encoded “\\<cls\\>” token.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "538ab181",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "class NextSentencePred(nn.Module):\n",
    "    \"\"\"The next sentence prediction task of BERT.\"\"\"\n",
    "    def __init__(self, **kwargs):\n",
    "        super(NextSentencePred, self).__init__(**kwargs)\n",
    "        self.output = nn.LazyLinear(2)\n",
    "\n",
    "    def forward(self, X):\n",
    "        # `X` shape: (batch size, `num_hiddens`)\n",
    "        return self.output(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "87f837a7",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 2])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# PyTorch by default will not flatten the tensor as seen in mxnet where, if\n",
    "# flatten=True, all but the first axis of input data are collapsed together\n",
    "encoded_X = torch.flatten(encoded_X, start_dim=1)\n",
    "# input_shape for NSP: (batch size, `num_hiddens`)\n",
    "nsp = NextSentencePred()\n",
    "nsp_Y_hat = nsp(encoded_X)\n",
    "nsp_Y_hat.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "128f61b4",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2])"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nsp_y = torch.tensor([0, 1])\n",
    "nsp_l = loss(nsp_Y_hat, nsp_y)\n",
    "nsp_l.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98db9612",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Putting It All Together\n",
    "\n",
    "- When pretraining BERT, the final loss function is a linear combination of both the loss functions for masked language modeling and next sentence prediction. \n",
    "- Now we can define the `BERTModel` class by instantiating the three classes `BERTEncoder`, `MaskLM`, and `NextSentencePred`. The forward inference returns the encoded BERT representations `encoded_X`, predictions of masked language modeling `mlm_Y_hat`, and next sentence predictions `nsp_Y_hat`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "4de5525a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "class BERTModel(nn.Module):\n",
    "    \"\"\"The BERT model.\"\"\"\n",
    "    def __init__(self, vocab_size, num_hiddens, ffn_num_hiddens,\n",
    "                 num_heads, num_blks, dropout, max_len=1000):\n",
    "        super(BERTModel, self).__init__()\n",
    "        self.encoder = BERTEncoder(vocab_size, num_hiddens, ffn_num_hiddens,\n",
    "                                   num_heads, num_blks, dropout, max_len=max_len)\n",
    "        self.hidden = nn.Sequential(nn.LazyLinear(num_hiddens), nn.Tanh())\n",
    "        self.mlm = MaskLM(vocab_size, num_hiddens)\n",
    "        self.nsp = NextSentencePred()\n",
    "\n",
    "    def forward(self, tokens, segments, valid_lens=None, pred_positions=None):\n",
    "        encoded_X = self.encoder(tokens, segments, valid_lens)\n",
    "        if pred_positions is not None:\n",
    "            mlm_Y_hat = self.mlm(encoded_X, pred_positions)\n",
    "        else:\n",
    "            mlm_Y_hat = None\n",
    "        # The hidden layer of the MLP classifier for next sentence prediction.\n",
    "        # 0 is the index of the '<cls>' token\n",
    "        nsp_Y_hat = self.nsp(self.hidden(encoded_X[:, 0, :]))\n",
    "        return encoded_X, mlm_Y_hat, nsp_Y_hat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80bc500c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Pretraining BERT\n",
    "\n",
    "- To start, we load the WikiText-2 dataset as minibatches of pretraining examples for masked language modeling and next sentence prediction. The batch size is 512 and the maximum length of a BERT input sequence is 64. Note that in the original BERT model, the maximum length is 512."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "1e5818b2",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "batch_size, max_len = 512, 64\n",
    "train_iter, vocab = d2l.load_data_wiki(batch_size, max_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a40ca070",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- The original BERT has two versions of different model sizes.\n",
    "- The base model ($\\text{BERT}_{\\text{BASE}}$) uses 12 layers (Transformer encoder blocks) with 768 hidden units (hidden size) and 12 self-attention heads.\n",
    "- The large model ($\\text{BERT}_{\\text{LARGE}}$) uses 24 layers with 1024 hidden units and 16 self-attention heads. \n",
    "- Notably, the former has 110 million parameters while the latter has 340 million parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "896d1047",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# For demonstration with ease, we define a small BERT, \n",
    "# using 2 layers, 128 hidden units, and 2 self-attention heads.\n",
    "\n",
    "net = d2l.BERTModel(len(vocab), num_hiddens=128, ffn_num_hiddens=256, num_heads=2, num_blks=2, dropout=0.2)\n",
    "devices = d2l.try_all_gpus()\n",
    "loss = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a208ead",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Before defining the training loop, we define a helper function `_get_batch_loss_bert`. \n",
    "- Given the shard of training examples, this function computes the loss for both the masked language modeling and next sentence prediction tasks. Note that the final loss of BERT pretraining is just the sum of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "75b998e3",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "def _get_batch_loss_bert(net, loss, vocab_size, tokens_X, segments_X, valid_lens_x,\n",
    "                         pred_positions_X, mlm_weights_X, mlm_Y, nsp_y):\n",
    "    # Forward pass\n",
    "    _, mlm_Y_hat, nsp_Y_hat = net(tokens_X, segments_X,\n",
    "                                  valid_lens_x.reshape(-1),\n",
    "                                  pred_positions_X)\n",
    "    # Compute masked language model loss\n",
    "    mlm_l = loss(mlm_Y_hat.reshape(-1, vocab_size), mlm_Y.reshape(-1)) *\\\n",
    "    mlm_weights_X.reshape(-1, 1)\n",
    "    mlm_l = mlm_l.sum() / (mlm_weights_X.sum() + 1e-8)\n",
    "    # Compute next sentence prediction loss\n",
    "    nsp_l = loss(nsp_Y_hat, nsp_y)\n",
    "    l = mlm_l + nsp_l\n",
    "    return mlm_l, nsp_l, l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "3e231f21",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def train_bert(train_iter, net, loss, vocab_size, devices, num_steps):\n",
    "    net(*next(iter(train_iter))[:4])\n",
    "    net = nn.DataParallel(net, device_ids=devices).to(devices[0])\n",
    "    trainer = torch.optim.Adam(net.parameters(), lr=0.01)\n",
    "    step, timer = 0, d2l.Timer()\n",
    "    animator = d2l.Animator(xlabel='step', ylabel='loss',\n",
    "                            xlim=[1, num_steps], legend=['mlm', 'nsp'])\n",
    "    # Sum of masked language modeling losses, sum of next sentence prediction\n",
    "    # losses, no. of sentence pairs, count\n",
    "    metric = d2l.Accumulator(4)\n",
    "    num_steps_reached = False\n",
    "    while step < num_steps and not num_steps_reached:\n",
    "        for tokens_X, segments_X, valid_lens_x, pred_positions_X,\\\n",
    "            mlm_weights_X, mlm_Y, nsp_y in train_iter:\n",
    "            tokens_X = tokens_X.to(devices[0])\n",
    "            segments_X = segments_X.to(devices[0])\n",
    "            valid_lens_x = valid_lens_x.to(devices[0])\n",
    "            pred_positions_X = pred_positions_X.to(devices[0])\n",
    "            mlm_weights_X = mlm_weights_X.to(devices[0])\n",
    "            mlm_Y, nsp_y = mlm_Y.to(devices[0]), nsp_y.to(devices[0])\n",
    "            trainer.zero_grad()\n",
    "            timer.start()\n",
    "            mlm_l, nsp_l, l = _get_batch_loss_bert(\n",
    "                net, loss, vocab_size, tokens_X, segments_X, valid_lens_x,\n",
    "                pred_positions_X, mlm_weights_X, mlm_Y, nsp_y)\n",
    "            l.backward()\n",
    "            trainer.step()\n",
    "            metric.add(mlm_l, nsp_l, tokens_X.shape[0], 1)\n",
    "            timer.stop()\n",
    "            animator.add(step + 1,\n",
    "                         (metric[0] / metric[3], metric[1] / metric[3]))\n",
    "            step += 1\n",
    "            if step == num_steps:\n",
    "                num_steps_reached = True\n",
    "                break\n",
    "\n",
    "    print(f'MLM loss {metric[0] / metric[3]:.3f}, '\n",
    "          f'NSP loss {metric[1] / metric[3]:.3f}')\n",
    "    print(f'{metric[2] / timer.sum():.1f} sentence pairs/sec on '\n",
    "          f'{str(devices)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "068e0868",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MLM loss 5.389, NSP loss 0.773\n",
      "8976.9 sentence pairs/sec on [device(type='cuda', index=0)]\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"249.465625pt\" height=\"180.65625pt\" viewBox=\"0 0 249.465625 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-03-21T14:36:47.055262</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 249.465625 180.65625 \n",
       "L 249.465625 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.603125 143.1 \n",
       "L 235.903125 143.1 \n",
       "L 235.903125 7.2 \n",
       "L 40.603125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 76.474554 143.1 \n",
       "L 76.474554 7.2 \n",
       "\" clip-path=\"url(#p09e7d1e7f1)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m34eb76646a\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m34eb76646a\" x=\"76.474554\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(70.112054 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 116.331696 143.1 \n",
       "L 116.331696 7.2 \n",
       "\" clip-path=\"url(#p09e7d1e7f1)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m34eb76646a\" x=\"116.331696\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(109.969196 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 156.188839 143.1 \n",
       "L 156.188839 7.2 \n",
       "\" clip-path=\"url(#p09e7d1e7f1)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m34eb76646a\" x=\"156.188839\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 30 -->\n",
       "      <g transform=\"translate(149.826339 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 196.045982 143.1 \n",
       "L 196.045982 7.2 \n",
       "\" clip-path=\"url(#p09e7d1e7f1)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m34eb76646a\" x=\"196.045982\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(189.683482 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 235.903125 143.1 \n",
       "L 235.903125 7.2 \n",
       "\" clip-path=\"url(#p09e7d1e7f1)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m34eb76646a\" x=\"235.903125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 50 -->\n",
       "      <g transform=\"translate(229.540625 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- step -->\n",
       "     <g transform=\"translate(127.4375 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-73\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"52.099609\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"91.308594\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"152.832031\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 40.603125 119.694364 \n",
       "L 235.903125 119.694364 \n",
       "\" clip-path=\"url(#p09e7d1e7f1)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m2b692752b2\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m2b692752b2\" x=\"40.603125\" y=\"119.694364\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(27.240625 123.493583)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 40.603125 92.801003 \n",
       "L 235.903125 92.801003 \n",
       "\" clip-path=\"url(#p09e7d1e7f1)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m2b692752b2\" x=\"40.603125\" y=\"92.801003\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(27.240625 96.600221)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 40.603125 65.907641 \n",
       "L 235.903125 65.907641 \n",
       "\" clip-path=\"url(#p09e7d1e7f1)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m2b692752b2\" x=\"40.603125\" y=\"65.907641\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 6 -->\n",
       "      <g transform=\"translate(27.240625 69.706859)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 40.603125 39.014279 \n",
       "L 235.903125 39.014279 \n",
       "\" clip-path=\"url(#p09e7d1e7f1)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m2b692752b2\" x=\"40.603125\" y=\"39.014279\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 8 -->\n",
       "      <g transform=\"translate(27.240625 42.813497)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 40.603125 12.120917 \n",
       "L 235.903125 12.120917 \n",
       "\" clip-path=\"url(#p09e7d1e7f1)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m2b692752b2\" x=\"40.603125\" y=\"12.120917\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(20.878125 15.920135)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_12\">\n",
       "     <!-- loss -->\n",
       "     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_21\">\n",
       "    <path d=\"M 40.603125 13.377273 \n",
       "L 44.588839 22.41977 \n",
       "L 48.574554 24.260951 \n",
       "L 52.560268 31.94398 \n",
       "L 56.545982 38.463669 \n",
       "L 60.531696 43.649317 \n",
       "L 64.517411 47.728938 \n",
       "L 68.503125 50.692276 \n",
       "L 72.488839 53.163619 \n",
       "L 76.474554 55.458719 \n",
       "L 80.460268 57.526354 \n",
       "L 84.445982 59.191827 \n",
       "L 88.431696 60.623636 \n",
       "L 92.417411 61.843396 \n",
       "L 96.403125 62.969555 \n",
       "L 100.388839 63.978663 \n",
       "L 104.374554 64.928464 \n",
       "L 108.360268 65.747265 \n",
       "L 112.345982 66.520388 \n",
       "L 116.331696 67.160557 \n",
       "L 120.317411 67.751031 \n",
       "L 124.303125 68.238201 \n",
       "L 128.288839 68.613408 \n",
       "L 132.274554 68.982171 \n",
       "L 136.260268 69.266302 \n",
       "L 140.245982 69.56853 \n",
       "L 144.231696 69.965865 \n",
       "L 148.217411 70.247772 \n",
       "L 152.203125 70.510981 \n",
       "L 156.188839 70.764374 \n",
       "L 160.174554 70.994936 \n",
       "L 164.160268 71.184978 \n",
       "L 168.145982 71.358927 \n",
       "L 172.131696 71.540917 \n",
       "L 176.117411 71.720657 \n",
       "L 180.103125 71.875654 \n",
       "L 184.088839 72.04244 \n",
       "L 188.074554 72.18533 \n",
       "L 192.060268 72.331378 \n",
       "L 196.045982 72.497133 \n",
       "L 200.031696 72.648788 \n",
       "L 204.017411 72.825431 \n",
       "L 208.003125 72.988285 \n",
       "L 211.988839 73.173638 \n",
       "L 215.974554 73.324304 \n",
       "L 219.960268 73.504448 \n",
       "L 223.945982 73.645602 \n",
       "L 227.931696 73.804803 \n",
       "L 231.917411 73.960039 \n",
       "L 235.903125 74.122134 \n",
       "\" clip-path=\"url(#p09e7d1e7f1)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_22\">\n",
       "    <path d=\"M 40.603125 136.922727 \n",
       "L 44.588839 132.544846 \n",
       "L 48.574554 125.665789 \n",
       "L 52.560268 125.981016 \n",
       "L 56.545982 128.081927 \n",
       "L 60.531696 129.52532 \n",
       "L 64.517411 130.534173 \n",
       "L 68.503125 131.364727 \n",
       "L 72.488839 131.85864 \n",
       "L 76.474554 132.369446 \n",
       "L 80.460268 132.803626 \n",
       "L 84.445982 133.166943 \n",
       "L 88.431696 133.480086 \n",
       "L 92.417411 133.749182 \n",
       "L 96.403125 133.952101 \n",
       "L 100.388839 134.160288 \n",
       "L 104.374554 134.338718 \n",
       "L 108.360268 134.465959 \n",
       "L 112.345982 134.609933 \n",
       "L 116.331696 134.730178 \n",
       "L 120.317411 134.833373 \n",
       "L 124.303125 134.933146 \n",
       "L 128.288839 135.019219 \n",
       "L 132.274554 135.084564 \n",
       "L 136.260268 135.164503 \n",
       "L 140.245982 135.243279 \n",
       "L 144.231696 135.318321 \n",
       "L 148.217411 135.377857 \n",
       "L 152.203125 135.44158 \n",
       "L 156.188839 135.500635 \n",
       "L 160.174554 135.557112 \n",
       "L 164.160268 135.606351 \n",
       "L 168.145982 135.654194 \n",
       "L 172.131696 135.70125 \n",
       "L 176.117411 135.745497 \n",
       "L 180.103125 135.786439 \n",
       "L 184.088839 135.827628 \n",
       "L 188.074554 135.861168 \n",
       "L 192.060268 135.896221 \n",
       "L 196.045982 135.929728 \n",
       "L 200.031696 135.962184 \n",
       "L 204.017411 135.991047 \n",
       "L 208.003125 136.019933 \n",
       "L 211.988839 136.047022 \n",
       "L 215.974554 136.071524 \n",
       "L 219.960268 136.096997 \n",
       "L 223.945982 136.121865 \n",
       "L 227.931696 136.144566 \n",
       "L 231.917411 136.166798 \n",
       "L 235.903125 136.188703 \n",
       "\" clip-path=\"url(#p09e7d1e7f1)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 40.603125 143.1 \n",
       "L 40.603125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 235.903125 143.1 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.603125 143.1 \n",
       "L 235.903125 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.603125 7.2 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 174.64375 44.55625 \n",
       "L 228.903125 44.55625 \n",
       "Q 230.903125 44.55625 230.903125 42.55625 \n",
       "L 230.903125 14.2 \n",
       "Q 230.903125 12.2 228.903125 12.2 \n",
       "L 174.64375 12.2 \n",
       "Q 172.64375 12.2 172.64375 14.2 \n",
       "L 172.64375 42.55625 \n",
       "Q 172.64375 44.55625 174.64375 44.55625 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_23\">\n",
       "     <path d=\"M 176.64375 20.298437 \n",
       "L 186.64375 20.298437 \n",
       "L 196.64375 20.298437 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- mlm -->\n",
       "     <g transform=\"translate(204.64375 23.798437)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"97.412109\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"125.195312\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_24\">\n",
       "     <path d=\"M 176.64375 34.976562 \n",
       "L 186.64375 34.976562 \n",
       "L 196.64375 34.976562 \n",
       "\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- nsp -->\n",
       "     <g transform=\"translate(204.64375 38.476562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"63.378906\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"115.478516\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p09e7d1e7f1\">\n",
       "   <rect x=\"40.603125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_bert(train_iter, net, loss, len(vocab), devices, 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bfef72e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Representing Text with BERT\n",
    "\n",
    "- The following function returns the BERT (`net`) representations for all tokens in `tokens_a` and `tokens_b`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "f43ba171",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def get_bert_encoding(net, tokens_a, tokens_b=None):\n",
    "    tokens, segments = d2l.get_tokens_and_segments(tokens_a, tokens_b)\n",
    "    token_ids = torch.tensor(vocab[tokens], device=devices[0]).unsqueeze(0)\n",
    "    segments = torch.tensor(segments, device=devices[0]).unsqueeze(0)\n",
    "    valid_len = torch.tensor(len(tokens), device=devices[0]).unsqueeze(0)\n",
    "    encoded_X, _, _ = net(token_ids, segments, valid_len)\n",
    "    return encoded_X"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c94bee8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Consider the sentence “a crane is flying”.\n",
    "- After inserting special tokens “\\<cls\\>” (used for classification) and “\\<sep\\>” (used for separation), the BERT input sequence has a length of six. \n",
    "- Since zero is the index of the “\\<cls\\>” token, `encoded_text[:, 0, :]` is the BERT representation of the entire input sentence. To evaluate the polysemy token “crane”, we also print out the first three elements of the BERT representation of the token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "5ddc49e2",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([1, 6, 128]),\n",
       " torch.Size([1, 128]),\n",
       " tensor([ 0.8837,  1.0539, -0.8053], device='cuda:0', grad_fn=<SliceBackward0>))"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens_a = ['a', 'crane', 'is', 'flying']\n",
    "encoded_text = get_bert_encoding(net, tokens_a)\n",
    "# Tokens: '<cls>', 'a', 'crane', 'is', 'flying', '<sep>'\n",
    "encoded_text_cls = encoded_text[:, 0, :]\n",
    "encoded_text_crane = encoded_text[:, 2, :]\n",
    "encoded_text.shape, encoded_text_cls.shape, encoded_text_crane[0][:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d555978e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Now consider a sentence pair “a crane driver came” and “he just left”. Similarly, `encoded_pair[:, 0, :]` is the encoded result of the entire sentence pair from the pretrained BERT. \n",
    "- Note that the first three elements of the polysemy token “crane” are different from those when the context is different. This supports that BERT representations are **context-sensitive**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "283884f4",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([1, 10, 128]),\n",
       " torch.Size([1, 128]),\n",
       " tensor([ 0.6331,  0.7607, -0.7883], device='cuda:0', grad_fn=<SliceBackward0>))"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens_a, tokens_b = ['a', 'crane', 'driver', 'came'], ['he', 'just', 'left']\n",
    "encoded_pair = get_bert_encoding(net, tokens_a, tokens_b)\n",
    "# Tokens: '<cls>', 'a', 'crane', 'driver', 'came', '<sep>', 'he', 'just',\n",
    "# 'left', '<sep>'\n",
    "encoded_pair_cls = encoded_pair[:, 0, :]\n",
    "encoded_pair_crane = encoded_pair[:, 2, :]\n",
    "encoded_pair.shape, encoded_pair_cls.shape, encoded_pair_crane[0][:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f20e2f19",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Sentiment Analysis: Using RNN\n",
    "\n",
    "![nlp-map-sa-rnn.svg](./images/8/nlp-map-sa-rnn.svg)\n",
    "\n",
    "- Like word similarity and analogy tasks, we can also apply pretrained word vectors to sentiment analysis.\n",
    "- IMDb review dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "f8ed9090",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "batch_size = 64\n",
    "train_iter, test_iter, vocab = d2l.load_data_imdb(batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eed5a379",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Representing Single Text with RNNs\n",
    "\n",
    "- In the following `BiRNN` class, while \n",
    "    - each token of a text sequence gets its individual pretrained GloVe representation via the embedding layer (`self.embedding`), \n",
    "    - the entire sequence is encoded by a bidirectional RNN (`self.encoder`). More concretely, the hidden states (at the last layer) of the bidirectional LSTM at both the initial and final time steps are concatenated as the representation of the text sequence. \n",
    "    - This single text representation is then transformed into output categories by a fully connected layer (`self.decoder`) with two outputs (“positive” and “negative”)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "d666f5c0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "class BiRNN(nn.Module):\n",
    "    def __init__(self, vocab_size, embed_size, num_hiddens,\n",
    "                 num_layers, **kwargs):\n",
    "        super(BiRNN, self).__init__(**kwargs)\n",
    "        self.embedding = nn.Embedding(vocab_size, embed_size)\n",
    "        # Set `bidirectional` to True to get a bidirectional RNN\n",
    "        self.encoder = nn.LSTM(embed_size, num_hiddens, num_layers=num_layers,\n",
    "                                bidirectional=True)\n",
    "        self.decoder = nn.Linear(4 * num_hiddens, 2)\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        # The shape of `inputs` is (batch size, no. of time steps). Because\n",
    "        # LSTM requires its input's first dimension to be the temporal\n",
    "        # dimension, the input is transposed before obtaining token\n",
    "        # representations. The output shape is (no. of time steps, batch size,\n",
    "        # word vector dimension)\n",
    "        embeddings = self.embedding(inputs.T)\n",
    "        self.encoder.flatten_parameters()\n",
    "        # Returns hidden states of the last hidden layer at different time\n",
    "        # steps. The shape of `outputs` is (no. of time steps, batch size,\n",
    "        # 2 * no. of hidden units)\n",
    "        outputs, _ = self.encoder(embeddings)\n",
    "        # Concatenate the hidden states at the initial and final time steps as\n",
    "        # the input of the fully connected layer. Its shape is (batch size,\n",
    "        # 4 * no. of hidden units)\n",
    "        encoding = torch.cat((outputs[0], outputs[-1]), dim=1)\n",
    "        outs = self.decoder(encoding)\n",
    "        return outs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "235d6fcd",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "embed_size, num_hiddens, num_layers, devices = 100, 100, 2, d2l.try_all_gpus()\n",
    "net = BiRNN(len(vocab), embed_size, num_hiddens, num_layers)\n",
    "\n",
    "def init_weights(module):\n",
    "    if type(module) == nn.Linear:\n",
    "        nn.init.xavier_uniform_(module.weight)\n",
    "    if type(module) == nn.LSTM:\n",
    "        for param in module._flat_weights_names:\n",
    "            if \"weight\" in param:\n",
    "                nn.init.xavier_uniform_(module._parameters[param])\n",
    "net.apply(init_weights);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "801ac1b6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Loading Pretrained Word Vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "17ff5c83",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "glove_embedding = d2l.TokenEmbedding('glove.6b.100d')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "1c566bd1",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([49346, 100])"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embeds = glove_embedding[vocab.idx_to_token]\n",
    "embeds.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "9f03bbcc",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "net.embedding.weight.data.copy_(embeds)\n",
    "net.embedding.weight.requires_grad = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50ee083a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Training and Evaluating the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "c9cfc9b9",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.307, train acc 0.873, test acc 0.846\n",
      "1802.1 examples/sec on [device(type='cuda', index=0)]\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"235.784375pt\" height=\"184.455469pt\" viewBox=\"0 0 235.784375 184.455469\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-03-21T14:38:48.997340</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 184.455469 \n",
       "L 235.784375 184.455469 \n",
       "L 235.784375 -0 \n",
       "L 0 -0 \n",
       "L 0 184.455469 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 30.103125 146.899219 \n",
       "L 225.403125 146.899219 \n",
       "L 225.403125 10.999219 \n",
       "L 30.103125 10.999219 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 30.103125 146.899219 \n",
       "L 30.103125 10.999219 \n",
       "\" clip-path=\"url(#p5c80bb7786)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m155d9c6c5f\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m155d9c6c5f\" x=\"30.103125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 1 -->\n",
       "      <g transform=\"translate(26.921875 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 78.928125 146.899219 \n",
       "L 78.928125 10.999219 \n",
       "\" clip-path=\"url(#p5c80bb7786)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m155d9c6c5f\" x=\"78.928125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(75.746875 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 127.753125 146.899219 \n",
       "L 127.753125 10.999219 \n",
       "\" clip-path=\"url(#p5c80bb7786)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m155d9c6c5f\" x=\"127.753125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 3 -->\n",
       "      <g transform=\"translate(124.571875 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 176.578125 146.899219 \n",
       "L 176.578125 10.999219 \n",
       "\" clip-path=\"url(#p5c80bb7786)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m155d9c6c5f\" x=\"176.578125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(173.396875 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 225.403125 146.899219 \n",
       "L 225.403125 10.999219 \n",
       "\" clip-path=\"url(#p5c80bb7786)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m155d9c6c5f\" x=\"225.403125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(222.221875 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(112.525 175.175781)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 30.103125 146.899219 \n",
       "L 225.403125 146.899219 \n",
       "\" clip-path=\"url(#p5c80bb7786)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m4a35a9ab7f\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4a35a9ab7f\" x=\"30.103125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0.0 -->\n",
       "      <g transform=\"translate(7.2 150.698437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 30.103125 119.719219 \n",
       "L 225.403125 119.719219 \n",
       "\" clip-path=\"url(#p5c80bb7786)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4a35a9ab7f\" x=\"30.103125\" y=\"119.719219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.2 -->\n",
       "      <g transform=\"translate(7.2 123.518437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 30.103125 92.539219 \n",
       "L 225.403125 92.539219 \n",
       "\" clip-path=\"url(#p5c80bb7786)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4a35a9ab7f\" x=\"30.103125\" y=\"92.539219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.4 -->\n",
       "      <g transform=\"translate(7.2 96.338437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 30.103125 65.359219 \n",
       "L 225.403125 65.359219 \n",
       "\" clip-path=\"url(#p5c80bb7786)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4a35a9ab7f\" x=\"30.103125\" y=\"65.359219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.6 -->\n",
       "      <g transform=\"translate(7.2 69.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 30.103125 38.179219 \n",
       "L 225.403125 38.179219 \n",
       "\" clip-path=\"url(#p5c80bb7786)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4a35a9ab7f\" x=\"30.103125\" y=\"38.179219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.8 -->\n",
       "      <g transform=\"translate(7.2 41.978437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-38\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path d=\"M 30.103125 10.999219 \n",
       "L 225.403125 10.999219 \n",
       "\" clip-path=\"url(#p5c80bb7786)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4a35a9ab7f\" x=\"30.103125\" y=\"10.999219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 1.0 -->\n",
       "      <g transform=\"translate(7.2 14.798437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_23\">\n",
       "    <path d=\"M -1 55.882703 \n",
       "L 0.758176 56.766552 \n",
       "L 10.498202 62.819627 \n",
       "L 20.238227 69.071568 \n",
       "L 29.978253 73.28985 \n",
       "L 30.103125 73.332933 \n",
       "L 39.843151 93.839046 \n",
       "L 49.583176 91.9479 \n",
       "L 59.323202 92.578598 \n",
       "L 69.063227 92.922946 \n",
       "L 78.803253 93.643022 \n",
       "L 78.928125 93.649362 \n",
       "L 88.668151 100.867468 \n",
       "L 98.408176 100.105498 \n",
       "L 108.148202 100.166611 \n",
       "L 117.888227 100.973083 \n",
       "L 127.628253 101.306281 \n",
       "L 127.753125 101.304304 \n",
       "L 137.493151 104.155757 \n",
       "L 147.233176 105.01357 \n",
       "L 156.973202 104.938467 \n",
       "L 166.713227 104.970831 \n",
       "L 176.453253 104.934439 \n",
       "L 176.578125 104.93847 \n",
       "L 186.318151 104.744505 \n",
       "L 196.058176 104.612716 \n",
       "L 205.798202 104.682701 \n",
       "L 215.538227 105.023554 \n",
       "L 225.278253 105.245865 \n",
       "L 225.403125 105.240437 \n",
       "\" clip-path=\"url(#p5c80bb7786)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_24\">\n",
       "    <path d=\"M -1 67.772072 \n",
       "L 0.758176 66.317488 \n",
       "L 10.498202 58.821935 \n",
       "L 20.238227 53.440745 \n",
       "L 29.978253 49.901683 \n",
       "L 30.103125 49.872055 \n",
       "L 39.843151 34.629267 \n",
       "L 49.583176 35.432362 \n",
       "L 59.323202 34.847055 \n",
       "L 69.063227 34.493149 \n",
       "L 78.803253 34.112019 \n",
       "L 78.928125 34.102219 \n",
       "L 88.668151 30.055709 \n",
       "L 98.408176 30.777133 \n",
       "L 108.148202 30.836118 \n",
       "L 117.888227 30.287109 \n",
       "L 127.628253 30.082933 \n",
       "L 127.753125 30.068707 \n",
       "L 137.493151 27.877825 \n",
       "L 147.233176 28.136448 \n",
       "L 156.973202 28.195433 \n",
       "L 166.713227 28.279372 \n",
       "L 176.453253 28.454964 \n",
       "L 176.578125 28.465087 \n",
       "L 186.318151 28.830649 \n",
       "L 196.058176 28.953155 \n",
       "L 205.798202 28.876022 \n",
       "L 215.538227 28.483549 \n",
       "L 225.278253 28.24262 \n",
       "L 225.403125 28.236775 \n",
       "\" clip-path=\"url(#p5c80bb7786)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_25\">\n",
       "    <path d=\"M 30.103125 33.977191 \n",
       "L 78.928125 32.330083 \n",
       "L 127.753125 31.666891 \n",
       "L 176.578125 31.851715 \n",
       "L 225.403125 31.862587 \n",
       "\" clip-path=\"url(#p5c80bb7786)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 30.103125 146.899219 \n",
       "L 30.103125 10.999219 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 225.403125 146.899219 \n",
       "L 225.403125 10.999219 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 30.103125 146.899219 \n",
       "L 225.403125 146.899219 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 30.103125 10.999219 \n",
       "L 225.403125 10.999219 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 140.634375 102.466406 \n",
       "L 218.403125 102.466406 \n",
       "Q 220.403125 102.466406 220.403125 100.466406 \n",
       "L 220.403125 57.432031 \n",
       "Q 220.403125 55.432031 218.403125 55.432031 \n",
       "L 140.634375 55.432031 \n",
       "Q 138.634375 55.432031 138.634375 57.432031 \n",
       "L 138.634375 100.466406 \n",
       "Q 138.634375 102.466406 140.634375 102.466406 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_26\">\n",
       "     <path d=\"M 142.634375 63.530469 \n",
       "L 152.634375 63.530469 \n",
       "L 162.634375 63.530469 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- train loss -->\n",
       "     <g transform=\"translate(170.634375 67.030469)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"264.550781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"292.333984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"353.515625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"405.615234\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_27\">\n",
       "     <path d=\"M 142.634375 78.208594 \n",
       "L 152.634375 78.208594 \n",
       "L 162.634375 78.208594 \n",
       "\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- train acc -->\n",
       "     <g transform=\"translate(170.634375 81.708594)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"264.550781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"325.830078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"380.810547\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_28\">\n",
       "     <path d=\"M 142.634375 92.886719 \n",
       "L 152.634375 92.886719 \n",
       "L 162.634375 92.886719 \n",
       "\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_15\">\n",
       "     <!-- test acc -->\n",
       "     <g transform=\"translate(170.634375 96.386719)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"192.041016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"223.828125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"285.107422\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"340.087891\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p5c80bb7786\">\n",
       "   <rect x=\"30.103125\" y=\"10.999219\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr, num_epochs = 0.01, 5\n",
    "trainer = torch.optim.Adam(net.parameters(), lr=lr)\n",
    "loss = nn.CrossEntropyLoss(reduction=\"none\")\n",
    "d2l.train_ch13(net, train_iter, test_iter, loss, trainer, num_epochs, devices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "f060a564",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "def predict_sentiment(net, vocab, sequence):\n",
    "    \"\"\"Predict the sentiment of a text sequence.\"\"\"\n",
    "    sequence = torch.tensor(vocab[sequence.split()], device=d2l.try_gpu())\n",
    "    label = torch.argmax(net(sequence.reshape(1, -1)), dim=1)\n",
    "    return 'positive' if label == 1 else 'negative'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "12d9c3c7",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'positive'"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict_sentiment(net, vocab, 'this movie is so great')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "61b0071e",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'negative'"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict_sentiment(net, vocab, 'this movie is so bad')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d496c530",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Sentiment Analysis: Using CNN\n",
    "\n",
    "![nlp-map-sa-cnn.svg](./images/8/nlp-map-sa-cnn.svg)\n",
    "\n",
    "- Though originally designed for computer vision, CNNs are also widely used for natural language processing. Simply put, just think of any text sequence as a one-dimensional image. In this way, one-dimensional CNNs can process local features such as $n$-grams in text.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06d39b5f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![conv1d-channel.svg](./images/8/conv1d-2d.svg)\n",
    "\n",
    "- Two-dimensional cross-correlation operation with a single input channel. The shaded portions are the first output element as well as the input and kernel tensor elements used for the output computation:\n",
    "\n",
    "$$2\\times(-1)+3\\times(-3)+1\\times3+2\\times4+0\\times1+1\\times2=2$$\n",
    "\n",
    "- The *max-over-time pooling* used in textCNN works like the one-dimensional global max-pooling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "b43a5887",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "batch_size = 64\n",
    "train_iter, test_iter, vocab = d2l.load_data_imdb(batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "499bc988",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The textCNN Model\n",
    "\n",
    "![textcnn.svg](./images/8/textcnn.svg)\n",
    "\n",
    "- For a single text sequence with $n$ tokens represented by $d$-dimensional vectors, the width, height, and number of channels of the input tensor are $n$, $1$, and $d$, respectively.\n",
    "- The textCNN model transforms the input into the output as follows:\n",
    "\n",
    "1. Define multiple one-dimensional convolution kernels and perform convolution operations separately on the inputs. Convolution kernels with different widths may capture local features among different numbers of adjacent tokens.\n",
    "2. Perform max-over-time pooling on all the output channels, and then concatenate all the scalar pooling outputs as a vector.\n",
    "3. Transform the concatenated vector into the output categories using the fully connected layer. Dropout can be used for reducing overfitting.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "5d6bb153",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "class TextCNN(nn.Module):\n",
    "    def __init__(self, vocab_size, embed_size, kernel_sizes, num_channels,\n",
    "                 **kwargs):\n",
    "        super(TextCNN, self).__init__(**kwargs)\n",
    "        self.embedding = nn.Embedding(vocab_size, embed_size)\n",
    "        # The embedding layer not to be trained\n",
    "        self.constant_embedding = nn.Embedding(vocab_size, embed_size)\n",
    "        self.dropout = nn.Dropout(0.5)\n",
    "        self.decoder = nn.Linear(sum(num_channels), 2)\n",
    "        # The max-over-time pooling layer has no parameters, so this instance\n",
    "        # can be shared\n",
    "        self.pool = nn.AdaptiveAvgPool1d(1)\n",
    "        self.relu = nn.ReLU()\n",
    "        # Create multiple one-dimensional convolutional layers\n",
    "        self.convs = nn.ModuleList()\n",
    "        for c, k in zip(num_channels, kernel_sizes):\n",
    "            self.convs.append(nn.Conv1d(2 * embed_size, c, k))\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        # Concatenate two embedding layer outputs with shape (batch size, no.\n",
    "        # of tokens, token vector dimension) along vectors\n",
    "        embeddings = torch.cat((\n",
    "            self.embedding(inputs), self.constant_embedding(inputs)), dim=2)\n",
    "        # Per the input format of one-dimensional convolutional layers,\n",
    "        # rearrange the tensor so that the second dimension stores channels\n",
    "        embeddings = embeddings.permute(0, 2, 1)\n",
    "        # For each one-dimensional convolutional layer, after max-over-time\n",
    "        # pooling, a tensor of shape (batch size, no. of channels, 1) is\n",
    "        # obtained. Remove the last dimension and concatenate along channels\n",
    "        encoding = torch.cat([\n",
    "            torch.squeeze(self.relu(self.pool(conv(embeddings))), dim=-1)\n",
    "            for conv in self.convs], dim=1)\n",
    "        outputs = self.decoder(self.dropout(encoding))\n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d8a9ce2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Let’s create a textCNN instance. It has 3 convolutional layers with kernel widths of 3, 4, and 5, all with 100 output channels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "83ca5efb",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "embed_size, kernel_sizes, nums_channels = 100, [3, 4, 5], [100, 100, 100]\n",
    "devices = d2l.try_all_gpus()\n",
    "net = TextCNN(len(vocab), embed_size, kernel_sizes, nums_channels)\n",
    "\n",
    "def init_weights(module):\n",
    "    if type(module) in (nn.Linear, nn.Conv1d):\n",
    "        nn.init.xavier_uniform_(module.weight)\n",
    "\n",
    "net.apply(init_weights);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "315c5230",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Loading Pretrained Word Vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "5b9a285f",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "glove_embedding = d2l.TokenEmbedding('glove.6b.100d')\n",
    "embeds = glove_embedding[vocab.idx_to_token]\n",
    "net.embedding.weight.data.copy_(embeds)\n",
    "net.constant_embedding.weight.data.copy_(embeds)\n",
    "net.constant_embedding.weight.requires_grad = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3988f894",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Training and Evaluating the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "b9ceb104",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.065, train acc 0.979, test acc 0.869\n",
      "4350.3 examples/sec on [device(type='cuda', index=0)]\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"235.784375pt\" height=\"184.455469pt\" viewBox=\"0 0 235.784375 184.455469\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-03-21T14:39:51.237148</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 184.455469 \n",
       "L 235.784375 184.455469 \n",
       "L 235.784375 -0 \n",
       "L 0 -0 \n",
       "L 0 184.455469 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 30.103125 146.899219 \n",
       "L 225.403125 146.899219 \n",
       "L 225.403125 10.999219 \n",
       "L 30.103125 10.999219 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 30.103125 146.899219 \n",
       "L 30.103125 10.999219 \n",
       "\" clip-path=\"url(#p8361350b7c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m1b903d7b6f\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1b903d7b6f\" x=\"30.103125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 1 -->\n",
       "      <g transform=\"translate(26.921875 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 78.928125 146.899219 \n",
       "L 78.928125 10.999219 \n",
       "\" clip-path=\"url(#p8361350b7c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1b903d7b6f\" x=\"78.928125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(75.746875 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 127.753125 146.899219 \n",
       "L 127.753125 10.999219 \n",
       "\" clip-path=\"url(#p8361350b7c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1b903d7b6f\" x=\"127.753125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 3 -->\n",
       "      <g transform=\"translate(124.571875 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 176.578125 146.899219 \n",
       "L 176.578125 10.999219 \n",
       "\" clip-path=\"url(#p8361350b7c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1b903d7b6f\" x=\"176.578125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(173.396875 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 225.403125 146.899219 \n",
       "L 225.403125 10.999219 \n",
       "\" clip-path=\"url(#p8361350b7c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1b903d7b6f\" x=\"225.403125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(222.221875 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(112.525 175.175781)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 30.103125 146.899219 \n",
       "L 225.403125 146.899219 \n",
       "\" clip-path=\"url(#p8361350b7c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m1309c4871b\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1309c4871b\" x=\"30.103125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0.0 -->\n",
       "      <g transform=\"translate(7.2 150.698437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 30.103125 119.719219 \n",
       "L 225.403125 119.719219 \n",
       "\" clip-path=\"url(#p8361350b7c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1309c4871b\" x=\"30.103125\" y=\"119.719219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.2 -->\n",
       "      <g transform=\"translate(7.2 123.518437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 30.103125 92.539219 \n",
       "L 225.403125 92.539219 \n",
       "\" clip-path=\"url(#p8361350b7c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1309c4871b\" x=\"30.103125\" y=\"92.539219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.4 -->\n",
       "      <g transform=\"translate(7.2 96.338437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 30.103125 65.359219 \n",
       "L 225.403125 65.359219 \n",
       "\" clip-path=\"url(#p8361350b7c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1309c4871b\" x=\"30.103125\" y=\"65.359219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.6 -->\n",
       "      <g transform=\"translate(7.2 69.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 30.103125 38.179219 \n",
       "L 225.403125 38.179219 \n",
       "\" clip-path=\"url(#p8361350b7c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1309c4871b\" x=\"30.103125\" y=\"38.179219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.8 -->\n",
       "      <g transform=\"translate(7.2 41.978437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-38\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path d=\"M 30.103125 10.999219 \n",
       "L 225.403125 10.999219 \n",
       "\" clip-path=\"url(#p8361350b7c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1309c4871b\" x=\"30.103125\" y=\"10.999219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 1.0 -->\n",
       "      <g transform=\"translate(7.2 14.798437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_23\">\n",
       "    <path d=\"M -1 59.045645 \n",
       "L 0.758176 60.012281 \n",
       "L 10.498202 65.047203 \n",
       "L 20.238227 69.622787 \n",
       "L 29.978253 74.119002 \n",
       "L 30.103125 74.171528 \n",
       "L 39.843151 105.085306 \n",
       "L 49.583176 104.981379 \n",
       "L 59.323202 106.589108 \n",
       "L 69.063227 107.454127 \n",
       "L 78.803253 108.079874 \n",
       "L 78.928125 108.083602 \n",
       "L 88.668151 120.432491 \n",
       "L 98.408176 121.246449 \n",
       "L 108.148202 122.169414 \n",
       "L 117.888227 122.241683 \n",
       "L 127.628253 122.769633 \n",
       "L 127.753125 122.761288 \n",
       "L 137.493151 131.096518 \n",
       "L 147.233176 132.199018 \n",
       "L 156.973202 131.601313 \n",
       "L 166.713227 131.926955 \n",
       "L 176.453253 131.980293 \n",
       "L 176.578125 131.977134 \n",
       "L 186.318151 137.355049 \n",
       "L 196.058176 137.785872 \n",
       "L 205.798202 138.038002 \n",
       "L 215.538227 138.023052 \n",
       "L 225.278253 138.058295 \n",
       "L 225.403125 138.062493 \n",
       "\" clip-path=\"url(#p8361350b7c)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_24\">\n",
       "    <path d=\"M -1 61.483039 \n",
       "L 0.758176 59.974399 \n",
       "L 10.498202 54.457091 \n",
       "L 20.238227 50.874925 \n",
       "L 29.978253 47.544123 \n",
       "L 30.103125 47.512831 \n",
       "L 39.843151 28.612861 \n",
       "L 49.583176 28.109225 \n",
       "L 59.323202 27.433173 \n",
       "L 69.063227 27.204041 \n",
       "L 78.803253 26.848774 \n",
       "L 78.928125 26.850595 \n",
       "L 88.668151 20.7997 \n",
       "L 98.408176 20.282452 \n",
       "L 108.148202 20.091887 \n",
       "L 117.888227 20.098693 \n",
       "L 127.628253 19.93399 \n",
       "L 127.753125 19.941439 \n",
       "L 137.493151 16.580048 \n",
       "L 147.233176 16.076412 \n",
       "L 156.973202 16.471154 \n",
       "L 166.713227 16.280589 \n",
       "L 176.453253 16.302368 \n",
       "L 176.578125 16.299319 \n",
       "L 186.318151 13.776022 \n",
       "L 196.058176 13.626292 \n",
       "L 205.798202 13.685276 \n",
       "L 215.538227 13.741992 \n",
       "L 225.278253 13.786911 \n",
       "L 225.403125 13.787887 \n",
       "\" clip-path=\"url(#p8361350b7c)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_25\">\n",
       "    <path d=\"M 30.103125 32.574703 \n",
       "L 78.928125 28.138927 \n",
       "L 127.753125 28.486831 \n",
       "L 176.578125 27.877999 \n",
       "L 225.403125 28.769503 \n",
       "\" clip-path=\"url(#p8361350b7c)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 30.103125 146.899219 \n",
       "L 30.103125 10.999219 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 225.403125 146.899219 \n",
       "L 225.403125 10.999219 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 30.103125 146.899219 \n",
       "L 225.403125 146.899219 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 30.103125 10.999219 \n",
       "L 225.403125 10.999219 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 140.634375 102.466406 \n",
       "L 218.403125 102.466406 \n",
       "Q 220.403125 102.466406 220.403125 100.466406 \n",
       "L 220.403125 57.432031 \n",
       "Q 220.403125 55.432031 218.403125 55.432031 \n",
       "L 140.634375 55.432031 \n",
       "Q 138.634375 55.432031 138.634375 57.432031 \n",
       "L 138.634375 100.466406 \n",
       "Q 138.634375 102.466406 140.634375 102.466406 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_26\">\n",
       "     <path d=\"M 142.634375 63.530469 \n",
       "L 152.634375 63.530469 \n",
       "L 162.634375 63.530469 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- train loss -->\n",
       "     <g transform=\"translate(170.634375 67.030469)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"264.550781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"292.333984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"353.515625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"405.615234\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_27\">\n",
       "     <path d=\"M 142.634375 78.208594 \n",
       "L 152.634375 78.208594 \n",
       "L 162.634375 78.208594 \n",
       "\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- train acc -->\n",
       "     <g transform=\"translate(170.634375 81.708594)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"264.550781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"325.830078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"380.810547\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_28\">\n",
       "     <path d=\"M 142.634375 92.886719 \n",
       "L 152.634375 92.886719 \n",
       "L 162.634375 92.886719 \n",
       "\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_15\">\n",
       "     <!-- test acc -->\n",
       "     <g transform=\"translate(170.634375 96.386719)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"192.041016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"223.828125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"285.107422\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"340.087891\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p8361350b7c\">\n",
       "   <rect x=\"30.103125\" y=\"10.999219\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr, num_epochs = 0.001, 5\n",
    "trainer = torch.optim.Adam(net.parameters(), lr=lr)\n",
    "loss = nn.CrossEntropyLoss(reduction=\"none\")\n",
    "d2l.train_ch13(net, train_iter, test_iter, loss, trainer, num_epochs, devices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "4c6dadff",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'positive'"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d2l.predict_sentiment(net, vocab, 'this movie is so great')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "4ef7bbac",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'negative'"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d2l.predict_sentiment(net, vocab, 'this movie is so bad')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c665918",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Natural Language Inference\n",
    "\n",
    "- *Natural language inference* studies whether a *hypothesis* can be inferred from a *premise*, where both are a text sequence. In other words, natural language inference determines the logical relationship between a pair of text sequences. Such relationships usually fall into three types:\n",
    "    - Entailment: the hypothesis can be inferred from the premise.\n",
    "    - Contradiction: the negation of the hypothesis can be inferred from the premise.\n",
    "    - Neutral: all the other cases.\n",
    "- Stanford Natural Language Inference (SNLI) Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "52cefa4b",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "d2l.DATA_HUB['SNLI'] = (\n",
    "    'https://nlp.stanford.edu/projects/snli/snli_1.0.zip',\n",
    "    '9fcde07509c7e87ec61c640c1b2753d9041758e4')\n",
    "\n",
    "data_dir = d2l.download_extract('SNLI')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8d5286e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- **Entailment**\n",
    "\n",
    "> Premise: Two women are hugging each other. <br>\n",
    "Hypothesis: Two women are showing affection.\n",
    "\n",
    "- **Contradiction**\n",
    "\n",
    "> Premise: A man is running the coding example from Dive into Deep Learning. <br>\n",
    "Hypothesis: The man is sleeping.\n",
    "\n",
    "- **Neutral**\n",
    "\n",
    "> Premise: The musicians are performing for us.<br>\n",
    "Hypothesis: The musicians are famous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "07f8297b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "#@save\n",
    "def read_snli(data_dir, is_train):\n",
    "    \"\"\"Read the SNLI dataset into premises, hypotheses, and labels.\"\"\"\n",
    "    def extract_text(s):\n",
    "        # Remove information that will not be used by us\n",
    "        s = re.sub('\\\\(', '', s)\n",
    "        s = re.sub('\\\\)', '', s)\n",
    "        # Substitute two or more consecutive whitespace with space\n",
    "        s = re.sub('\\\\s{2,}', ' ', s)\n",
    "        return s.strip()\n",
    "    label_set = {'entailment': 0, 'contradiction': 1, 'neutral': 2}\n",
    "    file_name = os.path.join(data_dir, 'snli_1.0_train.txt'\n",
    "                             if is_train else 'snli_1.0_test.txt')\n",
    "    with open(file_name, 'r') as f:\n",
    "        rows = [row.split('\\t') for row in f.readlines()[1:]]\n",
    "    premises = [extract_text(row[1]) for row in rows if row[0] in label_set]\n",
    "    hypotheses = [extract_text(row[2]) for row in rows if row[0] in label_set]\n",
    "    labels = [label_set[row[0]] for row in rows if row[0] in label_set]\n",
    "    return premises, hypotheses, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "0f1c5880",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "premise: A person on a horse jumps over a broken down airplane .\n",
      "hypothesis: A person is training his horse for a competition .\n",
      "label: 2\n",
      "premise: A person on a horse jumps over a broken down airplane .\n",
      "hypothesis: A person is at a diner , ordering an omelette .\n",
      "label: 1\n",
      "premise: A person on a horse jumps over a broken down airplane .\n",
      "hypothesis: A person is outdoors , on a horse .\n",
      "label: 0\n"
     ]
    }
   ],
   "source": [
    "train_data = read_snli(data_dir, is_train=True)\n",
    "for x0, x1, y in zip(train_data[0][:3], train_data[1][:3], train_data[2][:3]):\n",
    "    print('premise:', x0)\n",
    "    print('hypothesis:', x1)\n",
    "    print('label:', y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "f6282319",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[183416, 183187, 182764]\n",
      "[3368, 3237, 3219]\n"
     ]
    }
   ],
   "source": [
    "# the three labels “entailment”, “contradiction”, and “neutral” \n",
    "# are balanced in both the training set and the testing set\n",
    "\n",
    "test_data = read_snli(data_dir, is_train=False)\n",
    "for data in [train_data, test_data]:\n",
    "    print([[row for row in data[2]].count(i) for i in range(3)])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4eefa340",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Natural Language Inference: Using Attention\n",
    "\n",
    "![nlp-map-nli-attention.svg](./images/8/nlp-map-nli-attention.svg)\n",
    "\n",
    "- Parikh et al. (2016) proposed to address natural language inference with attention mechanisms and called it a “decomposable attention model”. \n",
    "- This results in a model without recurrent or convolutional layers, achieving the best result at the time on the SNLI dataset with much fewer parameters. \n",
    "- In this section, we will describe and implement this attention-based method (with MLPs) for natural language inference\n",
    "\n",
    "[Parikh et al., 2016] Parikh, A. P., Täckström, O., Das, D., & Uszkoreit, J. (2016). A decomposable attention model for natural language inference. arXiv preprint arXiv:1606.01933.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dde98308",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "###  The Model\n",
    "\n",
    "![nli-attention.svg](./images/8/nli-attention.svg)\n",
    "\n",
    "- We just align tokens in one text sequence to every token in the other, and vice versa, then compare and aggregate such information to predict the logical relationships between premises and hypotheses. \n",
    "- Similar to alignment of tokens between source and target sentences in machine translation, the alignment of tokens between premises and hypotheses can be neatly accomplished by attention mechanisms.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4ac06b5",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Attending\n",
    "\n",
    "- The first step is to align (softly) tokens in one text sequence to each token in the other sequence. Denote by $\\mathbf{A} = (\\mathbf{a}_1, \\ldots, \\mathbf{a}_m)$ and $\\mathbf{B} = (\\mathbf{b}_1, \\ldots, \\mathbf{b}_n)$ the premise and hypothesis, whose number of tokens are $m$ and $n$, respectively, where $\\mathbf{a}_i, \\mathbf{b}_j \\in \\mathbb{R}^{d}$ ($i = 1, \\ldots, m, j = 1, \\ldots, n$) is a $d$-dimensional word vector. For alignment, we compute the attention weights $e_{ij} \\in \\mathbb{R}$ as\n",
    "\n",
    "$$e_{ij} = f(\\mathbf{a}_i)^\\top f(\\mathbf{b}_j),$$\n",
    "\n",
    "- where the function $f$ is an MLP defined in the following `mlp` function. The output dimension of $f$ is specified by the `num_hiddens` argument of `mlp`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "56baa1d6",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from torch.nn import functional as F\n",
    "\n",
    "def mlp(num_inputs, num_hiddens, flatten):\n",
    "    net = []\n",
    "    net.append(nn.Dropout(0.2))\n",
    "    net.append(nn.Linear(num_inputs, num_hiddens))\n",
    "    net.append(nn.ReLU())\n",
    "    if flatten:\n",
    "        net.append(nn.Flatten(start_dim=1))\n",
    "    net.append(nn.Dropout(0.2))\n",
    "    net.append(nn.Linear(num_hiddens, num_hiddens))\n",
    "    net.append(nn.ReLU())\n",
    "    if flatten:\n",
    "        net.append(nn.Flatten(start_dim=1))\n",
    "    return nn.Sequential(*net)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c4d884a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- ormalizing the attention weights, we compute the weighted average of all the token vectors in the hypothesis to obtain representation of the hypothesis that is softly aligned with the token indexed by $i$ in the premise:\n",
    "\n",
    "$$\n",
    "\\boldsymbol{\\beta}_i = \\sum_{j=1}^{n}\\frac{\\exp(e_{ij})}{ \\sum_{k=1}^{n} \\exp(e_{ik})} \\mathbf{b}_j.\n",
    "$$\n",
    "\n",
    "- Likewise, we compute soft alignment of premise tokens for each token indexed by $j$ in the hypothesis:\n",
    "\n",
    "$$\n",
    "\\boldsymbol{\\alpha}_j = \\sum_{i=1}^{m}\\frac{\\exp(e_{ij})}{ \\sum_{k=1}^{m} \\exp(e_{kj})} \\mathbf{a}_i.\n",
    "$$\n",
    "\n",
    "- Like the **Dynamic Programming** in Reinforcement Learning.\n",
    "- Below we define the `Attend` class to compute the soft alignment of hypotheses (`beta`) with input premises `A` and soft alignment of premises (`alpha`) with input hypotheses `B`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "b8729b29",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "class Attend(nn.Module):\n",
    "    def __init__(self, num_inputs, num_hiddens, **kwargs):\n",
    "        super(Attend, self).__init__(**kwargs)\n",
    "        self.f = mlp(num_inputs, num_hiddens, flatten=False)\n",
    "\n",
    "    def forward(self, A, B):\n",
    "        # Shape of `A`/`B`: (`batch_size`, no. of tokens in sequence A/B,\n",
    "        # `embed_size`)\n",
    "        # Shape of `f_A`/`f_B`: (`batch_size`, no. of tokens in sequence A/B,\n",
    "        # `num_hiddens`)\n",
    "        f_A = self.f(A)\n",
    "        f_B = self.f(B)\n",
    "        # Shape of `e`: (`batch_size`, no. of tokens in sequence A,\n",
    "        # no. of tokens in sequence B)\n",
    "        e = torch.bmm(f_A, f_B.permute(0, 2, 1))\n",
    "        # Shape of `beta`: (`batch_size`, no. of tokens in sequence A,\n",
    "        # `embed_size`), where sequence B is softly aligned with each token\n",
    "        # (axis 1 of `beta`) in sequence A\n",
    "        beta = torch.bmm(F.softmax(e, dim=-1), B)\n",
    "        # Shape of `alpha`: (`batch_size`, no. of tokens in sequence B,\n",
    "        # `embed_size`), where sequence A is softly aligned with each token\n",
    "        # (axis 1 of `alpha`) in sequence B\n",
    "        alpha = torch.bmm(F.softmax(e.permute(0, 2, 1), dim=-1), A)\n",
    "        return beta, alpha"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d2c6a4d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Comparing\n",
    "\n",
    "- In the comparing step, we feed the concatenation (operator $[\\cdot, \\cdot]$) of tokens from one sequence and aligned tokens from the other sequence into a function $g$ (an MLP):\n",
    "\n",
    "$$\\mathbf{v}_{A,i} = g([\\mathbf{a}_i, \\boldsymbol{\\beta}_i]), i = 1, \\ldots, m\\\\ \\mathbf{v}_{B,j} = g([\\mathbf{b}_j, \\boldsymbol{\\alpha}_j]), j = 1, \\ldots, n.$$\n",
    "\n",
    "- where $\\mathbf{v}_{A,i}$ is the comparison between token $i$ in the premise and all the hypothesis tokens that are softly aligned with token $i$; while $\\mathbf{v}_{B,j}$ is the comparison between token $j$ in the hypothesis and all the premise tokens that are softly aligned with token $j$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "b1f58372",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "class Compare(nn.Module):\n",
    "    def __init__(self, num_inputs, num_hiddens, **kwargs):\n",
    "        super(Compare, self).__init__(**kwargs)\n",
    "        self.g = mlp(num_inputs, num_hiddens, flatten=False)\n",
    "\n",
    "    def forward(self, A, B, beta, alpha):\n",
    "        V_A = self.g(torch.cat([A, beta], dim=2))\n",
    "        V_B = self.g(torch.cat([B, alpha], dim=2))\n",
    "        return V_A, V_B"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d43d94b7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Aggregating\n",
    "\n",
    "- With two sets of comparison vectors $\\mathbf{v}_{A,i}$ ($i = 1, \\ldots, m$) and $\\mathbf{v}_{B,j}$ ($j = 1, \\ldots, n$) on hand, in the last step we will aggregate such information to infer the logical relationship. We begin by summing up both sets:\n",
    "\n",
    "$$\n",
    "\\mathbf{v}_A = \\sum_{i=1}^{m} \\mathbf{v}_{A,i}, \\quad \\mathbf{v}_B = \\sum_{j=1}^{n}\\mathbf{v}_{B,j}.\n",
    "$$\n",
    "\n",
    "- Next we feed the concatenation of both summarization results into function $h$ (an MLP) to obtain the classification result of the logical relationship:\n",
    "\n",
    "$$\n",
    "\\hat{\\mathbf{y}} = h([\\mathbf{v}_A, \\mathbf{v}_B]).\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "cd48d2a1",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "class Aggregate(nn.Module):\n",
    "    def __init__(self, num_inputs, num_hiddens, num_outputs, **kwargs):\n",
    "        super(Aggregate, self).__init__(**kwargs)\n",
    "        self.h = mlp(num_inputs, num_hiddens, flatten=True)\n",
    "        self.linear = nn.Linear(num_hiddens, num_outputs)\n",
    "\n",
    "    def forward(self, V_A, V_B):\n",
    "        # Sum up both sets of comparison vectors\n",
    "        V_A = V_A.sum(dim=1)\n",
    "        V_B = V_B.sum(dim=1)\n",
    "        # Feed the concatenation of both summarization results into an MLP\n",
    "        Y_hat = self.linear(self.h(torch.cat([V_A, V_B], dim=1)))\n",
    "        return Y_hat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "004c4a50",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Putting It All Together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "5bb00d55",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "class DecomposableAttention(nn.Module):\n",
    "    def __init__(self, vocab, embed_size, num_hiddens, num_inputs_attend=100,\n",
    "                 num_inputs_compare=200, num_inputs_agg=400, **kwargs):\n",
    "        super(DecomposableAttention, self).__init__(**kwargs)\n",
    "        self.embedding = nn.Embedding(len(vocab), embed_size)\n",
    "        self.attend = Attend(num_inputs_attend, num_hiddens)\n",
    "        self.compare = Compare(num_inputs_compare, num_hiddens)\n",
    "        # There are 3 possible outputs: entailment, contradiction, and neutral\n",
    "        self.aggregate = Aggregate(num_inputs_agg, num_hiddens, num_outputs=3)\n",
    "\n",
    "    def forward(self, X):\n",
    "        premises, hypotheses = X\n",
    "        A = self.embedding(premises)\n",
    "        B = self.embedding(hypotheses)\n",
    "        beta, alpha = self.attend(A, B)\n",
    "        V_A, V_B = self.compare(A, B, beta, alpha)\n",
    "        Y_hat = self.aggregate(V_A, V_B)\n",
    "        return Y_hat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30a2275d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Training and Evaluating the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "de598d67",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "read 549367 examples\n",
      "read 9824 examples\n"
     ]
    }
   ],
   "source": [
    "batch_size, num_steps = 256, 50\n",
    "train_iter, test_iter, vocab = d2l.load_data_snli(batch_size, num_steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "12759a79",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "embed_size, num_hiddens, devices = 100, 200, d2l.try_all_gpus()\n",
    "net = DecomposableAttention(vocab, embed_size, num_hiddens)\n",
    "glove_embedding = d2l.TokenEmbedding('glove.6b.100d')\n",
    "embeds = glove_embedding[vocab.idx_to_token]\n",
    "net.embedding.weight.data.copy_(embeds);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "73b7bc74",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "def split_batch_multi_inputs(X, y, devices):\n",
    "    \"\"\"Split multi-input `X` and `y` into multiple devices.\"\"\"\n",
    "    X = list(zip(*[gluon.utils.split_and_load(\n",
    "        feature, devices, even_split=False) for feature in X]))\n",
    "    return (X, gluon.utils.split_and_load(y, devices, even_split=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "4c8e40f9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.493, train acc 0.806, test acc 0.821\n",
      "21352.3 examples/sec on [device(type='cuda', index=0)]\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"240.554687pt\" height=\"184.455469pt\" viewBox=\"0 0 240.554687 184.455469\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-03-21T14:42:22.706606</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 184.455469 \n",
       "L 240.554687 184.455469 \n",
       "L 240.554687 -0 \n",
       "L 0 -0 \n",
       "L 0 184.455469 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 30.103125 146.899219 \n",
       "L 225.403125 146.899219 \n",
       "L 225.403125 10.999219 \n",
       "L 30.103125 10.999219 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 30.103125 146.899219 \n",
       "L 30.103125 10.999219 \n",
       "\" clip-path=\"url(#p4447eea8fa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"me1340d781b\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#me1340d781b\" x=\"30.103125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 1.0 -->\n",
       "      <g transform=\"translate(22.151563 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 62.653125 146.899219 \n",
       "L 62.653125 10.999219 \n",
       "\" clip-path=\"url(#p4447eea8fa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me1340d781b\" x=\"62.653125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 1.5 -->\n",
       "      <g transform=\"translate(54.701563 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 95.203125 146.899219 \n",
       "L 95.203125 10.999219 \n",
       "\" clip-path=\"url(#p4447eea8fa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me1340d781b\" x=\"95.203125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 2.0 -->\n",
       "      <g transform=\"translate(87.251563 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 127.753125 146.899219 \n",
       "L 127.753125 10.999219 \n",
       "\" clip-path=\"url(#p4447eea8fa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me1340d781b\" x=\"127.753125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 2.5 -->\n",
       "      <g transform=\"translate(119.801563 161.497656)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 160.303125 146.899219 \n",
       "L 160.303125 10.999219 \n",
       "\" clip-path=\"url(#p4447eea8fa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me1340d781b\" x=\"160.303125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 3.0 -->\n",
       "      <g transform=\"translate(152.351562 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 192.853125 146.899219 \n",
       "L 192.853125 10.999219 \n",
       "\" clip-path=\"url(#p4447eea8fa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me1340d781b\" x=\"192.853125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 3.5 -->\n",
       "      <g transform=\"translate(184.901562 161.497656)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 225.403125 146.899219 \n",
       "L 225.403125 10.999219 \n",
       "\" clip-path=\"url(#p4447eea8fa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me1340d781b\" x=\"225.403125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 4.0 -->\n",
       "      <g transform=\"translate(217.451562 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_8\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(112.525 175.175781)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 30.103125 146.899219 \n",
       "L 225.403125 146.899219 \n",
       "\" clip-path=\"url(#p4447eea8fa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <defs>\n",
       "       <path id=\"m1359be594f\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1359be594f\" x=\"30.103125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.0 -->\n",
       "      <g transform=\"translate(7.2 150.698437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 30.103125 119.719219 \n",
       "L 225.403125 119.719219 \n",
       "\" clip-path=\"url(#p4447eea8fa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1359be594f\" x=\"30.103125\" y=\"119.719219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.2 -->\n",
       "      <g transform=\"translate(7.2 123.518437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 30.103125 92.539219 \n",
       "L 225.403125 92.539219 \n",
       "\" clip-path=\"url(#p4447eea8fa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1359be594f\" x=\"30.103125\" y=\"92.539219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.4 -->\n",
       "      <g transform=\"translate(7.2 96.338437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path d=\"M 30.103125 65.359219 \n",
       "L 225.403125 65.359219 \n",
       "\" clip-path=\"url(#p4447eea8fa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1359be594f\" x=\"30.103125\" y=\"65.359219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0.6 -->\n",
       "      <g transform=\"translate(7.2 69.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_23\">\n",
       "      <path d=\"M 30.103125 38.179219 \n",
       "L 225.403125 38.179219 \n",
       "\" clip-path=\"url(#p4447eea8fa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_24\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1359be594f\" x=\"30.103125\" y=\"38.179219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 0.8 -->\n",
       "      <g transform=\"translate(7.2 41.978437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-38\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_25\">\n",
       "      <path d=\"M 30.103125 10.999219 \n",
       "L 225.403125 10.999219 \n",
       "\" clip-path=\"url(#p4447eea8fa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_26\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1359be594f\" x=\"30.103125\" y=\"10.999219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 1.0 -->\n",
       "      <g transform=\"translate(7.2 14.798437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_27\">\n",
       "    <path d=\"M -1 38.259182 \n",
       "L 4.044924 41.114325 \n",
       "L 17.058857 45.982008 \n",
       "L 30.072789 49.464411 \n",
       "L 30.103125 49.472724 \n",
       "L 43.117058 68.510212 \n",
       "L 56.130991 69.046749 \n",
       "L 69.144924 69.48945 \n",
       "L 82.158857 70.019546 \n",
       "L 95.172789 70.527288 \n",
       "L 95.203125 70.530181 \n",
       "L 108.217058 75.773083 \n",
       "L 121.230991 75.831242 \n",
       "L 134.244924 76.141247 \n",
       "L 147.258857 76.278472 \n",
       "L 160.272789 76.397233 \n",
       "L 160.303125 76.396046 \n",
       "L 173.317058 79.649125 \n",
       "L 186.330991 79.946869 \n",
       "L 199.344924 79.755417 \n",
       "L 212.358857 79.907432 \n",
       "L 225.372789 79.896037 \n",
       "L 225.403125 79.895621 \n",
       "\" clip-path=\"url(#p4447eea8fa)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_28\">\n",
       "    <path d=\"M -1 60.229284 \n",
       "L 4.044924 58.355422 \n",
       "L 17.058857 55.356603 \n",
       "L 30.072789 53.258347 \n",
       "L 30.103125 53.2532 \n",
       "L 43.117058 42.345908 \n",
       "L 56.130991 42.255575 \n",
       "L 69.144924 42.048924 \n",
       "L 82.158857 41.810099 \n",
       "L 95.172789 41.577214 \n",
       "L 95.203125 41.575834 \n",
       "L 108.217058 39.264696 \n",
       "L 121.230991 39.140953 \n",
       "L 134.244924 38.991223 \n",
       "L 147.258857 38.941107 \n",
       "L 160.272789 38.85857 \n",
       "L 160.303125 38.857918 \n",
       "L 173.317058 37.570648 \n",
       "L 186.330991 37.425869 \n",
       "L 199.344924 37.428344 \n",
       "L 212.358857 37.339867 \n",
       "L 225.372789 37.344693 \n",
       "L 225.403125 37.346948 \n",
       "\" clip-path=\"url(#p4447eea8fa)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_29\">\n",
       "    <path d=\"M 30.103125 39.966503 \n",
       "L 95.203125 36.203799 \n",
       "L 160.303125 35.456792 \n",
       "L 225.403125 35.346124 \n",
       "\" clip-path=\"url(#p4447eea8fa)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 30.103125 146.899219 \n",
       "L 30.103125 10.999219 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 225.403125 146.899219 \n",
       "L 225.403125 10.999219 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 30.103125 146.899219 \n",
       "L 225.403125 146.899219 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 30.103125 10.999219 \n",
       "L 225.403125 10.999219 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 37.103125 141.899219 \n",
       "L 114.871875 141.899219 \n",
       "Q 116.871875 141.899219 116.871875 139.899219 \n",
       "L 116.871875 96.864844 \n",
       "Q 116.871875 94.864844 114.871875 94.864844 \n",
       "L 37.103125 94.864844 \n",
       "Q 35.103125 94.864844 35.103125 96.864844 \n",
       "L 35.103125 139.899219 \n",
       "Q 35.103125 141.899219 37.103125 141.899219 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_30\">\n",
       "     <path d=\"M 39.103125 102.963281 \n",
       "L 49.103125 102.963281 \n",
       "L 59.103125 102.963281 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_15\">\n",
       "     <!-- train loss -->\n",
       "     <g transform=\"translate(67.103125 106.463281)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"264.550781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"292.333984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"353.515625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"405.615234\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_31\">\n",
       "     <path d=\"M 39.103125 117.641406 \n",
       "L 49.103125 117.641406 \n",
       "L 59.103125 117.641406 \n",
       "\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_16\">\n",
       "     <!-- train acc -->\n",
       "     <g transform=\"translate(67.103125 121.141406)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"264.550781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"325.830078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"380.810547\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_32\">\n",
       "     <path d=\"M 39.103125 132.319531 \n",
       "L 49.103125 132.319531 \n",
       "L 59.103125 132.319531 \n",
       "\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_17\">\n",
       "     <!-- test acc -->\n",
       "     <g transform=\"translate(67.103125 135.819531)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"192.041016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"223.828125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"285.107422\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"340.087891\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p4447eea8fa\">\n",
       "   <rect x=\"30.103125\" y=\"10.999219\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr, num_epochs = 0.001, 4\n",
    "trainer = torch.optim.Adam(net.parameters(), lr=lr)\n",
    "loss = nn.CrossEntropyLoss(reduction=\"none\")\n",
    "d2l.train_ch13(net, train_iter, test_iter, loss, trainer, num_epochs, devices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "702e57f0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "def predict_snli(net, vocab, premise, hypothesis):\n",
    "    \"\"\"Predict the logical relationship between the premise and hypothesis.\"\"\"\n",
    "    net.eval()\n",
    "    premise = torch.tensor(vocab[premise], device=d2l.try_gpu())\n",
    "    hypothesis = torch.tensor(vocab[hypothesis], device=d2l.try_gpu())\n",
    "    label = torch.argmax(net([premise.reshape((1, -1)),\n",
    "                           hypothesis.reshape((1, -1))]), dim=1)\n",
    "    return 'entailment' if label == 0 else 'contradiction' if label == 1 else 'neutral'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "9d33dae6",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'contradiction'"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict_snli(net, vocab, ['he', 'is', 'good', '.'], ['he', 'is', 'bad', '.'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dde364e9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Natural Language Inference: Fine-Tuning BERT\n",
    "\n",
    "![nlp-map-nli-bert.svg](./images/8/nlp-map-nli-bert.svg)\n",
    "\n",
    "- Now we revisit this task by fine-tuning BERT, which *only* requires an additional MLP-based architecture.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9bf8bc22",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import multiprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf80b953",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Loading Pretrained BERT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f9000397",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "d2l.DATA_HUB['bert.base'] = (d2l.DATA_URL + 'bert.base.torch.zip',\n",
    "                             '225d66f04cae318b841a13d32af3acc165f253ac')\n",
    "d2l.DATA_HUB['bert.small'] = (d2l.DATA_URL + 'bert.small.torch.zip',\n",
    "                              'c72329e68a732bef0452e4b96a1c341c8910f81f')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cf5f3bc",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Either pretrained BERT model contains a “vocab.json” file that defines the vocabulary set and a “pretrained.params” file of the pretrained parameters. \n",
    "- We implement the following `load_pretrained_model` function to load pretrained BERT parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "489cca73",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def load_pretrained_model(pretrained_model, num_hiddens, ffn_num_hiddens,\n",
    "                          num_heads, num_blks, dropout, max_len, devices):\n",
    "    data_dir = d2l.download_extract(pretrained_model)\n",
    "    # Define an empty vocabulary to load the predefined vocabulary\n",
    "    vocab = d2l.Vocab()\n",
    "    vocab.idx_to_token = json.load(open(os.path.join(data_dir, 'vocab.json')))\n",
    "    vocab.token_to_idx = {token: idx for idx, token in enumerate(vocab.idx_to_token)}\n",
    "    bert = d2l.BERTModel(\n",
    "        len(vocab), num_hiddens, ffn_num_hiddens=ffn_num_hiddens, num_heads=4,\n",
    "        num_blks=2, dropout=0.2, max_len=max_len)\n",
    "    # Load pretrained BERT parameters\n",
    "    bert.load_state_dict(torch.load(os.path.join(data_dir,\n",
    "                                                 'pretrained.params')))\n",
    "    return bert, vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "90d226ff",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/haodong/Applications/Miniconda/Miniconda/envs/dl/lib/python3.9/site-packages/torch/nn/modules/lazy.py:178: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.\n",
      "  warnings.warn('Lazy modules are a new feature under heavy development '\n"
     ]
    }
   ],
   "source": [
    "devices = d2l.try_all_gpus()\n",
    "bert, vocab = load_pretrained_model(\n",
    "    'bert.small', num_hiddens=256, ffn_num_hiddens=512, num_heads=4,\n",
    "    num_blks=2, dropout=0.1, max_len=512, devices=devices)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "849ff7f2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "eee6dd0a",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "class SNLIBERTDataset(torch.utils.data.Dataset):\n",
    "    def __init__(self, dataset, max_len, vocab=None):\n",
    "        all_premise_hypothesis_tokens = [[\n",
    "            p_tokens, h_tokens] for p_tokens, h_tokens in zip(\n",
    "            *[d2l.tokenize([s.lower() for s in sentences])\n",
    "              for sentences in dataset[:2]])]\n",
    "\n",
    "        self.labels = torch.tensor(dataset[2])\n",
    "        self.vocab = vocab\n",
    "        self.max_len = max_len\n",
    "        (self.all_token_ids, self.all_segments,\n",
    "         self.valid_lens) = self._preprocess(all_premise_hypothesis_tokens)\n",
    "        print('read ' + str(len(self.all_token_ids)) + ' examples')\n",
    "\n",
    "    def _preprocess(self, all_premise_hypothesis_tokens):\n",
    "        pool = multiprocessing.Pool(4)  # Use 4 worker processes\n",
    "        out = pool.map(self._mp_worker, all_premise_hypothesis_tokens)\n",
    "        all_token_ids = [\n",
    "            token_ids for token_ids, segments, valid_len in out]\n",
    "        all_segments = [segments for token_ids, segments, valid_len in out]\n",
    "        valid_lens = [valid_len for token_ids, segments, valid_len in out]\n",
    "        return (torch.tensor(all_token_ids, dtype=torch.long),\n",
    "                torch.tensor(all_segments, dtype=torch.long),\n",
    "                torch.tensor(valid_lens))\n",
    "\n",
    "    def _mp_worker(self, premise_hypothesis_tokens):\n",
    "        p_tokens, h_tokens = premise_hypothesis_tokens\n",
    "        self._truncate_pair_of_tokens(p_tokens, h_tokens)\n",
    "        tokens, segments = d2l.get_tokens_and_segments(p_tokens, h_tokens)\n",
    "        token_ids = self.vocab[tokens] + [self.vocab['<pad>']] \\\n",
    "                             * (self.max_len - len(tokens))\n",
    "        segments = segments + [0] * (self.max_len - len(segments))\n",
    "        valid_len = len(tokens)\n",
    "        return token_ids, segments, valid_len\n",
    "\n",
    "    def _truncate_pair_of_tokens(self, p_tokens, h_tokens):\n",
    "        # Reserve slots for '<CLS>', '<SEP>', and '<SEP>' tokens for the BERT\n",
    "        # input\n",
    "        while len(p_tokens) + len(h_tokens) > self.max_len - 3:\n",
    "            if len(p_tokens) > len(h_tokens):\n",
    "                p_tokens.pop()\n",
    "            else:\n",
    "                h_tokens.pop()\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        return (self.all_token_ids[idx], self.all_segments[idx],\n",
    "                self.valid_lens[idx]), self.labels[idx]\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.all_token_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "44a5dfd5",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "read 549367 examples\n",
      "read 9824 examples\n"
     ]
    }
   ],
   "source": [
    "# Reduce `batch_size` if there is an out of memory error. In the original BERT\n",
    "# model, `max_len` = 512\n",
    "batch_size, max_len, num_workers = 512, 128, d2l.get_dataloader_workers()\n",
    "data_dir = d2l.download_extract('SNLI')\n",
    "train_set = SNLIBERTDataset(d2l.read_snli(data_dir, True), max_len, vocab)\n",
    "test_set = SNLIBERTDataset(d2l.read_snli(data_dir, False), max_len, vocab)\n",
    "train_iter = torch.utils.data.DataLoader(train_set, batch_size, shuffle=True, num_workers=num_workers)\n",
    "test_iter = torch.utils.data.DataLoader(test_set, batch_size, num_workers=num_workers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b8a2de9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Fine-Tuning BERT\n",
    "\n",
    "- Fine-tuning BERT for natural language inference requires only an extra MLP consisting of two fully connected layers (see `self.hidden` and `self.output` in the following `BERTClassifier` class). \n",
    "- This MLP transforms the BERT representation of the special “\\<cls\\>” token, which encodes the information of both the premise and the hypothesis, into three outputs of natural language inference: entailment, contradiction, and neutral."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0c1e346c",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "class BERTClassifier(nn.Module):\n",
    "    def __init__(self, bert):\n",
    "        super(BERTClassifier, self).__init__()\n",
    "        self.encoder = bert.encoder\n",
    "        self.hidden = bert.hidden\n",
    "        self.output = nn.LazyLinear(3)\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        tokens_X, segments_X, valid_lens_x = inputs\n",
    "        encoded_X = self.encoder(tokens_X, segments_X, valid_lens_x)\n",
    "        return self.output(self.hidden(encoded_X[:, 0, :]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bc81c13",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- In the following, the pretrained BERT model bert is fed into the `BERTClassifier` instance net for the downstream application. \n",
    "- In common implementations of BERT fine-tuning, only the parameters of the output layer of the additional MLP (`net.output`) will be learned from scratch. All the parameters of the pretrained BERT encoder (`net.encoder`) and the hidden layer of the additional MLP (`net.hidden`) will be fine-tuned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "06e121b3",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "net = BERTClassifier(bert)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c503b26",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- both the `MaskLM` class and the `NextSentencePred` class have parameters in their employed MLPs.\n",
    "- These parameters are part of those in the pretrained BERT model `bert`, and thus part of parameters in `net`.\n",
    "- However, these two parts are irrelevant to fine-tuning downstream applications, thus the parameters of the employed MLPs in `MaskLM` and `NextSentencePred` are not updated (staled) when BERT is fine-tuned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b8da7976",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.423, train acc 0.834, test acc 0.794\n",
      "3136.9 examples/sec on [device(type='cuda', index=0)]\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"238.965625pt\" height=\"184.455469pt\" viewBox=\"0 0 238.965625 184.455469\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-03-21T15:18:21.483134</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 184.455469 \n",
       "L 238.965625 184.455469 \n",
       "L 238.965625 -0 \n",
       "L 0 -0 \n",
       "L 0 184.455469 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 30.103125 146.899219 \n",
       "L 225.403125 146.899219 \n",
       "L 225.403125 10.999219 \n",
       "L 30.103125 10.999219 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 51.803125 146.899219 \n",
       "L 51.803125 10.999219 \n",
       "\" clip-path=\"url(#pd8061e30b9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"mc801ea8f72\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc801ea8f72\" x=\"51.803125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(48.621875 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 95.203125 146.899219 \n",
       "L 95.203125 10.999219 \n",
       "\" clip-path=\"url(#pd8061e30b9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc801ea8f72\" x=\"95.203125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(92.021875 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 138.603125 146.899219 \n",
       "L 138.603125 10.999219 \n",
       "\" clip-path=\"url(#pd8061e30b9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc801ea8f72\" x=\"138.603125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 6 -->\n",
       "      <g transform=\"translate(135.421875 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 182.003125 146.899219 \n",
       "L 182.003125 10.999219 \n",
       "\" clip-path=\"url(#pd8061e30b9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc801ea8f72\" x=\"182.003125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 8 -->\n",
       "      <g transform=\"translate(178.821875 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 225.403125 146.899219 \n",
       "L 225.403125 10.999219 \n",
       "\" clip-path=\"url(#pd8061e30b9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc801ea8f72\" x=\"225.403125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(219.040625 161.497656)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(112.525 175.175781)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 30.103125 146.899219 \n",
       "L 225.403125 146.899219 \n",
       "\" clip-path=\"url(#pd8061e30b9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m6de12a1cc3\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6de12a1cc3\" x=\"30.103125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0.0 -->\n",
       "      <g transform=\"translate(7.2 150.698437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 30.103125 119.719219 \n",
       "L 225.403125 119.719219 \n",
       "\" clip-path=\"url(#pd8061e30b9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6de12a1cc3\" x=\"30.103125\" y=\"119.719219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.2 -->\n",
       "      <g transform=\"translate(7.2 123.518437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 30.103125 92.539219 \n",
       "L 225.403125 92.539219 \n",
       "\" clip-path=\"url(#pd8061e30b9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6de12a1cc3\" x=\"30.103125\" y=\"92.539219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.4 -->\n",
       "      <g transform=\"translate(7.2 96.338437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 30.103125 65.359219 \n",
       "L 225.403125 65.359219 \n",
       "\" clip-path=\"url(#pd8061e30b9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6de12a1cc3\" x=\"30.103125\" y=\"65.359219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.6 -->\n",
       "      <g transform=\"translate(7.2 69.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 30.103125 38.179219 \n",
       "L 225.403125 38.179219 \n",
       "\" clip-path=\"url(#pd8061e30b9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6de12a1cc3\" x=\"30.103125\" y=\"38.179219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.8 -->\n",
       "      <g transform=\"translate(7.2 41.978437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-38\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path d=\"M 30.103125 10.999219 \n",
       "L 225.403125 10.999219 \n",
       "\" clip-path=\"url(#pd8061e30b9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6de12a1cc3\" x=\"30.103125\" y=\"10.999219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 1.0 -->\n",
       "      <g transform=\"translate(7.2 14.798437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_23\">\n",
       "    <path d=\"M 12.730991 18.278113 \n",
       "L 17.058857 27.874792 \n",
       "L 21.386722 33.06892 \n",
       "L 25.714588 36.569774 \n",
       "L 30.042454 39.123384 \n",
       "L 30.103125 39.173402 \n",
       "L 34.430991 55.370049 \n",
       "L 38.758857 55.858531 \n",
       "L 43.086722 56.36867 \n",
       "L 47.414588 56.77183 \n",
       "L 51.742454 57.232664 \n",
       "L 51.803125 57.246754 \n",
       "L 56.130991 65.211462 \n",
       "L 60.458857 65.153042 \n",
       "L 64.786722 65.182951 \n",
       "L 69.114588 65.341736 \n",
       "L 73.442454 65.47796 \n",
       "L 73.503125 65.491433 \n",
       "L 77.830991 71.945667 \n",
       "L 82.158857 71.609222 \n",
       "L 86.486722 71.413016 \n",
       "L 90.814588 71.391004 \n",
       "L 95.142454 71.41412 \n",
       "L 95.203125 71.400731 \n",
       "L 99.530991 76.731239 \n",
       "L 103.858857 76.418073 \n",
       "L 108.186722 76.123496 \n",
       "L 112.514588 76.019289 \n",
       "L 116.842454 75.870654 \n",
       "L 116.903125 75.864347 \n",
       "L 121.230991 80.770389 \n",
       "L 125.558857 80.329106 \n",
       "L 129.886722 79.905537 \n",
       "L 134.214588 79.65336 \n",
       "L 138.542454 79.325569 \n",
       "L 138.603125 79.331088 \n",
       "L 142.930991 83.672621 \n",
       "L 147.258857 83.156227 \n",
       "L 151.586722 82.835146 \n",
       "L 155.914588 82.680628 \n",
       "L 160.242454 82.522763 \n",
       "L 160.303125 82.521973 \n",
       "L 164.630991 86.317631 \n",
       "L 168.958857 86.041985 \n",
       "L 173.286722 85.782113 \n",
       "L 177.614588 85.271961 \n",
       "L 181.942454 85.023055 \n",
       "L 182.003125 85.02311 \n",
       "L 186.330991 88.986035 \n",
       "L 190.658857 88.599499 \n",
       "L 194.986722 88.270113 \n",
       "L 199.314588 87.742605 \n",
       "L 203.642454 87.404139 \n",
       "L 203.703125 87.393602 \n",
       "L 208.030991 91.08853 \n",
       "L 212.358857 90.482077 \n",
       "L 216.686722 90.105841 \n",
       "L 221.014588 89.701298 \n",
       "L 225.342454 89.433545 \n",
       "L 225.403125 89.431808 \n",
       "\" clip-path=\"url(#pd8061e30b9)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_24\">\n",
       "    <path d=\"M 12.730991 73.329303 \n",
       "L 17.058857 66.519295 \n",
       "L 21.386722 63.12349 \n",
       "L 25.714588 60.90155 \n",
       "L 30.042454 59.306182 \n",
       "L 30.103125 59.278776 \n",
       "L 34.430991 49.742031 \n",
       "L 38.758857 49.455515 \n",
       "L 43.086722 49.150809 \n",
       "L 47.414588 48.934579 \n",
       "L 51.742454 48.659226 \n",
       "L 51.803125 48.652756 \n",
       "L 56.130991 44.496693 \n",
       "L 60.458857 44.551888 \n",
       "L 64.786722 44.554989 \n",
       "L 69.114588 44.541345 \n",
       "L 73.442454 44.540601 \n",
       "L 73.503125 44.533457 \n",
       "L 77.830991 41.447973 \n",
       "L 82.158857 41.629681 \n",
       "L 86.486722 41.773765 \n",
       "L 90.814588 41.800225 \n",
       "L 95.142454 41.732256 \n",
       "L 95.203125 41.736629 \n",
       "L 99.530991 39.268721 \n",
       "L 103.858857 39.432444 \n",
       "L 108.186722 39.561851 \n",
       "L 112.514588 39.610431 \n",
       "L 116.842454 39.655702 \n",
       "L 116.903125 39.658921 \n",
       "L 121.230991 37.393348 \n",
       "L 125.558857 37.573196 \n",
       "L 129.886722 37.725756 \n",
       "L 134.214588 37.867773 \n",
       "L 138.542454 38.060644 \n",
       "L 138.603125 38.056916 \n",
       "L 142.930991 35.993021 \n",
       "L 147.258857 36.239846 \n",
       "L 151.586722 36.436644 \n",
       "L 155.914588 36.507756 \n",
       "L 160.242454 36.590362 \n",
       "L 160.303125 36.590721 \n",
       "L 164.630991 34.739052 \n",
       "L 168.958857 34.928821 \n",
       "L 173.286722 35.090477 \n",
       "L 177.614588 35.309911 \n",
       "L 181.942454 35.475061 \n",
       "L 182.003125 35.474067 \n",
       "L 186.330991 33.532215 \n",
       "L 190.658857 33.751132 \n",
       "L 194.986722 33.967156 \n",
       "L 199.314588 34.188657 \n",
       "L 203.642454 34.341155 \n",
       "L 203.703125 34.349003 \n",
       "L 208.030991 32.734685 \n",
       "L 212.358857 32.922595 \n",
       "L 216.686722 33.122907 \n",
       "L 221.014588 33.354848 \n",
       "L 225.342454 33.52899 \n",
       "L 225.403125 33.531179 \n",
       "\" clip-path=\"url(#pd8061e30b9)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_25\">\n",
       "    <path d=\"M 30.103125 49.138093 \n",
       "L 51.803125 45.41689 \n",
       "L 73.503125 43.341869 \n",
       "L 95.203125 42.34586 \n",
       "L 116.903125 41.612686 \n",
       "L 138.603125 40.796511 \n",
       "L 160.303125 39.828168 \n",
       "L 182.003125 39.828168 \n",
       "L 203.703125 39.31633 \n",
       "L 225.403125 38.970493 \n",
       "\" clip-path=\"url(#pd8061e30b9)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 30.103125 146.899219 \n",
       "L 30.103125 10.999219 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 225.403125 146.899219 \n",
       "L 225.403125 10.999219 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 30.103125 146.899219 \n",
       "L 225.403125 146.899219 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 30.103125 10.999219 \n",
       "L 225.403125 10.999219 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 37.103125 141.899219 \n",
       "L 114.871875 141.899219 \n",
       "Q 116.871875 141.899219 116.871875 139.899219 \n",
       "L 116.871875 96.864844 \n",
       "Q 116.871875 94.864844 114.871875 94.864844 \n",
       "L 37.103125 94.864844 \n",
       "Q 35.103125 94.864844 35.103125 96.864844 \n",
       "L 35.103125 139.899219 \n",
       "Q 35.103125 141.899219 37.103125 141.899219 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_26\">\n",
       "     <path d=\"M 39.103125 102.963281 \n",
       "L 49.103125 102.963281 \n",
       "L 59.103125 102.963281 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- train loss -->\n",
       "     <g transform=\"translate(67.103125 106.463281)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"264.550781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"292.333984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"353.515625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"405.615234\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_27\">\n",
       "     <path d=\"M 39.103125 117.641406 \n",
       "L 49.103125 117.641406 \n",
       "L 59.103125 117.641406 \n",
       "\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- train acc -->\n",
       "     <g transform=\"translate(67.103125 121.141406)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"264.550781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"325.830078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"380.810547\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_28\">\n",
       "     <path d=\"M 39.103125 132.319531 \n",
       "L 49.103125 132.319531 \n",
       "L 59.103125 132.319531 \n",
       "\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_15\">\n",
       "     <!-- test acc -->\n",
       "     <g transform=\"translate(67.103125 135.819531)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"192.041016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"223.828125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"285.107422\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"340.087891\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pd8061e30b9\">\n",
       "   <rect x=\"30.103125\" y=\"10.999219\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr, num_epochs = 1e-4, 10\n",
    "trainer = torch.optim.Adam(net.parameters(), lr=lr)\n",
    "loss = nn.CrossEntropyLoss(reduction='none')\n",
    "net(next(iter(train_iter))[0])\n",
    "d2l.train_ch13(net, train_iter, test_iter, loss, trainer, num_epochs, devices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "f3a0a13a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def predict_snli_bert(net, vocab, premise, hypothesis):\n",
    "    \"\"\"Predict the logical relationship between the premise and hypothesis.\"\"\"\n",
    "    net.eval()\n",
    "    tokens_X = [vocab.token_to_idx['<cls>']] + \\\n",
    "        [vocab.token_to_idx[item] for item in premise] + [vocab.token_to_idx['<sep>']] + \\\n",
    "        [vocab.token_to_idx[item] for item in hypothesis] + [vocab.token_to_idx['<sep>']]\n",
    "    segments_X = [0] + [0 for item in premise] + [0] + [1 for item in hypothesis] + [1]\n",
    "    valid_lens_x = len(tokens_X)\n",
    "    \n",
    "    for i in range(128):\n",
    "        if i < valid_lens_x:\n",
    "            continue\n",
    "        else:\n",
    "            tokens_X.append(1)\n",
    "            segments_X.append(0)\n",
    "\n",
    "    tokens_X = torch.tensor(tokens_X, device=d2l.try_gpu()).reshape((1, -1))\n",
    "    segments_X = torch.tensor(segments_X, device=d2l.try_gpu()).reshape((1, -1))\n",
    "    valid_lens_x = torch.tensor([valid_lens_x], device=d2l.try_gpu())\n",
    "    \n",
    "    result = net([tokens_X, segments_X, valid_lens_x])\n",
    "    print(result[0])\n",
    "    label = torch.argmax(result, dim=1)[0]\n",
    "    return 'entailment' if label == 0 else 'contradiction' if label == 1 else 'neutral'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "e7cc3f84",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.4168, 0.0197, 0.2690], device='cuda:0', grad_fn=<SelectBackward0>)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'entailment'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict_snli_bert(net, vocab, ['he', 'is', 'delighted', '.'], ['he', 'feels', 'good', '.'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "0cc41a33",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-2.2405, -0.3519,  2.2688], device='cuda:0', grad_fn=<SelectBackward0>)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'neutral'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict_snli_bert(net, vocab, ['he', 'is', 'depressed', '.'], ['he', 'is', 'tall', '.'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "544ee553",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-3.1151,  3.7257, -0.1885], device='cuda:0', grad_fn=<SelectBackward0>)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'contradiction'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict_snli_bert(net, vocab, ['he', 'is', 'so', 'thin', '.'], ['he', 'eats', 'a', 'lot', '.'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efa0aa46",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Chat with ChatGPT\n",
    "\n",
    "[https://lebronlihd.github.io/blog/2023/chat-with-chatgpt/](https://lebronlihd.github.io/blog/2023/chat-with-chatgpt/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "453ceddc",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: openai in /home/haodong/Applications/Miniconda/Miniconda/envs/dl/lib/python3.9/site-packages (0.27.2)\r\n",
      "Requirement already satisfied: tqdm in /home/haodong/Applications/Miniconda/Miniconda/envs/dl/lib/python3.9/site-packages (from openai) (4.64.1)\r\n",
      "Requirement already satisfied: requests>=2.20 in /home/haodong/Applications/Miniconda/Miniconda/envs/dl/lib/python3.9/site-packages (from openai) (2.28.1)\r\n",
      "Requirement already satisfied: aiohttp in /home/haodong/Applications/Miniconda/Miniconda/envs/dl/lib/python3.9/site-packages (from openai) (3.8.1)\r\n",
      "Requirement already satisfied: charset-normalizer<3,>=2 in /home/haodong/Applications/Miniconda/Miniconda/envs/dl/lib/python3.9/site-packages (from requests>=2.20->openai) (2.1.0)\r\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/haodong/Applications/Miniconda/Miniconda/envs/dl/lib/python3.9/site-packages (from requests>=2.20->openai) (3.3)\r\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/haodong/Applications/Miniconda/Miniconda/envs/dl/lib/python3.9/site-packages (from requests>=2.20->openai) (1.26.11)\r\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/haodong/Applications/Miniconda/Miniconda/envs/dl/lib/python3.9/site-packages (from requests>=2.20->openai) (2022.12.7)\r\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /home/haodong/Applications/Miniconda/Miniconda/envs/dl/lib/python3.9/site-packages (from aiohttp->openai) (4.0.2)\r\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /home/haodong/Applications/Miniconda/Miniconda/envs/dl/lib/python3.9/site-packages (from aiohttp->openai) (6.0.2)\r\n",
      "Requirement already satisfied: attrs>=17.3.0 in /home/haodong/Applications/Miniconda/Miniconda/envs/dl/lib/python3.9/site-packages (from aiohttp->openai) (21.4.0)\r\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /home/haodong/Applications/Miniconda/Miniconda/envs/dl/lib/python3.9/site-packages (from aiohttp->openai) (1.7.2)\r\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /home/haodong/Applications/Miniconda/Miniconda/envs/dl/lib/python3.9/site-packages (from aiohttp->openai) (1.3.1)\r\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /home/haodong/Applications/Miniconda/Miniconda/envs/dl/lib/python3.9/site-packages (from aiohttp->openai) (1.3.3)\r\n"
     ]
    }
   ],
   "source": [
    "!pip install openai # !conda install -c conda-forge openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "96474026",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You: Hi, what is the main idea about AlphaFold2?\n",
      "\n",
      "ChatGPT: AlphaFold2 is a revolutionary artificial intelligence system that can predict the 3D structure of proteins with remarkable accuracy. It has the potential to transform the field of protein science and could lead to major breakthroughs in drug development, disease understanding, and personalized medicine. The system was developed by researchers at the University of California, San Francisco (UCSF) and the University of Washington, and was recently recognized as a breakthrough achievement in artificial intelligence, winning the prestigious Casp14 protein-structure prediction competition.\n",
      "\n",
      "You: Will it replace protein scientists in the future?\n",
      "\n",
      "ChatGPT: No, it is not likely that AlphaFold2 will completely replace protein scientists in the future. While AlphaFold2 is certainly a powerful tool that can greatly aid in protein research, there are still areas where human expertise is required. For example, AlphaFold2 currently requires extensive data processing and optimization, and its predictions still require validation through laboratory experiments. Additionally, protein science is a complex field that requires a deep understanding of biology, chemistry, and physics, which are skills that only human scientists possess. Therefore, it is more likely that AlphaFold2 will be seen as a valuable support tool for protein scientists, rather than a replacement.\n",
      "\n",
      "You: I agree with you, thanks!\n",
      "\n",
      "ChatGPT: You're welcome! If you have any more questions, feel free to ask.\n",
      "\n",
      "You: q\n"
     ]
    }
   ],
   "source": [
    "# export OPENAI_API_KEY='sk-...'\n",
    "import openai\n",
    "\n",
    "messages = [] # [{\"role\": \"system\", \"content\": \"You are a strict interviewer\"}]\n",
    "user_text = input(\"You: \")\n",
    "\n",
    "def no_br_in_begin(reply):\n",
    "    for i in range(len(reply)):\n",
    "        if reply[i] != '\\n':\n",
    "            return reply[i:]\n",
    "    raise KeyboardInterrupt\n",
    "\n",
    "def interact_with_ChatGPT():\n",
    "    global messages, user_text\n",
    "\n",
    "    # I want this code to run indefinitely, until any of the parties say Thank You\n",
    "    while user_text not in [\"q\", \"Q\"]:\n",
    "        messages.append({\"role\": \"user\", \"content\": user_text})\n",
    "        # Now gpt-4 is availavle for ChatGPT-Plus, New Bing, or Dev. API\n",
    "        response = openai.ChatCompletion.create(model=\"gpt-3.5-turbo\", messages=messages)\n",
    "        reply = response.choices[0].message.content\n",
    "        print(\"\\nChatGPT:\", no_br_in_begin(reply))\n",
    "        messages.append({\"role\": \"assistant\", \"content\": reply})\n",
    "        user_text = input(\"\\nYou: \")\n",
    "\n",
    "interact_with_ChatGPT()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae915d5c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### GPT-3.5 v.s. GPT-4\n",
    "\n",
    "![gpt-3.5-vs-4.png](./images/8/gpt-3.5-vs-4.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "c243b3a8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You: Two dogs eating pizza, in realistic style\n",
      "Generated image was saved at -> ./images/8/Two_dogs_eating_pizza,_in_realistic_style.png\n"
     ]
    }
   ],
   "source": [
    "# Written by ChatGPT on 03/20/2023\n",
    "import openai\n",
    "import requests\n",
    "from PIL import Image\n",
    "\n",
    "# Define the text prompt for the image\n",
    "text = input('You: ')\n",
    "\n",
    "# Call the OpenAI DALL-E2 API to generate the image\n",
    "response = openai.Image.create(\n",
    "    prompt=text,\n",
    "    size='512x512',\n",
    "    model='image-alpha-001'\n",
    ")\n",
    "\n",
    "# Get the image data from the API response\n",
    "url = response['data'][0]['url']\n",
    "image_data = requests.get(url).content\n",
    "\n",
    "# Save the image to a file\n",
    "image_path = './images/8/' + text.replace(' ', '_') + '.png'\n",
    "print('Generated image was saved at ->', image_path)\n",
    "with open(image_path, 'wb') as f:\n",
    "    f.write(image_data)\n",
    "\n",
    "# Load the image to a PIL Image object and display it\n",
    "image = Image.open(image_path)\n",
    "image.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "426f846a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![png](./images/8/Two_dogs_eating_pizza,_in_realistic_style.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52ee66ab",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## nanoGPT\n",
    "\n",
    "- The simplest, fastest repository for training/finetuning medium-sized GPTs\n",
    "- [https://github.com/karpathy/nanoGPT](https://github.com/karpathy/nanoGPT)\n",
    "- by: [Andrej Karpathy](https://karpathy.ai/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c9e7fb1",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "```\n",
    "ANGELO:\n",
    "And cowards it be strawn to my bed,\n",
    "And thrust the gates of my threats,\n",
    "Because he that ale away, and hang'd\n",
    "An one with him.\n",
    "\n",
    "DUKE VINCENTIO:\n",
    "I thank your eyes against it.\n",
    "\n",
    "DUKE VINCENTIO:\n",
    "Then will answer him to save the malm:\n",
    "And what have you tyrannous shall do this?\n",
    "\n",
    "DUKE VINCENTIO:\n",
    "If you have done evils of all disposition\n",
    "To end his power, the day of thrust for a common men\n",
    "That I leave, to fight with over-liking\n",
    "Hasting in a roseman.\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}