foodnames_autoencoder.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recipe Generation with Seq2Seq Autoencoders\n",
    "\n",
    "In this notebook, I train a Seq2Seq autoencoder to encode and decode recipe names. Using this, I use random state inputs to the decoder to generate fictitious recipe names."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0EsGxJqwgmcg"
   },
   "source": [
    "## Load and preprocess data\n",
    "\n",
    "First, we must acquire the data. For my experiment, I used data from [Eight Portions](https://eightportions.com/datasets/Recipes/), who provide a very useful dataset of recipes including names, ingredients, and directions. I only plan to use the names of the recipes, so I will trim the data for that information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 272
    },
    "colab_type": "code",
    "id": "hbxsRoblagyD",
    "outputId": "dc1c7b3e-41b7-4a60-f203-764217806105"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 50.8M  100 50.8M    0     0  9730k      0  0:00:05  0:00:05 --:--:--  9.8M\n",
      "Archive:  recipes.zip\n",
      "  inflating: recipes_raw_nosource_ar.json  \n",
      "  inflating: recipes_raw_nosource_epi.json  \n",
      "  inflating: recipes_raw_nosource_fn.json  \n",
      "  inflating: LICENSE                 \n"
     ]
    }
   ],
   "source": [
    "# Download the recipes\n",
    "!curl -o recipes.zip 'https://storage.googleapis.com/recipe-box/recipes_raw.zip'\n",
    "\n",
    "# Unzip without remorse\n",
    "!unzip -o recipes.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "id": "ZETwTGLIasFj",
    "outputId": "0fe6409a-a43c-4df0-efb9-21654ac3bea5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"rmK12Uau.ntP510KeImX506H6Mr6jTu\": {\r\n",
      "    \"title\": \"Slow Cooker Chicken and Dumplings\",\r\n",
      "    \"ingredients\": [\r\n",
      "      \"4 skinless, boneless chicken breast halves ADVERTISEMENT\",\r\n",
      "      \"2 tablespoons butter ADVERTISEMENT\",\r\n",
      "      \"2 (10.75 ounce) cans condensed cream of chicken soup ADVERTISEMENT\",\r\n",
      "      \"1 onion, finely diced ADVERTISEMENT\",\r\n",
      "      \"2 (10 ounce) packages refrigerated biscuit dough, torn into pieces ADVERTISEMENT\",\r\n",
      "      \"ADVERTISEMENT\"\r\n"
     ]
    }
   ],
   "source": [
    "!head recipes_raw_nosource_ar.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8jTTzDG9bOya"
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "# Load the JSON data\n",
    "with open('recipes_raw_nosource_ar.json', 'r') as f:\n",
    "    data = json.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When extracting the names, I found that some recipes did not have names. So I had to filter those out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "5CV-uc4dbinQ",
    "outputId": "908287f4-7b55-44ab-9a35-f392010b3062"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Slow Cooker Chicken and Dumplings\n",
      "Awesome Slow Cooker Pot Roast\n",
      "Brown Sugar Meatloaf\n",
      "Best Chocolate Chip Cookies\n",
      "Homemade Mac and Cheese Casserole\n",
      "Banana Banana Bread\n",
      "Chef John's Fisherman's Pie\n",
      "Mom's Zucchini Bread\n",
      "The Best Rolled Sugar Cookies\n",
      "Singapore Chili Crabs\n",
      "39522 of 39802\n"
     ]
    }
   ],
   "source": [
    "# Pull out the names\n",
    "names = [data[k]['title'] for k in data if 'title' in data[k]]\n",
    "\n",
    "for r in names[:10]:\n",
    "    print(r)\n",
    "\n",
    "print(len(names), 'of', len(data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_string(txt):\n",
    "    # Trim special characters to simplify everyone's lives\n",
    "    for i in range(len(txt)-1, -1, -1):\n",
    "        if ord(txt[i]) > 127:\n",
    "            txt = txt[:i] + txt[i+1:]\n",
    "            \n",
    "    return (txt\n",
    "            .replace('(', ' ( ') # Left parentheses\n",
    "            .replace(')', ' ) ') # Right parentheses\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "names = list(map(preprocess_string, names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1ve_KX6UdPSK"
   },
   "source": [
    "## Tokenize data\n",
    "\n",
    "In order to pass the strings in, we need to create a numerical representation that can be used by the network. I define methods for encoding  a given string of text or decoding label predictions into a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8GG35yAZevnM"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import math, random\n",
    "\n",
    "import tensorflow_datasets as tfds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8qajOFm0b18d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['n', '9', '5', '?', 'T', '8', 'W', 'Q', 'X', 'R', 'a', 'D', 'O', 'x', '7', 'o', '-', 'M', ':', 'E', 'b', 's', '0', 'd', '.', 'i', 'A', 'p', 'S', 'g', 'c', 'f', 'q', '/', 'h', 'K', '@', '&', 'Z', 'N', 'l', '4', 'k', 'r', '\\t', 'G', 'w', '+', 'J', 'L', 'j', '\"', \"'\", '$', '*', '=', '2', '1', 'U', '(', 'B', ';', 'z', 'y', 't', 'v', 'e', '#', 'C', '6', '\\n', 'H', ' ', 'u', 'F', ',', 'V', 'I', 'm', '3', ')', '%', 'Y', 'P', '!']\n"
     ]
    }
   ],
   "source": [
    "# Extract all of the characters and words\n",
    "chars = set(c for n in names for c in n)\n",
    "words = set(w for n in names for w in n.split())\n",
    "\n",
    "# We also select special start and end characters. These are used\n",
    "# on the decoder side.\n",
    "chars.add('\\t')\n",
    "words.add('<start>')\n",
    "chars.add('\\n')\n",
    "words.add('<end>')\n",
    "\n",
    "chars = list(chars)\n",
    "words = list(words)\n",
    "\n",
    "print(chars)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "csIPpyJpdOT1"
   },
   "outputs": [],
   "source": [
    "# Given a word, provide the equivalent token\n",
    "word_idx = {w : i+1 for i, w in enumerate(words)}\n",
    "\n",
    "# Given a token, give back the word\n",
    "word_inv_idx = {i+1 : w for i, w in enumerate(words)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "Eyo63GZHMF93",
    "outputId": "c35842c6-87d1-4cc2-a102-0848fd4c4e51"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocab size: 11922\n"
     ]
    }
   ],
   "source": [
    "# Number of possible characters, including the 'zero' character used by embeddings\n",
    "vocab_size = 1 + len(words)\n",
    "\n",
    "print('Vocab size:', vocab_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can define the tokenization and detokenization behavior. We define two methods:\n",
    "\n",
    "- `str2tok()`: Converts text to tokens\n",
    "- `tok2str()`: Converts tokens to text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QJd2ENtIceNt"
   },
   "outputs": [],
   "source": [
    "def str2tok(txt, length=None):\n",
    "    # Split the string and break into tokens\n",
    "    enc = [word_idx[w] for w in txt.split()]\n",
    "    \n",
    "    if length:\n",
    "        enc += (length - len(enc)) * [0]\n",
    "        \n",
    "    return enc\n",
    "\n",
    "def tok2str(idxs):\n",
    "    # Rejoin\n",
    "    return ' '.join([word_inv_idx[i] for i in idxs if i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using this, we can tokenize the data into the format usable by the neural networks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 374
    },
    "colab_type": "code",
    "id": "wRtPoazLeeEg",
    "outputId": "b7cf3446-98c5-4dd9-8352-9a98b41f7204"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Max length: 20\n",
      "[ 3633  1689 11516  7502 11569     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0]\n",
      "[ 7622  3633  1689 11516  7502 11569     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0]\n",
      "[ 3633  1689 11516  7502 11569  2680     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0]\n",
      "(39522, 20)\n",
      "(39522, 22)\n",
      "(39522, 22)\n"
     ]
    }
   ],
   "source": [
    "# Encode the dataset\n",
    "max_len = max(len(str2tok(n)) for n in names)\n",
    "\n",
    "enc_input = np.array([str2tok(n, max_len) for n in names])\n",
    "dec_input = np.array([str2tok(f\"<start> {n}\", 2+max_len) for n in names])\n",
    "dec_output = np.array([str2tok(f\"{n} <end>\", 2+max_len) for n in names])\n",
    "\n",
    "print('Max length:', max_len)\n",
    "print(enc_input[0])\n",
    "print(dec_input[0])\n",
    "print(dec_output[0])\n",
    "\n",
    "print(enc_input.shape)\n",
    "print(dec_input.shape)\n",
    "print(dec_output.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to define a training/test split. This allows us to check for overfitting of the data during hyperparameter tuning. I do this by generating a list of indices and splitting them into two buckets: one that's used to train and one used solely for evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MTY81LE9qurx"
   },
   "outputs": [],
   "source": [
    "# Train-test-valid split\n",
    "idxs = list(range(len(enc_input)))\n",
    "random.shuffle(idxs)\n",
    "\n",
    "a = int(0.9 * len(idxs))\n",
    "train_idxs = idxs[:a]\n",
    "valid_idxs = idxs[a:]\n",
    "idxs = train_idxs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8FNuEDtLhBaT"
   },
   "source": [
    "## Model\n",
    "\n",
    "We will use a Seq2Seq model to generate our results. This constitutes an encoder to convert text to an encoding, and a decoder to convert it back to text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "gt76wwAygcEF",
    "outputId": "2417573e-c99e-46df-891b-97b87d374e0c"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "from tensorflow.keras.layers import *\n",
    "from tensorflow.keras.models import *\n",
    "from tensorflow.keras.optimizers import *\n",
    "\n",
    "# Silence warnings\n",
    "import logging\n",
    "tf.get_logger().setLevel(logging.ERROR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vvpl_vPehoDP"
   },
   "outputs": [],
   "source": [
    "latent_dim = 64\n",
    "dropout = 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Y8x987EykQiR"
   },
   "source": [
    "### Encoder\n",
    "\n",
    "The Seq2Seq encoder includes an embedding layer that converts a token into a representative vector and a recurrent neural network (RNN) component. The RNN takes in a sequence/list of these vectors and processes them one by one to compute a representative encoding of the input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 513
    },
    "colab_type": "code",
    "id": "GQVepj1FhZki",
    "outputId": "943751b5-78f0-4eb4-dfd9-d225c9206e70"
   },
   "outputs": [],
   "source": [
    "enc_in = Input(shape=(None,), name='enc_in')\n",
    "\n",
    "# Apply an embedding to the input\n",
    "emb = Embedding(vocab_size, latent_dim)\n",
    "y = emb(enc_in)\n",
    "\n",
    "# Pass through an RNN\n",
    "rnn = Bidirectional(GRU(latent_dim // 2, return_state=True, dropout=dropout))\n",
    "_, h1, h2 = rnn(y)\n",
    "\n",
    "# Concatenate the output states\n",
    "h = Concatenate()([h1, h2])\n",
    "\n",
    "encoder = Model(enc_in, h, name='encoder')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x8NvFwg_kSE5"
   },
   "source": [
    "### Decoder\n",
    "\n",
    "The decoder goes in the other way; taking the encoding and converting it into a list of characters. However, the network generates probabilities. This allows us to randomize our results or deterministically select the most likely character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 343
    },
    "colab_type": "code",
    "id": "YCkxZFUgieS1",
    "outputId": "a6614fd5-c249-4eb0-bf2b-32df67d8237f"
   },
   "outputs": [],
   "source": [
    "dec_in = Input(shape=(None,), name='dec_in')\n",
    "h = Input(shape=(latent_dim,), name='state_in')\n",
    "\n",
    "# Embed the decoder input\n",
    "emb = Embedding(vocab_size, latent_dim)\n",
    "y = emb(dec_in)\n",
    "\n",
    "# Pass through a generator LSTM\n",
    "rnn = GRU(latent_dim, return_state=True, return_sequences=True, dropout=dropout)\n",
    "y, c = rnn(y, initial_state = h)\n",
    "\n",
    "# Choose the the character by computing a probability distribution\n",
    "dense = Dense(vocab_size, activation='softmax')\n",
    "y = dense(y)\n",
    "\n",
    "decoder = Model([dec_in, h], [y, c], name='decoder')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IM1b0R73kT3A"
   },
   "source": [
    "### Trainer\n",
    "\n",
    "The trainer is a concatenation of the encoder and decoder. We take a sequence of tokens, pass it into the encoder to get an encoding, then pass it back into the decoder. The goal is to get the original string back, but in general we may want to decode into something else (ex: neural machine translation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 309
    },
    "colab_type": "code",
    "id": "mJHF1mBQj9Qn",
    "outputId": "ef4b4d20-3f11-4887-86d6-bf9f04a516a1"
   },
   "outputs": [],
   "source": [
    "# Model to train an autoencoder\n",
    "enc_in = Input(shape=(None,), name='enc_in')\n",
    "dec_in = Input(shape=(None,), name='dec_in')\n",
    "\n",
    "z = encoder(enc_in)\n",
    "y, _ = decoder([dec_in, z])\n",
    "\n",
    "model = Model([enc_in, dec_in], y, name='autoencoder_trainer')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bARxHfVakwd9"
   },
   "source": [
    "## Model training\n",
    "\n",
    "Now, we can train our model. I define a data generator that provides data one batch at a time. This is done because the actual values used by the network would require a massive amount of memory to store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XQqaawWOluhG"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.utils import to_categorical\n",
    "\n",
    "def data_gen(idxs, batch_size=64, repeat=True):\n",
    "    not_done = True\n",
    "    \n",
    "    while not_done:\n",
    "        random.shuffle(idxs)\n",
    "\n",
    "        for i in range(batch_size, len(idxs), batch_size):\n",
    "            # Chosen items\n",
    "            i = idxs[i-batch_size:i]\n",
    "\n",
    "            # Input\n",
    "            xe = enc_input[i]\n",
    "            xd = dec_input[i]\n",
    "            \n",
    "            # Find the length of the longest item in the data point\n",
    "            j = 0\n",
    "            while j < max_len:\n",
    "                if all(xe[:,j] == 0):\n",
    "                    break\n",
    "                j += 1\n",
    "\n",
    "            # Output\n",
    "            y = dec_output[i]\n",
    "            data = np.zeros((batch_size, y.shape[1], vocab_size))\n",
    "            for i in range(len(data)):\n",
    "                data[i] = to_categorical(y[i], num_classes=vocab_size)\n",
    "              \n",
    "            # Use the length to trim the string. This is a heuristic to\n",
    "            # reduce the amount of training time spent on groupings of\n",
    "            # shorter inputs.\n",
    "            xe = xe[:,:j]\n",
    "            xd = xd[:,:j+1]\n",
    "            data = data[:,:j+1]\n",
    "            \n",
    "            yield [xe, xd], data\n",
    "        \n",
    "        not_done = repeat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"encoder\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "enc_in (InputLayer)             [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding (Embedding)           (None, None, 64)     763008      enc_in[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional (Bidirectional)   [(None, 64), (None,  18816       embedding[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "concatenate (Concatenate)       (None, 64)           0           bidirectional[0][1]              \n",
      "                                                                 bidirectional[0][2]              \n",
      "==================================================================================================\n",
      "Total params: 781,824\n",
      "Trainable params: 781,824\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n",
      "Model: \"decoder\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "dec_in (InputLayer)             [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, None, 64)     763008      dec_in[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "state_in (InputLayer)           [(None, 64)]         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "gru_1 (GRU)                     [(None, None, 64), ( 24960       embedding_1[0][0]                \n",
      "                                                                 state_in[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "dense (Dense)                   (None, None, 11922)  774930      gru_1[0][0]                      \n",
      "==================================================================================================\n",
      "Total params: 1,562,898\n",
      "Trainable params: 1,562,898\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "encoder.summary()\n",
    "decoder.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"autoencoder_trainer\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "enc_in (InputLayer)             [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dec_in (InputLayer)             [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "encoder (Model)                 (None, 64)           781824      enc_in[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "decoder (Model)                 [(None, None, 11922) 1562898     dec_in[0][0]                     \n",
      "                                                                 encoder[1][0]                    \n",
      "==================================================================================================\n",
      "Total params: 2,344,722\n",
      "Trainable params: 2,344,722\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 105
    },
    "colab_type": "code",
    "id": "CklGcr74kXeN",
    "outputId": "0110908d-41df-4051-e79b-f592d365c4a8"
   },
   "outputs": [],
   "source": [
    "# Build the model\n",
    "model.compile(optimizer=Adam(), loss='categorical_crossentropy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = 32\n",
    "batch_size = 64\n",
    "steps_per_epoch = len(train_idxs) // batch_size\n",
    "validation_steps = len(valid_idxs) // batch_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 513
    },
    "colab_type": "code",
    "id": "uFQQ_fgBkqvD",
    "outputId": "77f31242-90fa-4ae3-86b0-af515bdda8b0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/32\n",
      "555/555 [==============================] - 190s 343ms/step - loss: 3.4992 - val_loss: 2.6739\n",
      "Epoch 2/32\n",
      "555/555 [==============================] - 171s 307ms/step - loss: 2.5068 - val_loss: 2.3901\n",
      "Epoch 3/32\n",
      "555/555 [==============================] - 170s 307ms/step - loss: 2.2589 - val_loss: 2.1957\n",
      "Epoch 4/32\n",
      "555/555 [==============================] - 171s 308ms/step - loss: 2.0589 - val_loss: 2.0664\n",
      "Epoch 5/32\n",
      "555/555 [==============================] - 168s 302ms/step - loss: 1.9080 - val_loss: 1.9620\n",
      "Epoch 6/32\n",
      "555/555 [==============================] - 169s 305ms/step - loss: 1.7702 - val_loss: 1.8530\n",
      "Epoch 7/32\n",
      "555/555 [==============================] - 168s 304ms/step - loss: 1.6581 - val_loss: 1.7199\n",
      "Epoch 8/32\n",
      "555/555 [==============================] - 169s 304ms/step - loss: 1.5334 - val_loss: 1.6678\n",
      "Epoch 9/32\n",
      "555/555 [==============================] - 167s 302ms/step - loss: 1.4364 - val_loss: 1.5844\n",
      "Epoch 10/32\n",
      "555/555 [==============================] - 169s 305ms/step - loss: 1.3419 - val_loss: 1.5057\n",
      "Epoch 11/32\n",
      "555/555 [==============================] - 168s 303ms/step - loss: 1.2562 - val_loss: 1.4249\n",
      "Epoch 12/32\n",
      "555/555 [==============================] - 168s 303ms/step - loss: 1.1769 - val_loss: 1.3786\n",
      "Epoch 13/32\n",
      "555/555 [==============================] - 169s 304ms/step - loss: 1.1051 - val_loss: 1.3175\n",
      "Epoch 14/32\n",
      "555/555 [==============================] - 168s 303ms/step - loss: 1.0386 - val_loss: 1.2568\n",
      "Epoch 15/32\n",
      "555/555 [==============================] - 168s 302ms/step - loss: 0.9786 - val_loss: 1.2273\n",
      "Epoch 16/32\n",
      "555/555 [==============================] - 168s 303ms/step - loss: 0.9192 - val_loss: 1.1901\n",
      "Epoch 17/32\n",
      "555/555 [==============================] - 169s 304ms/step - loss: 0.8651 - val_loss: 1.1877\n",
      "Epoch 18/32\n",
      "555/555 [==============================] - 169s 304ms/step - loss: 0.8150 - val_loss: 1.0959\n",
      "Epoch 19/32\n",
      "555/555 [==============================] - 168s 302ms/step - loss: 0.7732 - val_loss: 1.1059\n",
      "Epoch 20/32\n",
      "555/555 [==============================] - 168s 302ms/step - loss: 0.7329 - val_loss: 1.0456\n",
      "Epoch 21/32\n",
      "555/555 [==============================] - 170s 306ms/step - loss: 0.6877 - val_loss: 1.1046\n",
      "Epoch 22/32\n",
      "555/555 [==============================] - 168s 303ms/step - loss: 0.6567 - val_loss: 1.0253\n",
      "Epoch 23/32\n",
      "555/555 [==============================] - 167s 301ms/step - loss: 0.6253 - val_loss: 0.9906\n",
      "Epoch 24/32\n",
      "555/555 [==============================] - 168s 302ms/step - loss: 0.5928 - val_loss: 0.9754\n",
      "Epoch 25/32\n",
      "555/555 [==============================] - 168s 303ms/step - loss: 0.5601 - val_loss: 0.9940\n",
      "Epoch 26/32\n",
      "555/555 [==============================] - 169s 304ms/step - loss: 0.5305 - val_loss: 0.9455\n",
      "Epoch 27/32\n",
      "555/555 [==============================] - 168s 303ms/step - loss: 0.5073 - val_loss: 0.9482\n",
      "Epoch 28/32\n",
      "555/555 [==============================] - 166s 300ms/step - loss: 0.4850 - val_loss: 0.9340\n",
      "Epoch 29/32\n",
      "555/555 [==============================] - 166s 300ms/step - loss: 0.4614 - val_loss: 0.9175\n",
      "Epoch 30/32\n",
      "555/555 [==============================] - 167s 301ms/step - loss: 0.4392 - val_loss: 0.9139\n",
      "Epoch 31/32\n",
      "555/555 [==============================] - 167s 301ms/step - loss: 0.4198 - val_loss: 0.8790\n",
      "Epoch 32/32\n",
      "555/555 [==============================] - 168s 303ms/step - loss: 0.4003 - val_loss: 0.9038\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f92b3f33208>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit_generator(data_gen(train_idxs, batch_size=batch_size),\n",
    "                   steps_per_epoch=steps_per_epoch,\n",
    "                   validation_data=data_gen(valid_idxs, batch_size=batch_size),\n",
    "                   validation_steps=validation_steps,\n",
    "                   epochs=epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate\n",
    "\n",
    "Now that we have a model, let's make some food! First, I define some utility functions to do the generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8LGUppSok_WV"
   },
   "outputs": [],
   "source": [
    "def choose_char(p, temperature=0.2):\n",
    "    # Apply temperature\n",
    "    p = np.log(p)\n",
    "    p /= temperature\n",
    "\n",
    "    # Rescale\n",
    "    p = np.exp(p.astype('float64'))\n",
    "    p = p / np.sum(p)\n",
    "\n",
    "    # Randomly choose one from the distribution\n",
    "    p = np.random.multinomial(1, p, 1)\n",
    "\n",
    "    # Choose the most likely character\n",
    "    sampled_token_index = np.argmax(p)\n",
    "    \n",
    "    if sampled_token_index:\n",
    "        token = word_inv_idx[sampled_token_index]\n",
    "    else:\n",
    "        token = False\n",
    "    \n",
    "    return sampled_token_index, token\n",
    "\n",
    "def decode_state(h, temperature=0.2):\n",
    "    x = np.array([[str2tok('<start>')[0]]])\n",
    "    h = np.array([h])\n",
    "\n",
    "    res = []\n",
    "    for _ in range(max_len):\n",
    "        p, h = decoder.predict([x, h])\n",
    "        \n",
    "        i, c = choose_char(p[0][0], temperature=temperature)\n",
    "\n",
    "        if not i or c is False or c == '<end>' or c == '\\n':\n",
    "            # We reached the end of the text\n",
    "            break\n",
    "        else:\n",
    "            # Extend the result with the new token\n",
    "            res.append(i)\n",
    "            # The token to feed in is the one last generated\n",
    "            x[0,0] = i\n",
    "    \n",
    "    # Attach all of the words and return\n",
    "    return tok2str(res)\n",
    "\n",
    "def regenerate(txt, temperature=0.2):\n",
    "    x = np.array([str2tok(txt)])\n",
    "    h = encoder.predict(x)[0]\n",
    "    return decode_state(h)\n",
    "\n",
    "def generate(temperature=0.2):\n",
    "    res = None\n",
    "    while not res:\n",
    "        # Choose a completely random state\n",
    "        h = np.random.uniform(-1, 1, size=(latent_dim,))\n",
    "        # Use the state to generate some text\n",
    "        res = decode_state(h)\n",
    "    \n",
    "    return res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we use the network to generate new food, let's see how it works on existing food. Remember that it should produce the same thing we put in. Of course, it won't be perfect, but that's not really a problem here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source: Slow Cooker Chicken and Dumplings\n",
      "Target: Slow Cooker Chicken and Dumplings\n",
      "\n",
      "Source: Awesome Slow Cooker Pot Roast\n",
      "Target: Awesome Slow Cooker Pizza Roast\n",
      "\n",
      "Source: Brown Sugar Meatloaf\n",
      "Target: Brown Sugar Biscuits\n",
      "\n",
      "Source: Best Chocolate Chip Cookies\n",
      "Target: Best Chocolate Chip Cookies\n",
      "\n",
      "Source: Homemade Mac and Cheese Casserole\n",
      "Target: Homemade Mac and Cheese Casserole\n",
      "\n",
      "Source: Banana Banana Bread\n",
      "Target: Banana Bread Pudding\n",
      "\n",
      "Source: Chef John's Fisherman's Pie\n",
      "Target: Chef John's Fisherman's Pie\n",
      "\n",
      "Source: Mom's Zucchini Bread\n",
      "Target: Mom's Zucchini Bake\n",
      "\n",
      "Source: The Best Rolled Sugar Cookies\n",
      "Target: The Best Dog Sugar Cookies\n",
      "\n",
      "Source: Singapore Chili Crabs\n",
      "Target: Deer and One\n",
      "\n",
      "Source: Downeast Maine Pumpkin Bread\n",
      "Target: Downeast Layer Pumpkin Recipe\n",
      "\n",
      "Source: Best Big, Fat, Chewy Chocolate Chip Cookie\n",
      "Target: Best Big, Fat, Chewy Chocolate Chip Cookie\n",
      "\n",
      "Source: Aimee's Mashed Cauliflower 'Potatoes'\n",
      "Target: Sun Eggplant Tomato Oranges\n",
      "\n",
      "Source: Irish Lamb Stew\n",
      "Target: Irish Fish Stew\n",
      "\n",
      "Source: To Die For Blueberry Muffins\n",
      "Target: To Die For Blueberry Muffins\n",
      "\n",
      "Source: Broiled Tilapia Parmesan\n",
      "Target: Broiled Parmesan Parmesan\n",
      "\n",
      "Source: Award Winning Soft Chocolate Chip Cookies\n",
      "Target: Award Winning White Chocolate Chip Cookies\n",
      "\n",
      "Source: World's Best Lasagna\n",
      "Target: Best Best Burgers\n",
      "\n",
      "Source: Best Brownies\n",
      "Target: Blueberry Brownies\n",
      "\n",
      "Source: Irish Soda Bread\n",
      "Target: Irish Soda Bread\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Show usage on existing samples\n",
    "for n in names[:20]:\n",
    "    print('Source:', n)\n",
    "    print('Target:', regenerate(n, temperature=0.2))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assuming the previous results look good, we can now try generating new values. The code will put stars next to foods that don't exist in the recipe database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "colab_type": "code",
    "id": "3Qdt5rAVt6iP",
    "outputId": "9e01417a-4608-48f1-d137-0941eb553256"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* Eggless Slow Cooker Shots\n",
      "* Ever Mango Pudding\n",
      "* Bars from Reynolds Wrap\n",
      "* Italian Swedish King House\n",
      "* Orange Sprouts\n",
      "* Chile Chicken Livers Bundles\n",
      "* for One\n",
      "* Marshmallows\n",
      "* Tomato Banana Muffins with Raisins and Walnuts\n",
      "* Casserole Ever\n",
      "* Sauce\n",
      "* Quinoa\n",
      "* Punch II\n",
      "* Melt Cotta )\n",
      "* Pickles Pecans\n",
      "  Amaretto\n",
      "* Free )\n",
      "* Ricotta Pops\n",
      "* Souffle Fudge\n",
      "* Meatballs )\n"
     ]
    }
   ],
   "source": [
    "# Generate a text sample\n",
    "for _ in range(20):\n",
    "    n = generate()\n",
    "    if n not in names:\n",
    "        print('*', n)\n",
    "    else:\n",
    "        print(' ', n)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "foodnames.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}