colab_demo_BMT.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "colab_demo_BMT.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "machine_shape": "hm"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.6"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HHJGj1Lv9xxr"
      },
      "source": [
        "\n",
        "<img src=\"https://github.com/v-iashin/v-iashin.github.io/raw/master/images/bmt/bi_modal_transformer.svg\" alt=\"Bi-Modal Transformer with Proposal Generator\" width=\"900\">\n",
        "\n",
        "\n",
        "# Dense Video Captioning with Bi-modal Transformer\n",
        "[Project Page](https://v-iashin.github.io/bmt) • [ArXiv](https://arxiv.org/abs/2005.08271) • [BMVC Page](https://www.bmvc2020-conference.com/conference/papers/paper_0111.html) • [Presentation](https://www.youtube.com/watch?v=C4zYVIqGDVQ)\n",
        "\n",
        "This notebook accompanies the [source code](https://github.com/v-iashin/BMT) of the paper: \n",
        "_A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer_ (BMVC 2020).\n",
        "\n",
        "It is designed to run the custom video prediction demo on [Google Colab](https://colab.research.google.com) with GPU. \n",
        "Running the notebook on the basic Google Colab version from scratch will take around 30 minutes including downloading model checkpoints and installing environments.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mY9WQfMp-Wn6"
      },
      "source": [
        "## Cloning the Repository"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8hDJyJNKV7QK",
        "outputId": "becf1afb-b713-4373-b562-292d5d847fef"
      },
      "source": [
        "!git clone --recursive https://github.com/v-iashin/BMT.git\n",
        "%cd BMT/"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'BMT'...\n",
            "remote: Enumerating objects: 149, done.\u001b[K\n",
            "remote: Counting objects: 100% (26/26), done.\u001b[K\n",
            "remote: Compressing objects: 100% (21/21), done.\u001b[K\n",
            "remote: Total 149 (delta 9), reused 10 (delta 3), pack-reused 123\u001b[K\n",
            "Receiving objects: 100% (149/149), 13.34 MiB | 12.84 MiB/s, done.\n",
            "Resolving deltas: 100% (56/56), done.\n",
            "Submodule 'submodules/pycocoevalcap' (https://github.com/salaniz/pycocoevalcap.git) registered for path 'submodules/pycocoevalcap'\n",
            "Submodule 'submodules/video_features' (https://github.com/v-iashin/video_features.git) registered for path 'submodules/video_features'\n",
            "Cloning into '/content/BMT/submodules/pycocoevalcap'...\n",
            "remote: Enumerating objects: 808, done.        \n",
            "remote: Counting objects: 100% (11/11), done.        \n",
            "remote: Compressing objects: 100% (10/10), done.        \n",
            "remote: Total 808 (delta 1), reused 6 (delta 1), pack-reused 797        \n",
            "Receiving objects: 100% (808/808), 130.05 MiB | 30.33 MiB/s, done.\n",
            "Resolving deltas: 100% (420/420), done.\n",
            "Cloning into '/content/BMT/submodules/video_features'...\n",
            "remote: Enumerating objects: 455, done.        \n",
            "remote: Counting objects: 100% (425/425), done.        \n",
            "remote: Compressing objects: 100% (266/266), done.        \n",
            "remote: Total 455 (delta 185), reused 352 (delta 115), pack-reused 30        \n",
            "Receiving objects: 100% (455/455), 163.77 MiB | 29.74 MiB/s, done.\n",
            "Resolving deltas: 100% (186/186), done.\n",
            "Submodule path 'submodules/pycocoevalcap': checked out 'ca1b05fa0e99f86de2259f87d43c72b322523731'\n",
            "Submodule path 'submodules/video_features': checked out '4fa02bd5c5b8c34081dcfb609e2bcd5a973eaab2'\n",
            "/content/BMT\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2slyvAsL-I6X"
      },
      "source": [
        "## Installing Environments\n",
        "Since feature extraction and the captioning model rely on different Python environments, we use consider `conda` as the easiest solution here. \n",
        "Unfortunately, Google Colab does not support `conda` natively.\n",
        "To this end, we need to do a few tricks."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kyaQ13s4PZNl"
      },
      "source": [
        "First, let's install Miniconda and tweak Python import context"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "audwYM3BPZNl",
        "outputId": "a3438e10-ba5e-4a8e-e6ad-db4d7ea0bbb0"
      },
      "source": [
        "!wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh -q --show-progress\n",
        "!bash ./Miniconda3-py37_4.8.2-Linux-x86_64.sh -b -f -p /usr/local"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Miniconda3-py37_4.8 100%[===================>]  81.12M   150MB/s    in 0.5s    \n",
            "PREFIX=/usr/local\n",
            "Unpacking payload ...\n",
            "Collecting package metadata (current_repodata.json): - \b\b\\ \b\b| \b\b/ \b\bdone\n",
            "Solving environment: \\ \b\b| \b\bdone\n",
            "\n",
            "## Package Plan ##\n",
            "\n",
            "  environment location: /usr/local\n",
            "\n",
            "  added / updated specs:\n",
            "    - _libgcc_mutex==0.1=main\n",
            "    - asn1crypto==1.3.0=py37_0\n",
            "    - ca-certificates==2020.1.1=0\n",
            "    - certifi==2019.11.28=py37_0\n",
            "    - cffi==1.14.0=py37h2e261b9_0\n",
            "    - chardet==3.0.4=py37_1003\n",
            "    - conda-package-handling==1.6.0=py37h7b6447c_0\n",
            "    - conda==4.8.2=py37_0\n",
            "    - cryptography==2.8=py37h1ba5d50_0\n",
            "    - idna==2.8=py37_0\n",
            "    - ld_impl_linux-64==2.33.1=h53a641e_7\n",
            "    - libedit==3.1.20181209=hc058e9b_0\n",
            "    - libffi==3.2.1=hd88cf55_4\n",
            "    - libgcc-ng==9.1.0=hdf63c60_0\n",
            "    - libstdcxx-ng==9.1.0=hdf63c60_0\n",
            "    - ncurses==6.2=he6710b0_0\n",
            "    - openssl==1.1.1d=h7b6447c_4\n",
            "    - pip==20.0.2=py37_1\n",
            "    - pycosat==0.6.3=py37h7b6447c_0\n",
            "    - pycparser==2.19=py37_0\n",
            "    - pyopenssl==19.1.0=py37_0\n",
            "    - pysocks==1.7.1=py37_0\n",
            "    - python==3.7.6=h0371630_2\n",
            "    - readline==7.0=h7b6447c_5\n",
            "    - requests==2.22.0=py37_1\n",
            "    - ruamel_yaml==0.15.87=py37h7b6447c_0\n",
            "    - setuptools==45.2.0=py37_0\n",
            "    - six==1.14.0=py37_0\n",
            "    - sqlite==3.31.1=h7b6447c_0\n",
            "    - tk==8.6.8=hbc83047_0\n",
            "    - tqdm==4.42.1=py_0\n",
            "    - urllib3==1.25.8=py37_0\n",
            "    - wheel==0.34.2=py37_0\n",
            "    - xz==5.2.4=h14c3975_4\n",
            "    - yaml==0.1.7=had09818_2\n",
            "    - zlib==1.2.11=h7b6447c_3\n",
            "\n",
            "\n",
            "The following NEW packages will be INSTALLED:\n",
            "\n",
            "  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main\n",
            "  asn1crypto         pkgs/main/linux-64::asn1crypto-1.3.0-py37_0\n",
            "  ca-certificates    pkgs/main/linux-64::ca-certificates-2020.1.1-0\n",
            "  certifi            pkgs/main/linux-64::certifi-2019.11.28-py37_0\n",
            "  cffi               pkgs/main/linux-64::cffi-1.14.0-py37h2e261b9_0\n",
            "  chardet            pkgs/main/linux-64::chardet-3.0.4-py37_1003\n",
            "  conda              pkgs/main/linux-64::conda-4.8.2-py37_0\n",
            "  conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.6.0-py37h7b6447c_0\n",
            "  cryptography       pkgs/main/linux-64::cryptography-2.8-py37h1ba5d50_0\n",
            "  idna               pkgs/main/linux-64::idna-2.8-py37_0\n",
            "  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7\n",
            "  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0\n",
            "  libffi             pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4\n",
            "  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0\n",
            "  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0\n",
            "  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_0\n",
            "  openssl            pkgs/main/linux-64::openssl-1.1.1d-h7b6447c_4\n",
            "  pip                pkgs/main/linux-64::pip-20.0.2-py37_1\n",
            "  pycosat            pkgs/main/linux-64::pycosat-0.6.3-py37h7b6447c_0\n",
            "  pycparser          pkgs/main/linux-64::pycparser-2.19-py37_0\n",
            "  pyopenssl          pkgs/main/linux-64::pyopenssl-19.1.0-py37_0\n",
            "  pysocks            pkgs/main/linux-64::pysocks-1.7.1-py37_0\n",
            "  python             pkgs/main/linux-64::python-3.7.6-h0371630_2\n",
            "  readline           pkgs/main/linux-64::readline-7.0-h7b6447c_5\n",
            "  requests           pkgs/main/linux-64::requests-2.22.0-py37_1\n",
            "  ruamel_yaml        pkgs/main/linux-64::ruamel_yaml-0.15.87-py37h7b6447c_0\n",
            "  setuptools         pkgs/main/linux-64::setuptools-45.2.0-py37_0\n",
            "  six                pkgs/main/linux-64::six-1.14.0-py37_0\n",
            "  sqlite             pkgs/main/linux-64::sqlite-3.31.1-h7b6447c_0\n",
            "  tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0\n",
            "  tqdm               pkgs/main/noarch::tqdm-4.42.1-py_0\n",
            "  urllib3            pkgs/main/linux-64::urllib3-1.25.8-py37_0\n",
            "  wheel              pkgs/main/linux-64::wheel-0.34.2-py37_0\n",
            "  xz                 pkgs/main/linux-64::xz-5.2.4-h14c3975_4\n",
            "  yaml               pkgs/main/linux-64::yaml-0.1.7-had09818_2\n",
            "  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3\n",
            "\n",
            "\n",
            "Preparing transaction: - \b\b\\ \b\b| \b\b/ \b\bdone\n",
            "Executing transaction: \\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\bdone\n",
            "installation finished.\n",
            "WARNING:\n",
            "    You currently have a PYTHONPATH environment variable set. This may cause\n",
            "    unexpected behavior when running the Python interpreter in Miniconda3.\n",
            "    For best results, please verify that your PYTHONPATH only points to\n",
            "    directories of packages that are compatible with the Python interpreter\n",
            "    in Miniconda3: /usr/local\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SaOrNEf_V0OZ",
        "scrolled": true
      },
      "source": [
        "import os\n",
        "from pathlib import Path\n",
        "import sys\n",
        "sys.path.append('/usr/local/lib/python3.7/site-packages/')\n",
        "from sample.single_video_prediction import get_video_duration"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fHOFsPYdPZNm",
        "outputId": "12972041-92f0-4080-ce8d-03bfb301c2bf"
      },
      "source": [
        "# feature extraction\n",
        "!conda env create -f ./submodules/video_features/conda_env_i3d.yml\n",
        "!conda env create -f ./submodules/video_features/conda_env_vggish.yml\n",
        "# captioning model\n",
        "!conda env create -f ./conda_env.yml\n",
        "# spacy language model\n",
        "!/usr/local/envs/bmt/bin/python -m spacy download en"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting package metadata (repodata.json): - \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\bdone\n",
            "Solving environment: \\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\bdone\n",
            "\n",
            "\n",
            "==> WARNING: A newer version of conda exists. <==\n",
            "  current version: 4.8.2\n",
            "  latest version: 4.10.3\n",
            "\n",
            "Please update conda by running\n",
            "\n",
            "    $ conda update -n base -c defaults conda\n",
            "\n",
            "\n",
            "\n",
            "Downloading and Extracting Packages\n",
            "xz-5.2.5             | 341 KB    | : 100% 1.0/1 [00:00<00:00,  4.13it/s]               \n",
            "zstd-1.3.7           | 401 KB    | : 100% 1.0/1 [00:00<00:00,  8.58it/s]\n",
            "freetype-2.9.1       | 550 KB    | : 100% 1.0/1 [00:00<00:00,  8.20it/s]\n",
            "sqlite-3.31.1        | 1.1 MB    | : 100% 1.0/1 [00:00<00:00,  8.04it/s]\n",
            "mkl-service-2.3.0    | 218 KB    | : 100% 1.0/1 [00:00<00:00,  9.99it/s]\n",
            "mkl_fft-1.0.15       | 154 KB    | : 100% 1.0/1 [00:00<00:00,  9.61it/s]\n",
            "gnutls-3.6.5         | 1.6 MB    | : 100% 1.0/1 [00:00<00:00,  3.84it/s]\n",
            "libtiff-4.1.0        | 447 KB    | : 100% 1.0/1 [00:00<00:00,  8.57it/s]\n",
            "pytorch-1.2.0        | 428.0 MB  | : 100% 1.0/1 [01:05<00:00, 65.54s/it]               \n",
            "readline-8.0         | 356 KB    | : 100% 1.0/1 [00:00<00:00,  3.27it/s]               \n",
            "wheel-0.34.2         | 43 KB     | : 100% 1.0/1 [00:00<00:00,  8.47it/s]\n",
            "ld_impl_linux-64-2.3 | 653 KB    | : 100% 1.0/1 [00:00<00:00,  1.90s/it]                \n",
            "rope-0.17.0          | 127 KB    | : 100% 1.0/1 [00:00<00:00,  9.29it/s]\n",
            "ninja-1.9.0          | 1.2 MB    | : 100% 1.0/1 [00:00<00:00,  6.23it/s]\n",
            "mccabe-0.6.1         | 14 KB     | : 100% 1.0/1 [00:00<00:00, 11.25it/s]\n",
            "certifi-2020.4.5.1   | 155 KB    | : 100% 1.0/1 [00:00<00:00, 11.17it/s]\n",
            "_libgcc_mutex-0.1    | 2 KB      | : 100% 1.0/1 [00:00<00:00, 14.51it/s]\n",
            "tqdm-4.46.0          | 60 KB     | : 100% 1.0/1 [00:00<00:00,  8.99it/s]\n",
            "fastrlock-0.4        | 29 KB     | : 100% 1.0/1 [00:00<00:00, 10.42it/s]\n",
            "importlib-metadata-1 | 49 KB     | : 100% 1.0/1 [00:00<00:00,  9.73it/s]\n",
            "hdf5-1.10.2          | 3.8 MB    | : 100% 1.0/1 [00:00<00:00,  4.16it/s]\n",
            "intel-openmp-2020.1  | 780 KB    | : 100% 1.0/1 [00:00<00:00,  8.77it/s]\n",
            "pycparser-2.20       | 89 KB     | : 100% 1.0/1 [00:00<00:00, 18.69it/s]\n",
            "libpng-1.6.37        | 278 KB    | : 100% 1.0/1 [00:00<00:00,  8.58it/s]\n",
            "autopep8-1.4.4       | 38 KB     | : 100% 1.0/1 [00:00<00:00, 14.56it/s]\n",
            "cudnn-7.3.1          | 286.1 MB  | : 100% 1.0/1 [00:08<00:00, 55.27s/it]               \n",
            "h5py-2.8.0           | 910 KB    | : 100% 1.0/1 [00:00<00:00,  7.00it/s]\n",
            "cudatoolkit-10.0.130 | 261.2 MB  | : 100% 1.0/1 [00:25<00:00, 25.07s/it]               \n",
            "cupy-5.1.0           | 9.3 MB    | : 100% 1.0/1 [00:00<00:00,  2.75it/s]                \n",
            "libgfortran-ng-7.3.0 | 1006 KB   | : 100% 1.0/1 [00:00<00:00,  7.67it/s]\n",
            "blas-1.0             | 1 KB      | : 100% 1.0/1 [00:00<00:00, 21.62it/s]\n",
            "libffi-3.3           | 50 KB     | : 100% 1.0/1 [00:00<00:00, 11.35it/s]\n",
            "gmp-6.1.2            | 514 KB    | : 100% 1.0/1 [00:00<00:00,  4.11it/s]\n",
            "x264-1!152.20180806  | 631 KB    | : 100% 1.0/1 [00:00<00:00,  7.75it/s]\n",
            "nccl-1.3.5           | 1.3 MB    | : 100% 1.0/1 [00:00<00:00,  1.42it/s]\n",
            "openh264-1.8.0       | 659 KB    | : 100% 1.0/1 [00:00<00:00,  7.84it/s]\n",
            "flake8-3.8.2         | 134 KB    | : 100% 1.0/1 [00:00<00:00,  9.62it/s]\n",
            "six-1.14.0           | 23 KB     | : 100% 1.0/1 [00:00<00:00,  9.32it/s]\n",
            "python-3.7.7         | 45.1 MB   | : 100% 1.0/1 [00:09<00:00,  9.25s/it]               \n",
            "cffi-1.14.0          | 224 KB    | : 100% 1.0/1 [00:00<00:00,  3.11it/s]\n",
            "lame-3.100           | 323 KB    | : 100% 1.0/1 [00:00<00:00,  6.69it/s]\n",
            "pillow-6.1.0         | 586 KB    | : 100% 1.0/1 [00:00<00:00,  7.75it/s]\n",
            "libiconv-1.15        | 721 KB    | : 100% 1.0/1 [00:00<00:00,  7.27it/s]\n",
            "pycodestyle-2.6.0    | 43 KB     | : 100% 1.0/1 [00:00<00:00,  9.46it/s]\n",
            "setuptools-46.4.0    | 514 KB    | : 100% 1.0/1 [00:00<00:00,  6.31it/s]\n",
            "olefile-0.46         | 50 KB     | : 100% 1.0/1 [00:00<00:00, 10.19it/s]\n",
            "pyflakes-2.2.0       | 61 KB     | : 100% 1.0/1 [00:00<00:00, 10.68it/s]\n",
            "ffmpeg-4.1.3         | 75.8 MB   | : 100% 1.0/1 [00:10<00:00, 10.82s/it]               \n",
            "jpeg-9b              | 214 KB    | : 100% 1.0/1 [00:00<00:00,  9.35it/s]\n",
            "pip-20.0.2           | 1.7 MB    | : 100% 1.0/1 [00:00<00:00,  3.54it/s]               \n",
            "numpy-1.15.4         | 46 KB     | : 100% 1.0/1 [00:00<00:00, 10.72it/s]\n",
            "openssl-1.1.1g       | 2.5 MB    | : 100% 1.0/1 [00:00<00:00,  3.94it/s]               \n",
            "zipp-3.1.0           | 10 KB     | : 100% 1.0/1 [00:00<00:00, 15.83it/s]\n",
            "mkl_random-1.1.1     | 322 KB    | : 100% 1.0/1 [00:00<00:00,  9.02it/s]\n",
            "mkl-2020.1           | 129.0 MB  | : 100% 1.0/1 [00:19<00:00, 19.54s/it]               \n",
            "bzip2-1.0.8          | 78 KB     | : 100% 1.0/1 [00:00<00:00,  1.39it/s]\n",
            "nettle-3.4.1         | 5.0 MB    | : 100% 1.0/1 [00:00<00:00,  4.09it/s]               \n",
            "numpy-base-1.15.4    | 3.4 MB    | : 100% 1.0/1 [00:00<00:00,  3.01it/s]\n",
            "ncurses-6.2          | 817 KB    | : 100% 1.0/1 [00:01<00:00,  1.61s/it]\n",
            "Preparing transaction: - \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\bdone\n",
            "Verifying transaction: | \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\bdone\n",
            "Executing transaction: / \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\bdone\n",
            "Ran pip subprocess with arguments:\n",
            "['/usr/local/envs/i3d/bin/python', '-m', 'pip', 'install', '-U', '-r', '/content/BMT/submodules/video_features/condaenv.xlglfxk0.requirements.txt']\n",
            "Pip subprocess output:\n",
            "Collecting opencv-python==4.1.0.25\n",
            "  Downloading opencv_python-4.1.0.25-cp37-cp37m-manylinux1_x86_64.whl (26.6 MB)\n",
            "Requirement already satisfied, skipping upgrade: numpy>=1.14.5 in /usr/local/envs/i3d/lib/python3.7/site-packages (from opencv-python==4.1.0.25->-r /content/BMT/submodules/video_features/condaenv.xlglfxk0.requirements.txt (line 1)) (1.15.4)\n",
            "Installing collected packages: opencv-python\n",
            "Successfully installed opencv-python-4.1.0.25\n",
            "\n",
            "#\n",
            "# To activate this environment, use\n",
            "#\n",
            "#     $ conda activate i3d\n",
            "#\n",
            "# To deactivate an active environment, use\n",
            "#\n",
            "#     $ conda deactivate\n",
            "\n",
            "Collecting package metadata (repodata.json): - \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\bdone\n",
            "Solving environment: \\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\bdone\n",
            "\n",
            "\n",
            "==> WARNING: A newer version of conda exists. <==\n",
            "  current version: 4.8.2\n",
            "  latest version: 4.10.3\n",
            "\n",
            "Please update conda by running\n",
            "\n",
            "    $ conda update -n base -c defaults conda\n",
            "\n",
            "\n",
            "\n",
            "Downloading and Extracting Packages\n",
            "six-1.15.0           | 13 KB     | : 100% 1.0/1 [00:00<00:00, 10.45it/s]\n",
            "hdf5-1.10.4          | 5.3 MB    | : 100% 1.0/1 [00:00<00:00,  1.67it/s]                \n",
            "sqlite-3.31.1        | 2.0 MB    | : 100% 1.0/1 [00:00<00:00,  2.49it/s]\n",
            "absl-py-0.9.0        | 162 KB    | : 100% 1.0/1 [00:00<00:00, 12.02it/s]\n",
            "keras-applications-1 | 33 KB     | : 100% 1.0/1 [00:00<00:00, 22.75it/s]\n",
            "scipy-1.4.1          | 18.9 MB   | : 100% 1.0/1 [00:03<00:00,  2.99s/it]               \n",
            "_tflow_select-2.1.0  | 2 KB      | : 100% 1.0/1 [00:00<00:00, 22.46it/s]\n",
            "gmp-6.2.0            | 811 KB    | : 100% 1.0/1 [00:00<00:00,  5.15it/s]\n",
            "werkzeug-1.0.1       | 243 KB    | : 100% 1.0/1 [00:00<00:00, 10.99it/s]\n",
            "mkl-service-2.3.0    | 208 KB    | : 100% 1.0/1 [00:00<00:00,  9.22it/s]\n",
            "ninja-1.9.0          | 1.6 MB    | : 100% 1.0/1 [00:00<00:00,  3.20it/s]\n",
            "libffi-3.3           | 54 KB     | : 100% 1.0/1 [00:00<00:00, 12.10it/s]\n",
            "markdown-3.1.1       | 113 KB    | : 100% 1.0/1 [00:00<00:00, 15.27it/s]\n",
            "cffi-1.14.0          | 226 KB    | : 100% 1.0/1 [00:00<00:00,  9.99it/s]\n",
            "tensorflow-gpu-1.13. | 2 KB      | : 100% 1.0/1 [00:00<00:00, 30.25it/s]\n",
            "grpcio-1.27.2        | 1.4 MB    | : 100% 1.0/1 [00:00<00:00,  3.30it/s]\n",
            "tensorflow-estimator | 205 KB    | : 100% 1.0/1 [00:00<00:00,  9.53it/s]\n",
            "c-ares-1.15.0        | 102 KB    | : 100% 1.0/1 [00:00<00:00, 17.39it/s]\n",
            "openssl-1.1.1g       | 3.8 MB    | : 100% 1.0/1 [00:00<00:00,  1.41it/s]\n",
            "wget-1.20.1          | 894 KB    | : 100% 1.0/1 [00:00<00:00,  4.52it/s]\n",
            "keras-preprocessing- | 36 KB     | : 100% 1.0/1 [00:00<00:00, 23.06it/s]\n",
            "astor-0.8.0          | 45 KB     | : 100% 1.0/1 [00:00<00:00, 19.99it/s]\n",
            "mkl-2019.4           | 204.1 MB  | : 100% 1.0/1 [00:38<00:00, 38.18s/it]               \n",
            "pip-20.0.2           | 1.9 MB    | : 100% 1.0/1 [00:00<00:00,  2.06it/s]\n",
            "tensorflow-base-1.13 | 293.5 MB  | : 100% 1.0/1 [00:58<00:00, 58.91s/it]               \n",
            "libpng-1.6.37        | 308 KB    | : 100% 1.0/1 [00:00<00:00,  9.03it/s]\n",
            "lame-3.100           | 498 KB    | : 100% 1.0/1 [00:00<00:00,  7.97it/s]\n",
            "openh264-1.8.0       | 1.4 MB    | : 100% 1.0/1 [00:00<00:00,  3.95it/s]\n",
            "protobuf-3.11.4      | 711 KB    | : 100% 1.0/1 [00:00<00:00,  4.78it/s]\n",
            "tk-8.6.8             | 3.1 MB    | : 100% 1.0/1 [00:00<00:00,  1.64it/s]\n",
            "h5py-2.10.0          | 1.2 MB    | : 100% 1.0/1 [00:00<00:00,  3.96it/s]\n",
            "gast-0.3.3           | 12 KB     | : 100% 1.0/1 [00:00<00:00, 23.20it/s]\n",
            "cupti-10.0.130       | 1.8 MB    | : 100% 1.0/1 [00:00<00:00,  3.00it/s]\n",
            "ncurses-6.2          | 1.1 MB    | : 100% 1.0/1 [00:00<00:00,  1.56it/s]\n",
            "cudatoolkit-10.0.130 | 380.0 MB  | : 100% 1.0/1 [00:51<00:00, 51.82s/it]               \n",
            "x264-1!152.20180806  | 1.4 MB    | : 100% 1.0/1 [00:00<00:00,  3.62it/s]\n",
            "libedit-3.1.20181209 | 188 KB    | : 100% 1.0/1 [00:00<00:00, 10.08it/s]\n",
            "bzip2-1.0.8          | 396 KB    | : 100% 1.0/1 [00:00<00:00,  9.30it/s]\n",
            "zlib-1.2.11          | 120 KB    | : 100% 1.0/1 [00:00<00:00, 11.86it/s]\n",
            "python_abi-3.7       | 4 KB      | : 100% 1.0/1 [00:00<00:00, 31.62it/s]\n",
            "wheel-0.33.4         | 34 KB     | : 100% 1.0/1 [00:00<00:00, 21.14it/s]\n",
            "readline-8.0         | 428 KB    | : 100% 1.0/1 [00:00<00:00,  6.93it/s]\n",
            "gnutls-3.6.13        | 2.1 MB    | : 100% 1.0/1 [00:00<00:00,  2.02it/s]\n",
            "cudnn-7.6.5          | 226.4 MB  | : 100% 1.0/1 [00:30<00:00, 30.89s/it]               \n",
            "certifi-2020.4.5.1   | 159 KB    | : 100% 1.0/1 [00:00<00:00, 10.21it/s]\n",
            "freetype-2.10.2      | 905 KB    | : 100% 1.0/1 [00:00<00:00,  5.06it/s]\n",
            "tqdm-4.46.0          | 60 KB     | : 100% 1.0/1 [00:00<00:00, 13.13it/s]\n",
            "libstdcxx-ng-9.1.0   | 4.0 MB    | : 100% 1.0/1 [00:00<00:00,  1.36it/s]               \n",
            "libprotobuf-3.11.4   | 4.8 MB    | : 100% 1.0/1 [00:01<00:00,  1.15s/it]               \n",
            "mkl_fft-1.0.15       | 172 KB    | : 100% 1.0/1 [00:00<00:00, 11.31it/s]\n",
            "xz-5.2.5             | 438 KB    | : 100% 1.0/1 [00:00<00:00,  6.63it/s]\n",
            "libiconv-1.15        | 2.0 MB    | : 100% 1.0/1 [00:00<00:00,  2.62it/s]\n",
            "tensorboard-1.13.1   | 3.3 MB    | : 100% 1.0/1 [00:00<00:00,  1.89it/s]\n",
            "python-3.7.7         | 52.6 MB   | : 100% 1.0/1 [00:06<00:00, 13.64s/it]               \n",
            "termcolor-1.1.0      | 7 KB      | : 100% 1.0/1 [00:00<00:00, 26.45it/s]\n",
            "numpy-1.16.4         | 49 KB     | : 100% 1.0/1 [00:00<00:00, 20.93it/s]\n",
            "libgcc-ng-9.1.0      | 8.1 MB    | : 100% 1.0/1 [00:01<00:00,  1.10it/s]               \n",
            "mkl_random-1.1.0     | 376 KB    | : 100% 1.0/1 [00:00<00:00, 10.55it/s]\n",
            "tensorflow-1.13.1    | 4 KB      | : 100% 1.0/1 [00:00<00:00, 33.85it/s]\n",
            "ca-certificates-2020 | 132 KB    | : 100% 1.0/1 [00:00<00:00, 13.95it/s]\n",
            "libgfortran-ng-7.3.0 | 1.3 MB    | : 100% 1.0/1 [00:00<00:00,  3.09it/s]\n",
            "nettle-3.4.1         | 5.7 MB    | : 100% 1.0/1 [00:00<00:00,  1.12it/s]               \n",
            "numpy-base-1.16.4    | 4.4 MB    | : 100% 1.0/1 [00:01<00:00,  1.01s/it]\n",
            "intel-openmp-2020.1  | 940 KB    | : 100% 1.0/1 [00:00<00:00,  3.94it/s]\n",
            "mock-4.0.2           | 31 KB     | : 100% 1.0/1 [00:00<00:00, 21.89it/s]\n",
            "setuptools-47.1.1    | 647 KB    | : 100% 1.0/1 [00:00<00:00,  5.23it/s]\n",
            "Preparing transaction: | \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\bdone\n",
            "Verifying transaction: \\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\bdone\n",
            "Executing transaction: \\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\bdone\n",
            "Ran pip subprocess with arguments:\n",
            "['/usr/local/envs/vggish/bin/python', '-m', 'pip', 'install', '-U', '-r', '/content/BMT/submodules/video_features/condaenv.wi7a_d8o.requirements.txt']\n",
            "Pip subprocess output:\n",
            "Collecting llvmlite==0.32.1\n",
            "  Downloading llvmlite-0.32.1-cp37-cp37m-manylinux1_x86_64.whl (20.2 MB)\n",
            "Collecting numba==0.49.1\n",
            "  Downloading numba-0.49.1-cp37-cp37m-manylinux2014_x86_64.whl (3.6 MB)\n",
            "Collecting pysoundfile==0.9.0.post1\n",
            "  Downloading PySoundFile-0.9.0.post1-py2.py3-none-any.whl (24 kB)\n",
            "Collecting resampy==0.2.2\n",
            "  Downloading resampy-0.2.2.tar.gz (323 kB)\n",
            "Requirement already satisfied, skipping upgrade: setuptools in /usr/local/envs/vggish/lib/python3.7/site-packages (from numba==0.49.1->-r /content/BMT/submodules/video_features/condaenv.wi7a_d8o.requirements.txt (line 2)) (47.1.1.post20200604)\n",
            "Requirement already satisfied, skipping upgrade: numpy>=1.15 in /usr/local/envs/vggish/lib/python3.7/site-packages (from numba==0.49.1->-r /content/BMT/submodules/video_features/condaenv.wi7a_d8o.requirements.txt (line 2)) (1.16.4)\n",
            "Requirement already satisfied, skipping upgrade: cffi>=0.6 in /usr/local/envs/vggish/lib/python3.7/site-packages (from pysoundfile==0.9.0.post1->-r /content/BMT/submodules/video_features/condaenv.wi7a_d8o.requirements.txt (line 3)) (1.14.0)\n",
            "Requirement already satisfied, skipping upgrade: scipy>=0.13 in /usr/local/envs/vggish/lib/python3.7/site-packages (from resampy==0.2.2->-r /content/BMT/submodules/video_features/condaenv.wi7a_d8o.requirements.txt (line 4)) (1.4.1)\n",
            "Requirement already satisfied, skipping upgrade: six>=1.3 in /usr/local/envs/vggish/lib/python3.7/site-packages (from resampy==0.2.2->-r /content/BMT/submodules/video_features/condaenv.wi7a_d8o.requirements.txt (line 4)) (1.15.0)\n",
            "Requirement already satisfied, skipping upgrade: pycparser in /usr/local/envs/vggish/lib/python3.7/site-packages (from cffi>=0.6->pysoundfile==0.9.0.post1->-r /content/BMT/submodules/video_features/condaenv.wi7a_d8o.requirements.txt (line 3)) (2.20)\n",
            "Building wheels for collected packages: resampy\n",
            "  Building wheel for resampy (setup.py): started\n",
            "  Building wheel for resampy (setup.py): finished with status 'done'\n",
            "  Created wheel for resampy: filename=resampy-0.2.2-py3-none-any.whl size=320719 sha256=a79626cd4ad10c02f2e8dae061eb4015b484b5ba137fa8c726078b734979b726\n",
            "  Stored in directory: /root/.cache/pip/wheels/a0/18/0a/8ad18a597d8333a142c9789338a96a6208f1198d290ece356c\n",
            "Successfully built resampy\n",
            "Installing collected packages: llvmlite, numba, pysoundfile, resampy\n",
            "Successfully installed llvmlite-0.32.1 numba-0.49.1 pysoundfile-0.9.0.post1 resampy-0.2.2\n",
            "\n",
            "#\n",
            "# To activate this environment, use\n",
            "#\n",
            "#     $ conda activate vggish\n",
            "#\n",
            "# To deactivate an active environment, use\n",
            "#\n",
            "#     $ conda deactivate\n",
            "\n",
            "Collecting package metadata (repodata.json): - \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\bdone\n",
            "Solving environment: - \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\bdone\n",
            "\n",
            "\n",
            "==> WARNING: A newer version of conda exists. <==\n",
            "  current version: 4.8.2\n",
            "  latest version: 4.10.3\n",
            "\n",
            "Please update conda by running\n",
            "\n",
            "    $ conda update -n base -c defaults conda\n",
            "\n",
            "\n",
            "\n",
            "Downloading and Extracting Packages\n",
            "toolz-0.10.0         | 46 KB     | : 100% 1.0/1 [00:00<00:00,  7.03it/s]               \n",
            "pytz-2020.1          | 239 KB    | : 100% 1.0/1 [00:00<00:00,  6.50it/s]\n",
            "markdown-3.2.1       | 61 KB     | : 100% 1.0/1 [00:00<00:00, 18.55it/s]\n",
            "absl-py-0.8.1        | 158 KB    | : 100% 1.0/1 [00:00<00:00, 12.57it/s]\n",
            "ujson-2.0.3          | 47 KB     | : 100% 1.0/1 [00:00<00:00, 22.11it/s]\n",
            "setuptools-46.1.3    | 656 KB    | : 100% 1.0/1 [00:00<00:00,  5.15it/s]\n",
            "protobuf-3.11.4      | 695 KB    | : 100% 1.0/1 [00:00<00:00,  4.75it/s]\n",
            "future-0.17.1        | 699 KB    | : 100% 1.0/1 [00:00<00:00,  4.44it/s]\n",
            "libprotobuf-3.11.4   | 4.8 MB    | : 100% 1.0/1 [00:00<00:00,  1.00it/s]\n",
            "chardet-3.0.4        | 167 KB    | : 100% 1.0/1 [00:00<00:00,  8.68it/s]\n",
            "pandas-0.24.2        | 11.1 MB   | : 100% 1.0/1 [00:02<00:00,  1.45s/it]               \n",
            "python-3.7.7         | 52.5 MB   | : 100% 1.0/1 [00:06<00:00, 15.19s/it]               \n",
            "werkzeug-1.0.1       | 239 KB    | : 100% 1.0/1 [00:00<00:00, 11.47it/s]\n",
            "pysocks-1.7.1        | 27 KB     | : 100% 1.0/1 [00:00<00:00, 13.24it/s]\n",
            "mkl-service-2.3.0    | 218 KB    | : 100% 1.0/1 [00:00<00:00,  4.96it/s]               \n",
            "mkl_random-1.1.0     | 321 KB    | : 100% 1.0/1 [00:00<00:00,  7.94it/s]\n",
            "python-dateutil-2.8. | 220 KB    | : 100% 1.0/1 [00:00<00:00, 15.97it/s]\n",
            "asn1crypto-1.3.0     | 158 KB    | : 100% 1.0/1 [00:00<00:00,  8.70it/s]\n",
            "msgpack-numpy-0.4.4. | 8 KB      | : 100% 1.0/1 [00:00<00:00, 20.77it/s]\n",
            "thinc-6.10.3         | 1.5 MB    | : 100% 1.0/1 [00:00<00:00,  2.12it/s]\n",
            "cytoolz-0.9.0.1      | 418 KB    | : 100% 1.0/1 [00:00<00:00,  9.20it/s]\n",
            "msgpack-python-0.5.6 | 99 KB     | : 100% 1.0/1 [00:00<00:00, 23.19it/s]\n",
            "cymem-1.31.2         | 26 KB     | : 100% 1.0/1 [00:00<00:00, 25.83it/s]\n",
            "joblib-0.14.1        | 198 KB    | : 100% 1.0/1 [00:00<00:00, 11.06it/s]\n",
            "preshed-1.0.1        | 83 KB     | : 100% 1.0/1 [00:00<00:00, 19.64it/s]\n",
            "tensorboard-1.14.0   | 3.2 MB    | : 100% 1.0/1 [00:00<00:00,  1.91it/s]\n",
            "wrapt-1.10.11        | 45 KB     | : 100% 1.0/1 [00:00<00:00, 25.83it/s]\n",
            "idna-2.9             | 52 KB     | : 100% 1.0/1 [00:00<00:00, 22.09it/s]\n",
            "murmurhash-0.28.0    | 16 KB     | : 100% 1.0/1 [00:00<00:00, 23.70it/s]\n",
            "cryptography-2.8     | 612 KB    | : 100% 1.0/1 [00:00<00:00,  4.31it/s]\n",
            "pip-20.0.2           | 1.9 MB    | : 100% 1.0/1 [00:00<00:00,  2.06it/s]\n",
            "mkl-2020.0           | 202.1 MB  | : 100% 1.0/1 [00:37<00:00, 37.36s/it]               \n",
            "regex-2018.07.11     | 344 KB    | : 100% 1.0/1 [00:00<00:00,  8.97it/s]\n",
            "requests-2.23.0      | 85 KB     | : 100% 1.0/1 [00:00<00:00, 14.27it/s]\n",
            "scikit-learn-0.22.1  | 5.2 MB    | : 100% 1.0/1 [00:00<00:00,  2.19it/s]               \n",
            "cffi-1.14.0          | 225 KB    | : 100% 1.0/1 [00:00<00:00,  8.23it/s]\n",
            "urllib3-1.25.8       | 165 KB    | : 100% 1.0/1 [00:00<00:00,  8.67it/s]\n",
            "mkl_fft-1.0.15       | 154 KB    | : 100% 1.0/1 [00:00<00:00,  7.51it/s]\n",
            "scipy-1.3.1          | 14.0 MB   | : 100% 1.0/1 [00:00<00:00,  1.92it/s]               \n",
            "spacy-2.0.12         | 45.5 MB   | : 100% 1.0/1 [00:09<00:00,  9.16s/it]               \n",
            "dill-0.2.9           | 112 KB    | : 100% 1.0/1 [00:00<00:00, 13.49it/s]\n",
            "openjdk-8.0.152      | 128.5 MB  | : 100% 1.0/1 [00:22<00:00, 22.51s/it]               \n",
            "plac-0.9.6           | 36 KB     | : 100% 1.0/1 [00:00<00:00, 21.97it/s]\n",
            "intel-openmp-2020.0  | 916 KB    | : 100% 1.0/1 [00:00<00:00,  5.87it/s]\n",
            "pyopenssl-19.1.0     | 83 KB     | : 100% 1.0/1 [00:00<00:00,  9.95it/s]\n",
            "Preparing transaction: / \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\bdone\n",
            "Verifying transaction: \\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\bdone\n",
            "Executing transaction: / \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\bdone\n",
            "Ran pip subprocess with arguments:\n",
            "['/usr/local/envs/bmt/bin/python', '-m', 'pip', 'install', '-U', '-r', '/content/BMT/condaenv.je6yz9lw.requirements.txt']\n",
            "Pip subprocess output:\n",
            "Collecting torchtext==0.3.1\n",
            "  Downloading torchtext-0.3.1-py3-none-any.whl (62 kB)\n",
            "Requirement already satisfied, skipping upgrade: numpy in /usr/local/envs/bmt/lib/python3.7/site-packages (from torchtext==0.3.1->-r /content/BMT/condaenv.je6yz9lw.requirements.txt (line 1)) (1.15.4)\n",
            "Requirement already satisfied, skipping upgrade: requests in /usr/local/envs/bmt/lib/python3.7/site-packages (from torchtext==0.3.1->-r /content/BMT/condaenv.je6yz9lw.requirements.txt (line 1)) (2.23.0)\n",
            "Requirement already satisfied, skipping upgrade: torch in /usr/local/envs/bmt/lib/python3.7/site-packages (from torchtext==0.3.1->-r /content/BMT/condaenv.je6yz9lw.requirements.txt (line 1)) (1.2.0)\n",
            "Requirement already satisfied, skipping upgrade: tqdm in /usr/local/envs/bmt/lib/python3.7/site-packages (from torchtext==0.3.1->-r /content/BMT/condaenv.je6yz9lw.requirements.txt (line 1)) (4.46.0)\n",
            "Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /usr/local/envs/bmt/lib/python3.7/site-packages (from requests->torchtext==0.3.1->-r /content/BMT/condaenv.je6yz9lw.requirements.txt (line 1)) (2.9)\n",
            "Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /usr/local/envs/bmt/lib/python3.7/site-packages (from requests->torchtext==0.3.1->-r /content/BMT/condaenv.je6yz9lw.requirements.txt (line 1)) (3.0.4)\n",
            "Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/envs/bmt/lib/python3.7/site-packages (from requests->torchtext==0.3.1->-r /content/BMT/condaenv.je6yz9lw.requirements.txt (line 1)) (1.25.8)\n",
            "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/local/envs/bmt/lib/python3.7/site-packages (from requests->torchtext==0.3.1->-r /content/BMT/condaenv.je6yz9lw.requirements.txt (line 1)) (2020.4.5.1)\n",
            "Installing collected packages: torchtext\n",
            "Successfully installed torchtext-0.3.1\n",
            "\n",
            "#\n",
            "# To activate this environment, use\n",
            "#\n",
            "#     $ conda activate bmt\n",
            "#\n",
            "# To deactivate an active environment, use\n",
            "#\n",
            "#     $ conda deactivate\n",
            "\n",
            "Collecting en_core_web_sm==2.0.0\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz (37.4 MB)\n",
            "\u001b[K     |████████████████████████████████| 37.4 MB 1.4 MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: en-core-web-sm\n",
            "  Building wheel for en-core-web-sm (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for en-core-web-sm: filename=en_core_web_sm-2.0.0-py3-none-any.whl size=37405976 sha256=d4fbc317063fc3336930bcac6419823266b301c9f4df476d3f5508b90079e0c6\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-anlx8kd7/wheels/51/df/77/250d8a622c7fc066a42ea4238279337e4a5e04c2602c448ea5\n",
            "Successfully built en-core-web-sm\n",
            "Installing collected packages: en-core-web-sm\n",
            "Successfully installed en-core-web-sm-2.0.0\n",
            "\n",
            "\u001b[93m    Linking successful\u001b[0m\n",
            "    /usr/local/envs/bmt/lib/python3.7/site-packages/en_core_web_sm -->\n",
            "    /usr/local/envs/bmt/lib/python3.7/site-packages/spacy/data/en\n",
            "\n",
            "    You can now load the model via spacy.load('en')\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "32xPX2aUAKON"
      },
      "source": [
        "## Downloading Checkpoints"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UGXZyrd8PZNo"
      },
      "source": [
        "Downloading the GloVe embeddings. `torchtext` could do it for us but it uses a slow server. \n",
        "Therefore, we store it on our premises."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NrANDrihW5YW",
        "scrolled": false,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9eac797c-525f-4154-bdf2-1c6c79d7e7a0"
      },
      "source": [
        "!wget https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/bmt/glove.840B.300d.zip -q --show-progress\n",
        "!wget https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/bmt/best_cap_model.pt -q --show-progress\n",
        "!wget https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/bmt/best_prop_model.pt -q --show-progress\n",
        "!wget https://storage.googleapis.com/audioset/vggish_model.ckpt -q --show-progress\n",
        "\n",
        "!mkdir .vector_cache\n",
        "!mv glove.840B.300d.zip ./.vector_cache/\n",
        "!mv best_cap_model.pt ./sample/\n",
        "!mv best_prop_model.pt ./sample/\n",
        "!mv vggish_model.ckpt ./submodules/video_features/models/vggish/checkpoints/"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "glove.840B.300d.zip 100%[===================>]   2.03G  20.8MB/s    in 1m 42s  \n",
            "best_cap_model.pt   100%[===================>] 589.61M  21.1MB/s    in 28s     \n",
            "best_prop_model.pt  100%[===================>]   2.50G  21.2MB/s    in 2m 3s   \n",
            "vggish_model.ckpt   100%[===================>] 277.62M   169MB/s    in 1.6s    \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v1ySaHGXPZNp"
      },
      "source": [
        "## Running Inference on a Custom Video"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "puMRtzeDDhWr"
      },
      "source": [
        "Upload your video to Google Colab and select it in the `MY_VIDEO_PATH` variable. \n",
        "The script will prepare all necessary file paths and calculate the video duration for you"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JvTaNVHV_CNo",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "75f3ede1-f199-46d4-bd1a-14d0693f2b6a"
      },
      "source": [
        "# upload a video\n",
        "MY_VIDEO_PATH = '/content/BMT/sample/women_long_jump.mp4'\n",
        "\n",
        "# Preparing the paths\n",
        "VIDEO_DURATION = get_video_duration(MY_VIDEO_PATH)\n",
        "\n",
        "FEATURES_CACHE_PATH = '/content/BMT/tmp/'\n",
        "FEATURES_PATH_STUB = os.path.join(FEATURES_CACHE_PATH, Path(MY_VIDEO_PATH).stem)\n",
        "FEATURE_PATH_VGGISH = f'{FEATURES_PATH_STUB}_vggish.npy'\n",
        "FEATURE_PATH_RGB = f'{FEATURES_PATH_STUB}_rgb.npy'\n",
        "FEATURE_PATH_FLOW = f'{FEATURES_PATH_STUB}_flow.npy'\n",
        "\n",
        "PROPOSAL_CKPT = '/content/BMT/sample/best_prop_model.pt'\n",
        "CAPTIONING_CKPT = '/content/BMT/sample/best_cap_model.pt'"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Video Duration: 35.155\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZIuKhGNc_NXf"
      },
      "source": [
        "The script will extract audio and visual features from the video"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GanvtYi41tc1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2514cc8a-28fc-474e-a05c-94e1dba94b80"
      },
      "source": [
        "# Extract I3D features (visual)\n",
        "!cd ./submodules/video_features && /usr/local/envs/i3d/bin/python main.py \\\n",
        "    --feature_type i3d \\\n",
        "    --on_extraction save_numpy \\\n",
        "    --device_ids 0 \\\n",
        "    --extraction_fps 25 \\\n",
        "    --video_paths $MY_VIDEO_PATH \\\n",
        "    --output_path $FEATURES_CACHE_PATH\n",
        "\n",
        "# Extract VGGish features (audio)\n",
        "!cd ./submodules/video_features && /usr/local/envs/vggish/bin/python main.py \\\n",
        "    --feature_type vggish \\\n",
        "    --on_extraction save_numpy \\\n",
        "    --device_ids 0 \\\n",
        "    --video_paths $MY_VIDEO_PATH \\\n",
        "    --output_path $FEATURES_CACHE_PATH"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving features to /content/BMT/tmp/\n",
            "100% 1/1 [01:24<00:00, 84.85s/it]\n",
            "Saving features to /content/BMT/tmp/\n",
            "\n",
            "WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n",
            "100% 1/1 [00:13<00:00, 13.70s/it]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QvxSQmfH_OKk"
      },
      "source": [
        "Run dense video captioning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "txf8PnKE7Rr8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "648f12bd-924c-4850-c599-e0f8caa5a863"
      },
      "source": [
        "# captioning parameters\n",
        "MAX_PROP_PER_VIDEO = 100\n",
        "NMS_TIOU_THRESHOLD = 0.4\n",
        "\n",
        "# Running single video prediction\n",
        "!/usr/local/envs/bmt/bin/python ./sample/single_video_prediction.py \\\n",
        "    --prop_generator_model_path $PROPOSAL_CKPT \\\n",
        "    --pretrained_cap_model_path $CAPTIONING_CKPT \\\n",
        "    --vggish_features_path $FEATURE_PATH_VGGISH \\\n",
        "    --rgb_features_path $FEATURE_PATH_RGB \\\n",
        "    --flow_features_path $FEATURE_PATH_FLOW \\\n",
        "    --duration_in_secs $VIDEO_DURATION \\\n",
        "    --device_id 0 \\\n",
        "    --max_prop_per_vid $MAX_PROP_PER_VIDEO \\\n",
        "    --nms_tiou_thresh $NMS_TIOU_THRESHOLD"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Contructing caption_iterator for \"train\" phase\n",
            "tcmalloc: large alloc 2635227136 bytes == 0x5636aecea000 @  0x7fc3b863eb6b 0x7fc3b865e379 0x7fc3787826d5 0x7fc37878469a 0x7fc37bc7b163 0x7fc37be30e78 0x7fc37bc7b9f3 0x7fc37bc81e09 0x7fc37bfd48c3 0x7fc3a98908a4 0x56366e5e9ab4 0x56366e5e9bd1 0x56366e65057b 0x56366e595389 0x56366e5e92b7 0x56366e64cb84 0x56366e595389 0x56366e5966ef 0x56366e5b5a73 0x56366e5a7fde 0x56366e64d54d 0x56366e59566a 0x56366e5966ef 0x56366e5b5a73 0x56366e5fd1ba 0x56366e5fdd17 0x56366e5a7fde 0x56366e6a5297 0x56366e5a7fde 0x56366e64d54d 0x56366e595389\n",
            "100% 2196016/2196017 [04:30<00:00, 8104.37it/s]\n",
            "Using vanilla Generator\n",
            "initialization: xavier\n",
            "Glove emb of the same size as d_model_caps\n",
            "Pretrained caption path: \n",
            " /content/BMT/sample/best_cap_model.pt\n",
            "[{'start': 0.1, 'end': 4.9, 'sentence': 'We see a title screen'}, {'start': 5.0, 'end': 7.9, 'sentence': 'A large group of people are seen standing around a building'}, {'start': 0.7, 'end': 11.9, 'sentence': 'A man is seen standing in front of a large crowd'}, {'start': 19.6, 'end': 33.3, 'sentence': 'The woman runs down a track and jumps into a sand pit'}, {'start': 7.5, 'end': 10.0, 'sentence': 'A large group of people are seen standing around a building'}, {'start': 0.6, 'end': 35.1, 'sentence': 'A large group of people are seen running down a track while others watch on the sides'}, {'start': 8.2, 'end': 13.7, 'sentence': 'A man runs down a track'}, {'start': 0.1, 'end': 2.0, 'sentence': 'We see a title screen'}]\n"
          ]
        }
      ]
    }
  ]
}