NLI_TR_Hypothesis_only_baseline.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "NLI-TR / Hypothesis only baseline.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "USJybXYKS0y0"
      },
      "source": [
        "# NLI-TR / Hypothesis only baseline\n",
        "\n",
        "In this experimentation, we want to show the use of NLI-TR dataset on a particular use case where we would like to answer the main question: \n",
        "\n",
        "> *What is the difference between hypothesis-only baseline and the actual models when we fine-tune off-the-shelf models on NLI-TR.*\n",
        "\n",
        "Thanks **Lasha Abzianidze** for this insightful question in our Gather.town session at EMNLP 2020! \n",
        "\n",
        "*Disclaimer*: The code is mostly based on the examples in the following repositories and the documentation of Huggingface Datasets and Transformers.\n",
        "\n",
        "*   https://github.com/huggingface/transformers\n",
        "*   https://github.com/huggingface/datasets\n",
        "*   https://github.com/cgpotts/cs224u"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tW-efngmSTOS"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ATizxfSScM68",
        "outputId": "625065ee-616b-41e2-8a5a-b8947824f65e"
      },
      "source": [
        "!pip install datasets\n",
        "!pip install transformers"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting datasets\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/94/f8/ff7cd6e3b400b33dcbbfd31c6c1481678a2b2f669f521ad20053009a9aa3/datasets-1.7.0-py3-none-any.whl (234kB)\n",
            "\u001b[K     |████████████████████████████████| 235kB 7.4MB/s \n",
            "\u001b[?25hCollecting fsspec\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/bc/52/816d1a3a599176057bf29dfacb1f8fadb61d35fbd96cb1bab4aaa7df83c0/fsspec-2021.5.0-py3-none-any.whl (111kB)\n",
            "\u001b[K     |████████████████████████████████| 112kB 12.9MB/s \n",
            "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from datasets) (20.9)\n",
            "Requirement already satisfied: dill in /usr/local/lib/python3.7/dist-packages (from datasets) (0.3.3)\n",
            "Collecting xxhash\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7d/4f/0a862cad26aa2ed7a7cd87178cbbfa824fc1383e472d63596a0d018374e7/xxhash-2.0.2-cp37-cp37m-manylinux2010_x86_64.whl (243kB)\n",
            "\u001b[K     |████████████████████████████████| 245kB 12.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from datasets) (4.0.1)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from datasets) (1.19.5)\n",
            "Requirement already satisfied: pyarrow<4.0.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from datasets) (3.0.0)\n",
            "Collecting huggingface-hub<0.1.0\n",
            "  Downloading https://files.pythonhosted.org/packages/32/a1/7c5261396da23ec364e296a4fb8a1cd6a5a2ff457215c6447038f18c0309/huggingface_hub-0.0.9-py3-none-any.whl\n",
            "Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.7/dist-packages (from datasets) (2.23.0)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from datasets) (1.1.5)\n",
            "Requirement already satisfied: multiprocess in /usr/local/lib/python3.7/dist-packages (from datasets) (0.70.11.1)\n",
            "Requirement already satisfied: tqdm<4.50.0,>=4.27 in /usr/local/lib/python3.7/dist-packages (from datasets) (4.41.1)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->datasets) (2.4.7)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->datasets) (3.4.1)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->datasets) (3.7.4.3)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from huggingface-hub<0.1.0->datasets) (3.0.12)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->datasets) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->datasets) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->datasets) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->datasets) (2020.12.5)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->datasets) (2.8.1)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->datasets) (2018.9)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas->datasets) (1.15.0)\n",
            "Installing collected packages: fsspec, xxhash, huggingface-hub, datasets\n",
            "Successfully installed datasets-1.7.0 fsspec-2021.5.0 huggingface-hub-0.0.9 xxhash-2.0.2\n",
            "Collecting transformers\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d5/43/cfe4ee779bbd6a678ac6a97c5a5cdeb03c35f9eaebbb9720b036680f9a2d/transformers-4.6.1-py3-none-any.whl (2.2MB)\n",
            "\u001b[K     |████████████████████████████████| 2.3MB 7.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from transformers) (4.0.1)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (1.19.5)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2019.12.20)\n",
            "Collecting sacremoses\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/75/ee/67241dc87f266093c533a2d4d3d69438e57d7a90abb216fa076e7d475d4a/sacremoses-0.0.45-py3-none-any.whl (895kB)\n",
            "\u001b[K     |████████████████████████████████| 901kB 45.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers) (4.41.1)\n",
            "Collecting huggingface-hub==0.0.8\n",
            "  Downloading https://files.pythonhosted.org/packages/a1/88/7b1e45720ecf59c6c6737ff332f41c955963090a18e72acbcbeac6b25e86/huggingface_hub-0.0.8-py3-none-any.whl\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers) (2.23.0)\n",
            "Collecting tokenizers<0.11,>=0.10.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d4/e2/df3543e8ffdab68f5acc73f613de9c2b155ac47f162e725dcac87c521c11/tokenizers-0.10.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.3MB)\n",
            "\u001b[K     |████████████████████████████████| 3.3MB 46.9MB/s \n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers) (3.0.12)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers) (20.9)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers) (3.4.1)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers) (3.7.4.3)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.15.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.0.1)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2020.12.5)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers) (2.4.7)\n",
            "Installing collected packages: sacremoses, huggingface-hub, tokenizers, transformers\n",
            "  Found existing installation: huggingface-hub 0.0.9\n",
            "    Uninstalling huggingface-hub-0.0.9:\n",
            "      Successfully uninstalled huggingface-hub-0.0.9\n",
            "Successfully installed huggingface-hub-0.0.8 sacremoses-0.0.45 tokenizers-0.10.3 transformers-4.6.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AbZrcy1oDRVq"
      },
      "source": [
        "import transformers\n",
        "import datasets"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RkDI82WWSXd6"
      },
      "source": [
        "## Dataset readers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PgEJRvd5b9Vx"
      },
      "source": [
        "import torch\n",
        "from datasets import load_dataset\n",
        "\n",
        "class NLITRReader(torch.utils.data.Dataset):\n",
        "  def __init__(self, dataset_name, split_name, max_example_num=-1):\n",
        "    self.dataset = load_dataset('nli_tr', dataset_name)\n",
        "    self.split_name = split_name\n",
        "    self.max_example_num = max_example_num\n",
        "\n",
        "  def read(self):\n",
        "      count = 0\n",
        "      for example in self.dataset[self.split_name]:\n",
        "          if example['label'] == -1: # skip examples having no gold value.\n",
        "              continue\n",
        "          count += 1\n",
        "          if self.max_example_num > 0 and count >= self.max_example_num:\n",
        "             break\n",
        "          yield example"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cAyDuR_MvUJ5"
      },
      "source": [
        "import torch\n",
        "class NLITRDataset(torch.utils.data.Dataset):\n",
        "    def __init__(self, encodings, labels):\n",
        "        self.encodings = encodings\n",
        "        self.labels = labels\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}\n",
        "        item['labels'] = torch.tensor(self.labels[idx])\n",
        "        return item\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.labels)"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CFgjsCUySdrZ"
      },
      "source": [
        "## Trainer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5G3ZUNbIwqsQ"
      },
      "source": [
        "import torch\n",
        "import pandas as pd\n",
        "from transformers import TrainingArguments, Trainer, AutoConfig, AutoTokenizer, AutoModelForSequenceClassification\n",
        "\n",
        "from sklearn.metrics import accuracy_score, precision_recall_fscore_support\n",
        "\n",
        "def compute_metrics(pred):\n",
        "    labels = pred.label_ids\n",
        "    preds = pred.predictions.argmax(-1)\n",
        "    acc = accuracy_score(labels, preds)\n",
        "    return {\n",
        "        'accuracy': acc,\n",
        "    }\n",
        "\n",
        "MAX_TRAIN_EXAMPLE_NUM = -1\n",
        "MAX_EVALUATION_EXAMPLE_NUM = -1\n",
        "class NLITRTrainer():\n",
        "    def __init__(self, \n",
        "                 model_name='bert-base-cased', \n",
        "                 dataset_name='snli_tr',\n",
        "                 evaluation_split='validation',\n",
        "                 num_labels=3, \n",
        "                 hypothesis_only=False):\n",
        "        self.model_name = model_name\n",
        "        self.dataset_name = dataset_name\n",
        "        self.evaluation_split = evaluation_split\n",
        "        self.hypothesis_only = hypothesis_only\n",
        "        self.max_train_example_num = MAX_TRAIN_EXAMPLE_NUM\n",
        "        self.max_evaluation_example_num = MAX_EVALUATION_EXAMPLE_NUM\n",
        "\n",
        "        print('You can set the values of the following parameters via the global variables MAX_TRAIN_EXAMPLE_NUM and MAX_EVALUATION_EXAMPLE_NUM (-1 to use all examples in the splits)')\n",
        "        print('max_train_example_num',self.max_train_example_num)\n",
        "        print('max_evaluation_example_num',self.max_evaluation_example_num)\n",
        "        self.prepare_for_training()\n",
        "    \n",
        "    def prepare_for_training(self):\n",
        "        self.prepare_model()\n",
        "        self.prepare_datasets()\n",
        "        self.prepare_trainer()\n",
        "\n",
        "    def prepare_model(self):\n",
        "        self.tokenizer = AutoTokenizer.from_pretrained(self.model_name)\n",
        "        self.config = AutoConfig.from_pretrained(self.model_name, num_labels=3)\n",
        "        self.model = AutoModelForSequenceClassification.from_pretrained(self.model_name, config=self.config)\n",
        "        \n",
        "    def get_dataset(self, split_name, max_example_num):\n",
        "        df = pd.DataFrame(list(NLITRReader(dataset_name=self.dataset_name, split_name=split_name, max_example_num=max_example_num).read()))\n",
        "        labels = df['label'].values.tolist()\n",
        "        premises = df['premise'].values.tolist()\n",
        "        if self.hypothesis_only:\n",
        "            input = self.tokenizer(premises, truncation=True, padding=True)\n",
        "        else:\n",
        "            hypotheses = df['hypothesis'].values.tolist()\n",
        "            input = self.tokenizer(premises, hypotheses, truncation=True, padding=True)\n",
        "        \n",
        "        dataset = NLITRDataset(input, labels)\n",
        "        return dataset\n",
        "\n",
        "    def prepare_datasets(self):\n",
        "        self.train_dataset = self.get_dataset('train', max_example_num=self.max_train_example_num)\n",
        "        self.evaluation_dataset = self.get_dataset(self.evaluation_split, max_example_num=self.max_evaluation_example_num)\n",
        "      \n",
        "    def prepare_trainer(self):\n",
        "        training_args = TrainingArguments(\n",
        "            output_dir='./results',          # output directory\n",
        "            num_train_epochs=3,              # total number of training epochs\n",
        "            per_device_train_batch_size=16,   # batch size per device during training\n",
        "            per_device_eval_batch_size=4,   # batch size for evaluation\n",
        "            gradient_accumulation_steps=32,  # gradient accumulation steps to increase effective batch size on GPU.\n",
        "            warmup_steps=500,                # number of warmup steps for learning rate scheduler\n",
        "            weight_decay=0.01,               # strength of weight decay\n",
        "            logging_dir='./logs',            # directory for storing logs\n",
        "            logging_steps=10\n",
        "        )\n",
        "\n",
        "        self.trainer = Trainer(\n",
        "            model=self.model,                         # the instantiated 🤗 Transformers model to be trained\n",
        "            args=training_args,                       # training arguments, defined above\n",
        "            train_dataset=self.train_dataset,         # training dataset\n",
        "            eval_dataset=self.evaluation_dataset,     # evaluation dataset,\n",
        "            compute_metrics=compute_metrics\n",
        "        )\n",
        "\n",
        "    def train(self):\n",
        "        train_results = self.trainer.train()\n",
        "        return train_results\n",
        "    \n",
        "    def evaluate(self):\n",
        "        eval_results = self.trainer.evaluate()\n",
        "        return eval_results"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ekaCaOhSihS"
      },
      "source": [
        "## Experiment Manager\n",
        "\n",
        "This is a simple experiment manager that runs a series of experiments with the given set of hyperparameters and returns the resulting metrics."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-Jv2sHrI-0pq"
      },
      "source": [
        "import copy\n",
        "import numpy as np\n",
        "import random\n",
        "\n",
        "class NLIExperiment:\n",
        "    def __init__(self, experiment_parameters, seed=1234):\n",
        "        self.experiment_parameters = experiment_parameters\n",
        "        self.set_random_seed(seed)\n",
        "    \n",
        "    def set_random_seed(self, seed):\n",
        "        np.random.seed(seed)\n",
        "        random.seed(seed)\n",
        "        torch.manual_seed(seed)\n",
        "    \n",
        "    def run(self):\n",
        "        experiment_results = []\n",
        "        \n",
        "        for model_name in self.experiment_parameters['model_names']:\n",
        "            experiment_parameters = {}\n",
        "            experiment_parameters['model_name'] = model_name\n",
        "\n",
        "            for dataset_name, evaliation_split_names in self.experiment_parameters['dataset_info'].items():\n",
        "                experiment_parameters['dataset_name'] = dataset_name\n",
        "            \n",
        "                for evaliation_split_name in evaliation_split_names:\n",
        "                    experiment_parameters['evaliation_split_name'] = evaliation_split_name\n",
        "\n",
        "                    for param_key, param_values in self.experiment_parameters['params'].items():\n",
        "                        \n",
        "                        for param_value in param_values:\n",
        "                              experiment_parameters[param_key] = param_value\n",
        "                              print('\\n\\nA new experiment started...')\n",
        "                              nlitr_trainer = NLITRTrainer(model_name=model_name, dataset_name=dataset_name, evaluation_split=evaliation_split_name, **{param_key:param_value})\n",
        "\n",
        "                              print('Training...')\n",
        "                              train_results = nlitr_trainer.train()\n",
        "                              print('Evaluating...')\n",
        "                              eval_results = nlitr_trainer.evaluate()\n",
        "                              \n",
        "                              experiment_parameters.update(eval_results)\n",
        "                              print('\\nexperiment parameters:', experiment_parameters)\n",
        "                              print('experiment results:', eval_results)\n",
        "                              experiment_results.append(copy.deepcopy(experiment_parameters))\n",
        "        return experiment_results"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QvfoSU3CSsI-"
      },
      "source": [
        "## Experiments\n",
        "\n",
        "Below is a set of sample parameters to get a sense of how the results will look like.  You may execute the code with alternative sets of parameters to get a deeper understanding of the difference between the hypothesis-only baseline and the full models under different conditions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 916
        },
        "id": "r-BMWSOkzrWA",
        "outputId": "1d26e606-056b-4a21-c938-648aa99a8346"
      },
      "source": [
        "%%time\n",
        "# You may also experiment with some alternative values denoted as comment.\n",
        "experiment_parameters = {\n",
        "    'model_names' : ['dbmdz/bert-base-turkish-cased'], #alternative values: 'model_names' : ['bert-base-cased', 'bert-base-multilingual-cased', 'dbmdz/bert-base-turkish-cased'] \n",
        "    'dataset_info' : {'snli_tr': ['validation', 'test']},   #alternative values: {'snli_tr': ['validation', 'test'], 'multinli_tr': ['validation_matched', 'validation_mismatched']}\n",
        "    'params' : {'hypothesis_only': [False]}\n",
        "}\n",
        "\n",
        "#You may set different values for the size of training and evaluation splits for fast iterations (-1 to use all examples in the splits). \n",
        "MAX_TRAIN_EXAMPLE_NUM = -1\n",
        "MAX_EVALUATION_EXAMPLE_NUM = -1\n",
        "\n",
        "experiment = NLIExperiment(experiment_parameters)\n",
        "experiment_result = experiment.run()"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "A new experiment started...\n",
            "You can set the values of the following parameters via the global variables MAX_TRAIN_EXAMPLE_NUM and MAX_EVALUATION_EXAMPLE_NUM (-1 to use all examples in the splits)\n",
            "max_train_example_num 2048\n",
            "max_evaluation_example_num 512\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at dbmdz/bert-base-turkish-cased were not used when initializing BertForSequenceClassification: ['cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias']\n",
            "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at dbmdz/bert-base-turkish-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
            "Reusing dataset nli_tr (/root/.cache/huggingface/datasets/nli_tr/snli_tr/1.0.0/c2ddd0c0a70caddac6a81c2dae5ca7939f00060d517d08f1983927818dba6521)\n",
            "Reusing dataset nli_tr (/root/.cache/huggingface/datasets/nli_tr/snli_tr/1.0.0/c2ddd0c0a70caddac6a81c2dae5ca7939f00060d517d08f1983927818dba6521)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Training...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='12' max='12' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [12/12 01:03, Epoch 3/3]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>Step</th>\n",
              "      <th>Training Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>10</td>\n",
              "      <td>1.143100</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Evaluating...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='128' max='128' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [128/128 00:03]\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "experiment parameters: {'model_name': 'dbmdz/bert-base-turkish-cased', 'dataset_name': 'snli_tr', 'evaliation_split_name': 'validation', 'hypothesis_only': False, 'eval_loss': 1.1216628551483154, 'eval_accuracy': 0.28180039138943247, 'eval_runtime': 3.724, 'eval_samples_per_second': 137.219, 'epoch': 3.0, 'eval_mem_cpu_alloc_delta': 0, 'eval_mem_gpu_alloc_delta': 0, 'eval_mem_cpu_peaked_delta': 0, 'eval_mem_gpu_peaked_delta': 10937856}\n",
            "experiment results: {'eval_loss': 1.1216628551483154, 'eval_accuracy': 0.28180039138943247, 'eval_runtime': 3.724, 'eval_samples_per_second': 137.219, 'epoch': 3.0, 'eval_mem_cpu_alloc_delta': 0, 'eval_mem_gpu_alloc_delta': 0, 'eval_mem_cpu_peaked_delta': 0, 'eval_mem_gpu_peaked_delta': 10937856}\n",
            "\n",
            "\n",
            "A new experiment started...\n",
            "You can set the values of the following parameters via the global variables MAX_TRAIN_EXAMPLE_NUM and MAX_EVALUATION_EXAMPLE_NUM (-1 to use all examples in the splits)\n",
            "max_train_example_num 2048\n",
            "max_evaluation_example_num 512\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at dbmdz/bert-base-turkish-cased were not used when initializing BertForSequenceClassification: ['cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias']\n",
            "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at dbmdz/bert-base-turkish-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
            "Reusing dataset nli_tr (/root/.cache/huggingface/datasets/nli_tr/snli_tr/1.0.0/c2ddd0c0a70caddac6a81c2dae5ca7939f00060d517d08f1983927818dba6521)\n",
            "Reusing dataset nli_tr (/root/.cache/huggingface/datasets/nli_tr/snli_tr/1.0.0/c2ddd0c0a70caddac6a81c2dae5ca7939f00060d517d08f1983927818dba6521)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Training...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='12' max='12' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [12/12 01:03, Epoch 3/3]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>Step</th>\n",
              "      <th>Training Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>10</td>\n",
              "      <td>1.164600</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Evaluating...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='128' max='128' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [128/128 00:03]\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "experiment parameters: {'model_name': 'dbmdz/bert-base-turkish-cased', 'dataset_name': 'snli_tr', 'evaliation_split_name': 'test', 'hypothesis_only': False, 'eval_loss': 1.1589947938919067, 'eval_accuracy': 0.273972602739726, 'eval_runtime': 3.7754, 'eval_samples_per_second': 135.35, 'epoch': 3.0, 'eval_mem_cpu_alloc_delta': 0, 'eval_mem_gpu_alloc_delta': 0, 'eval_mem_cpu_peaked_delta': 0, 'eval_mem_gpu_peaked_delta': 9088512}\n",
            "experiment results: {'eval_loss': 1.1589947938919067, 'eval_accuracy': 0.273972602739726, 'eval_runtime': 3.7754, 'eval_samples_per_second': 135.35, 'epoch': 3.0, 'eval_mem_cpu_alloc_delta': 0, 'eval_mem_gpu_alloc_delta': 0, 'eval_mem_cpu_peaked_delta': 0, 'eval_mem_gpu_peaked_delta': 9088512}\n",
            "CPU times: user 3min 54s, sys: 43.7 s, total: 4min 37s\n",
            "Wall time: 2min 38s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PBI-BHp6SvWs"
      },
      "source": [
        "## Results\n",
        "\n",
        "And, here is the results 🙂 \n",
        "\n",
        "It should be noted that these results are obtained using only a fraction of the dataset splits due to the time limitation.  Please feel free to play with the global parameters MAX_TRAIN_EXAMPLE_NUM and MAX_EVALUATION_EXAMPLE_NUM (as explained above) to use a wider portion (or all). Playing with these parameters will help get a deeper understanding on the resulting difference between hypothesis-oly baseline and full models."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        },
        "id": "psW1sHKsMG1P",
        "outputId": "5ba9fc7f-ecb7-42ea-f48e-86a6824f179b"
      },
      "source": [
        "experiment_result_df = pd.DataFrame(experiment_result)\n",
        "experiment_result_df.head(n=100) #show all dataframe"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>model_name</th>\n",
              "      <th>dataset_name</th>\n",
              "      <th>evaliation_split_name</th>\n",
              "      <th>hypothesis_only</th>\n",
              "      <th>eval_loss</th>\n",
              "      <th>eval_accuracy</th>\n",
              "      <th>epoch</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>dbmdz/bert-base-turkish-cased</td>\n",
              "      <td>snli_tr</td>\n",
              "      <td>validation</td>\n",
              "      <td>True</td>\n",
              "      <td>1.122703</td>\n",
              "      <td>0.322896</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>dbmdz/bert-base-turkish-cased</td>\n",
              "      <td>snli_tr</td>\n",
              "      <td>validation</td>\n",
              "      <td>False</td>\n",
              "      <td>1.091679</td>\n",
              "      <td>0.412916</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>dbmdz/bert-base-turkish-cased</td>\n",
              "      <td>multinli_tr</td>\n",
              "      <td>validation_matched</td>\n",
              "      <td>True</td>\n",
              "      <td>1.105768</td>\n",
              "      <td>0.340509</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>dbmdz/bert-base-turkish-cased</td>\n",
              "      <td>multinli_tr</td>\n",
              "      <td>validation_matched</td>\n",
              "      <td>False</td>\n",
              "      <td>1.105668</td>\n",
              "      <td>0.369863</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                      model_name dataset_name  ... eval_accuracy  epoch\n",
              "0  dbmdz/bert-base-turkish-cased      snli_tr  ...      0.322896    1.0\n",
              "1  dbmdz/bert-base-turkish-cased      snli_tr  ...      0.412916    1.0\n",
              "2  dbmdz/bert-base-turkish-cased  multinli_tr  ...      0.340509    1.0\n",
              "3  dbmdz/bert-base-turkish-cased  multinli_tr  ...      0.369863    1.0\n",
              "\n",
              "[4 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    }
  ]
}