Mistral_Nemo_Instruct_2407_GGUF.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Troyanovsky/Local-LLM-Comparison-Colab-UI/blob/main/Mistral_Nemo_Instruct_2407_GGUF.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XNkpBtLvuTOp"
      },
      "source": [
        "## Mistral-Nemo-Instruct-2407-GGUF WebUI\n",
        "\n",
        "1. Run the following cell, takes ~5 min\n",
        "(You may need to confirm to proceed by typing \"Y\")\n",
        "2. Pick the version you need from one of the last two cells and only run that cell.\n",
        "3. Click the gradio link at the bottom\n",
        "4. In Parameter settings - Instruction Template:\n",
        "\n",
        "```\n",
        "<s>[INST] {user_message_1} [/INST]{assistant_message_1}</s>[INST] {user_message_2} [/INST]{assistant_message_2}</s>\n",
        "```\n",
        "\n",
        "Original model: https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407\n",
        "\n",
        "Quantized model: https://huggingface.co/second-state/Mistral-Nemo-Instruct-2407-GGUF\n",
        "\n",
        "Want to try other local LLMs? Check out this repo: https://github.com/Troyanovsky/Local-LLM-Comparison-Colab-UI/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "D-MiHp_bveP6",
        "outputId": "31fefc55-e936-4f0c-f6e6-f287842dbdd8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content\n",
            "Selecting previously unselected package libc-ares2:amd64.\n",
            "(Reading database ... 123589 files and directories currently installed.)\n",
            "Preparing to unpack .../libc-ares2_1.18.1-1ubuntu0.22.04.3_amd64.deb ...\n",
            "Unpacking libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.3) ...\n",
            "Selecting previously unselected package libaria2-0:amd64.\n",
            "Preparing to unpack .../libaria2-0_1.36.0-1_amd64.deb ...\n",
            "Unpacking libaria2-0:amd64 (1.36.0-1) ...\n",
            "Selecting previously unselected package aria2.\n",
            "Preparing to unpack .../aria2_1.36.0-1_amd64.deb ...\n",
            "Unpacking aria2 (1.36.0-1) ...\n",
            "Setting up libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.3) ...\n",
            "Setting up libaria2-0:amd64 (1.36.0-1) ...\n",
            "Setting up aria2 (1.36.0-1) ...\n",
            "Processing triggers for man-db (2.10.2-1) ...\n",
            "Processing triggers for libc-bin (2.35-0ubuntu3.4) ...\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link\n",
            "\n",
            "Cloning into 'text-generation-webui'...\n",
            "remote: Enumerating objects: 18617, done.\u001b[K\n",
            "remote: Counting objects: 100% (253/253), done.\u001b[K\n",
            "remote: Compressing objects: 100% (139/139), done.\u001b[K\n",
            "remote: Total 18617 (delta 169), reused 182 (delta 114), pack-reused 18364\u001b[K\n",
            "Receiving objects: 100% (18617/18617), 27.76 MiB | 19.68 MiB/s, done.\n",
            "Resolving deltas: 100% (13175/13175), done.\n",
            "Note: switching to 'af839d20acee2710022d735d2d1dc9cb48696f73'.\n",
            "\n",
            "You are in 'detached HEAD' state. You can look around, make experimental\n",
            "changes and commit them, and you can discard any commits you make in this\n",
            "state without impacting any branches by switching back to a branch.\n",
            "\n",
            "If you want to create a new branch to retain commits you create, you may\n",
            "do so (now or later) by using -c with the switch command. Example:\n",
            "\n",
            "  git switch -c <new-branch-name>\n",
            "\n",
            "Or undo this operation with:\n",
            "\n",
            "  git switch -\n",
            "\n",
            "Turn off this advice by setting config variable advice.detachedHead to false\n",
            "\n",
            "/content/text-generation-webui\n",
            "Ignoring llama-cpp-python: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting llama-cpp-python==0.2.83+cpuavx2 (from -r requirements.txt (line 39))\n",
            "  Downloading https://github.com/oobabooga/llama-cpp-python-cuBLAS-wheels/releases/download/cpu/llama_cpp_python-0.2.83+cpuavx2-cp310-cp310-linux_x86_64.whl (2.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.8/2.8 MB\u001b[0m \u001b[31m41.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring llama-cpp-python: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring llama-cpp-python: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting llama-cpp-python-cuda==0.2.83+cu121 (from -r requirements.txt (line 47))\n",
            "  Downloading https://github.com/oobabooga/llama-cpp-python-cuBLAS-wheels/releases/download/textgen-webui/llama_cpp_python_cuda-0.2.83+cu121-cp310-cp310-linux_x86_64.whl (397.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m397.4/397.4 MB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring llama-cpp-python-cuda-tensorcores: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda-tensorcores: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda-tensorcores: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting llama-cpp-python-cuda-tensorcores==0.2.83+cu121 (from -r requirements.txt (line 53))\n",
            "  Downloading https://github.com/oobabooga/llama-cpp-python-cuBLAS-wheels/releases/download/textgen-webui/llama_cpp_python_cuda_tensorcores-0.2.83+cu121-cp310-cp310-linux_x86_64.whl (434.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m434.0/434.0 MB\u001b[0m \u001b[31m3.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring exllamav2: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring exllamav2: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring exllamav2: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting exllamav2==0.1.7+cu121.torch2.2.2 (from -r requirements.txt (line 59))\n",
            "  Downloading https://github.com/oobabooga/exllamav2/releases/download/v0.1.7/exllamav2-0.1.7+cu121.torch2.2.2-cp310-cp310-linux_x86_64.whl (135.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.6/135.6 MB\u001b[0m \u001b[31m7.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring exllamav2: markers 'platform_system == \"Linux\" and platform_machine != \"x86_64\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting flash-attn==2.6.1+cu123torch2.2cxx11abiFALSE (from -r requirements.txt (line 64))\n",
            "  Downloading https://github.com/Dao-AILab/flash-attention/releases/download/v2.6.1/flash_attn-2.6.1+cu123torch2.2cxx11abiFALSE-cp310-cp310-linux_x86_64.whl (198.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m198.4/198.4 MB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: accelerate==0.32.* in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 1)) (0.32.1)\n",
            "Collecting aqlm==1.1.6 (from aqlm[cpu,gpu]==1.1.6->-r requirements.txt (line 2))\n",
            "  Downloading aqlm-1.1.6-py3-none-any.whl.metadata (1.7 kB)\n",
            "Collecting auto-gptq==0.7.1 (from -r requirements.txt (line 3))\n",
            "  Downloading auto_gptq-0.7.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (18 kB)\n",
            "Collecting bitsandbytes==0.43.* (from -r requirements.txt (line 4))\n",
            "  Downloading bitsandbytes-0.43.2-py3-none-manylinux_2_24_x86_64.whl.metadata (3.5 kB)\n",
            "Collecting colorama (from -r requirements.txt (line 5))\n",
            "  Downloading colorama-0.4.6-py2.py3-none-any.whl.metadata (17 kB)\n",
            "Collecting datasets (from -r requirements.txt (line 6))\n",
            "  Downloading datasets-2.20.0-py3-none-any.whl.metadata (19 kB)\n",
            "Collecting einops (from -r requirements.txt (line 7))\n",
            "  Downloading einops-0.8.0-py3-none-any.whl.metadata (12 kB)\n",
            "Collecting gradio==4.26.* (from -r requirements.txt (line 8))\n",
            "  Downloading gradio-4.26.0-py3-none-any.whl.metadata (15 kB)\n",
            "Collecting hqq==0.1.7.post3 (from -r requirements.txt (line 9))\n",
            "  Downloading hqq-0.1.7.post3.tar.gz (50 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.3/50.3 kB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: jinja2==3.1.4 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 10)) (3.1.4)\n",
            "Collecting lm_eval==0.3.0 (from -r requirements.txt (line 11))\n",
            "  Downloading lm_eval-0.3.0-py3-none-any.whl.metadata (6.8 kB)\n",
            "Requirement already satisfied: markdown in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 12)) (3.6)\n",
            "Collecting numba==0.59.* (from -r requirements.txt (line 13))\n",
            "  Downloading numba-0.59.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.7 kB)\n",
            "Collecting numpy==1.26.* (from -r requirements.txt (line 14))\n",
            "  Downloading numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.0/61.0 kB\u001b[0m \u001b[31m5.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting optimum==1.17.* (from -r requirements.txt (line 15))\n",
            "  Downloading optimum-1.17.1-py3-none-any.whl.metadata (18 kB)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 16)) (2.0.3)\n",
            "Collecting peft==0.8.* (from -r requirements.txt (line 17))\n",
            "  Downloading peft-0.8.2-py3-none-any.whl.metadata (25 kB)\n",
            "Collecting Pillow>=9.5.0 (from -r requirements.txt (line 18))\n",
            "  Downloading pillow-10.4.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (9.2 kB)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 19)) (5.9.5)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 20)) (6.0.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 21)) (2.31.0)\n",
            "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 22)) (13.7.1)\n",
            "Requirement already satisfied: safetensors==0.4.* in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 23)) (0.4.3)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 24)) (1.11.4)\n",
            "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 25)) (0.1.99)\n",
            "Requirement already satisfied: tensorboard in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 26)) (2.15.2)\n",
            "Requirement already satisfied: transformers==4.42.* in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 27)) (4.42.4)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 28)) (4.66.4)\n",
            "Collecting wandb (from -r requirements.txt (line 29))\n",
            "  Downloading wandb-0.17.5-py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (10 kB)\n",
            "Collecting SpeechRecognition==3.10.0 (from -r requirements.txt (line 32))\n",
            "  Downloading SpeechRecognition-3.10.0-py2.py3-none-any.whl.metadata (28 kB)\n",
            "Collecting flask_cloudflared==0.0.14 (from -r requirements.txt (line 33))\n",
            "  Downloading flask_cloudflared-0.0.14-py3-none-any.whl.metadata (4.6 kB)\n",
            "Collecting sse-starlette==1.6.5 (from -r requirements.txt (line 34))\n",
            "  Downloading sse_starlette-1.6.5-py3-none-any.whl.metadata (6.7 kB)\n",
            "Collecting tiktoken (from -r requirements.txt (line 35))\n",
            "  Downloading tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.32.*->-r requirements.txt (line 1)) (24.1)\n",
            "Requirement already satisfied: torch>=1.10.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.32.*->-r requirements.txt (line 1)) (2.3.1+cu121)\n",
            "Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (from accelerate==0.32.*->-r requirements.txt (line 1)) (0.23.5)\n",
            "Collecting rouge (from auto-gptq==0.7.1->-r requirements.txt (line 3))\n",
            "  Downloading rouge-1.0.1-py3-none-any.whl.metadata (4.1 kB)\n",
            "Collecting gekko (from auto-gptq==0.7.1->-r requirements.txt (line 3))\n",
            "  Downloading gekko-1.2.1-py3-none-any.whl.metadata (3.0 kB)\n",
            "Collecting aiofiles<24.0,>=22.0 (from gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading aiofiles-23.2.1-py3-none-any.whl.metadata (9.7 kB)\n",
            "Requirement already satisfied: altair<6.0,>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 8)) (4.2.2)\n",
            "Collecting fastapi (from gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading fastapi-0.111.1-py3-none-any.whl.metadata (26 kB)\n",
            "Collecting ffmpy (from gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading ffmpy-0.3.2.tar.gz (5.5 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting gradio-client==0.15.1 (from gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading gradio_client-0.15.1-py3-none-any.whl.metadata (7.1 kB)\n",
            "Collecting httpx>=0.24.1 (from gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading httpx-0.27.0-py3-none-any.whl.metadata (7.2 kB)\n",
            "Requirement already satisfied: importlib-resources<7.0,>=1.3 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 8)) (6.4.0)\n",
            "Requirement already satisfied: markupsafe~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 8)) (2.1.5)\n",
            "Requirement already satisfied: matplotlib~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 8)) (3.7.1)\n",
            "Collecting orjson~=3.0 (from gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading orjson-3.10.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (50 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.4/50.4 kB\u001b[0m \u001b[31m4.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pydantic>=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 8)) (2.8.2)\n",
            "Collecting pydub (from gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading pydub-0.25.1-py2.py3-none-any.whl.metadata (1.4 kB)\n",
            "Collecting python-multipart>=0.0.9 (from gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading python_multipart-0.0.9-py3-none-any.whl.metadata (2.5 kB)\n",
            "Collecting ruff>=0.2.2 (from gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading ruff-0.5.4-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (24 kB)\n",
            "Collecting semantic-version~=2.0 (from gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading semantic_version-2.10.0-py2.py3-none-any.whl.metadata (9.7 kB)\n",
            "Collecting tomlkit==0.12.0 (from gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading tomlkit-0.12.0-py3-none-any.whl.metadata (2.7 kB)\n",
            "Requirement already satisfied: typer<1.0,>=0.9 in /usr/local/lib/python3.10/dist-packages (from typer[all]<1.0,>=0.9; sys_platform != \"emscripten\"->gradio==4.26.*->-r requirements.txt (line 8)) (0.12.3)\n",
            "Requirement already satisfied: typing-extensions~=4.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.*->-r requirements.txt (line 8)) (4.12.2)\n",
            "Collecting uvicorn>=0.14.0 (from gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading uvicorn-0.30.3-py3-none-any.whl.metadata (6.5 kB)\n",
            "Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from hqq==0.1.7.post3->-r requirements.txt (line 9)) (2.4.0)\n",
            "Collecting jsonlines (from lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading jsonlines-4.0.0-py3-none-any.whl.metadata (1.6 kB)\n",
            "Requirement already satisfied: numexpr in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.3.0->-r requirements.txt (line 11)) (2.10.1)\n",
            "Collecting openai>=0.6.4 (from lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading openai-1.37.0-py3-none-any.whl.metadata (22 kB)\n",
            "Collecting pybind11>=2.6.2 (from lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading pybind11-2.13.1-py3-none-any.whl.metadata (9.5 kB)\n",
            "Collecting pycountry (from lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading pycountry-24.6.1-py3-none-any.whl.metadata (12 kB)\n",
            "Collecting pytablewriter (from lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading pytablewriter-1.2.0-py3-none-any.whl.metadata (37 kB)\n",
            "Collecting rouge-score>=0.0.4 (from lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading rouge_score-0.1.2.tar.gz (17 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting sacrebleu==1.5.0 (from lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading sacrebleu-1.5.0-py3-none-any.whl.metadata (1.3 kB)\n",
            "Requirement already satisfied: scikit-learn>=0.24.1 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.3.0->-r requirements.txt (line 11)) (1.2.2)\n",
            "Collecting sqlitedict (from lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading sqlitedict-2.1.0.tar.gz (21 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting tqdm-multiprocess (from lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading tqdm_multiprocess-0.0.11-py3-none-any.whl.metadata (5.7 kB)\n",
            "Collecting zstandard (from lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading zstandard-0.23.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)\n",
            "Collecting llvmlite<0.43,>=0.42.0dev0 (from numba==0.59.*->-r requirements.txt (line 13))\n",
            "  Downloading llvmlite-0.42.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.8 kB)\n",
            "Collecting coloredlogs (from optimum==1.17.*->-r requirements.txt (line 15))\n",
            "  Downloading coloredlogs-15.0.1-py2.py3-none-any.whl.metadata (12 kB)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from optimum==1.17.*->-r requirements.txt (line 15)) (1.13.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.42.*->-r requirements.txt (line 27)) (3.15.4)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.42.*->-r requirements.txt (line 27)) (2024.5.15)\n",
            "Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers==4.42.*->-r requirements.txt (line 27)) (0.19.1)\n",
            "Requirement already satisfied: Flask>=0.8 in /usr/local/lib/python3.10/dist-packages (from flask_cloudflared==0.0.14->-r requirements.txt (line 33)) (2.2.5)\n",
            "Collecting starlette (from sse-starlette==1.6.5->-r requirements.txt (line 34))\n",
            "  Downloading starlette-0.38.1-py3-none-any.whl.metadata (5.9 kB)\n",
            "Requirement already satisfied: triton>=2.1 in /usr/local/lib/python3.10/dist-packages (from aqlm[cpu,gpu]==1.1.6->-r requirements.txt (line 2)) (2.3.1)\n",
            "Collecting ninja (from aqlm[cpu,gpu]==1.1.6->-r requirements.txt (line 2))\n",
            "  Downloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio-client==0.15.1->gradio==4.26.*->-r requirements.txt (line 8)) (2023.6.0)\n",
            "Collecting websockets<12.0,>=10.0 (from gradio-client==0.15.1->gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading websockets-11.0.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)\n",
            "Collecting portalocker (from sacrebleu==1.5.0->lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading portalocker-2.10.1-py3-none-any.whl.metadata (8.5 kB)\n",
            "Collecting pyarrow>=15.0.0 (from datasets->-r requirements.txt (line 6))\n",
            "  Downloading pyarrow-17.0.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (3.3 kB)\n",
            "Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 6)) (0.6)\n",
            "Collecting dill<0.3.9,>=0.3.0 (from datasets->-r requirements.txt (line 6))\n",
            "  Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB)\n",
            "Collecting requests (from -r requirements.txt (line 21))\n",
            "  Downloading requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)\n",
            "Collecting xxhash (from datasets->-r requirements.txt (line 6))\n",
            "  Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n",
            "Collecting multiprocess (from datasets->-r requirements.txt (line 6))\n",
            "  Downloading multiprocess-0.70.16-py310-none-any.whl.metadata (7.2 kB)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 6)) (3.9.5)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 16)) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 16)) (2023.4)\n",
            "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 16)) (2024.1)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 21)) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 21)) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 21)) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 21)) (2024.7.4)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->-r requirements.txt (line 22)) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->-r requirements.txt (line 22)) (2.16.1)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 26)) (1.4.0)\n",
            "Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 26)) (1.64.1)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 26)) (2.27.0)\n",
            "Requirement already satisfied: google-auth-oauthlib<2,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 26)) (1.2.1)\n",
            "Requirement already satisfied: protobuf!=4.24.0,>=3.19.6 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 26)) (3.20.3)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 26)) (71.0.4)\n",
            "Requirement already satisfied: six>1.9 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 26)) (1.16.0)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 26)) (0.7.2)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 26)) (3.0.3)\n",
            "Requirement already satisfied: click!=8.0.0,>=7.1 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 29)) (8.1.7)\n",
            "Collecting docker-pycreds>=0.4.0 (from wandb->-r requirements.txt (line 29))\n",
            "  Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl.metadata (1.8 kB)\n",
            "Collecting gitpython!=3.1.29,>=1.0.0 (from wandb->-r requirements.txt (line 29))\n",
            "  Downloading GitPython-3.1.43-py3-none-any.whl.metadata (13 kB)\n",
            "Requirement already satisfied: platformdirs in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 29)) (4.2.2)\n",
            "Collecting sentry-sdk>=1.0.0 (from wandb->-r requirements.txt (line 29))\n",
            "  Downloading sentry_sdk-2.11.0-py2.py3-none-any.whl.metadata (14 kB)\n",
            "Collecting setproctitle (from wandb->-r requirements.txt (line 29))\n",
            "  Downloading setproctitle-1.3.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.9 kB)\n",
            "Collecting diskcache>=5.6.1 (from llama-cpp-python==0.2.83+cpuavx2->-r requirements.txt (line 39))\n",
            "  Downloading diskcache-5.6.3-py3-none-any.whl.metadata (20 kB)\n",
            "Collecting fastparquet (from exllamav2==0.1.7+cu121.torch2.2.2->-r requirements.txt (line 59))\n",
            "  Downloading fastparquet-2024.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB)\n",
            "Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 8)) (0.4)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 8)) (4.19.2)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 8)) (0.12.1)\n",
            "Requirement already satisfied: itsdangerous>=2.0 in /usr/local/lib/python3.10/dist-packages (from Flask>=0.8->flask_cloudflared==0.0.14->-r requirements.txt (line 33)) (2.2.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 6)) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 6)) (23.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 6)) (1.4.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 6)) (6.0.5)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 6)) (1.9.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 6)) (4.0.3)\n",
            "Collecting gitdb<5,>=4.0.1 (from gitpython!=3.1.29,>=1.0.0->wandb->-r requirements.txt (line 29))\n",
            "  Downloading gitdb-4.0.11-py3-none-any.whl.metadata (1.2 kB)\n",
            "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 26)) (5.4.0)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 26)) (0.4.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 26)) (4.9)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<2,>=0.5->tensorboard->-r requirements.txt (line 26)) (1.3.1)\n",
            "Requirement already satisfied: anyio in /usr/local/lib/python3.10/dist-packages (from httpx>=0.24.1->gradio==4.26.*->-r requirements.txt (line 8)) (3.7.1)\n",
            "Collecting httpcore==1.* (from httpx>=0.24.1->gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading httpcore-1.0.5-py3-none-any.whl.metadata (20 kB)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx>=0.24.1->gradio==4.26.*->-r requirements.txt (line 8)) (1.3.1)\n",
            "Collecting h11<0.15,>=0.13 (from httpcore==1.*->httpx>=0.24.1->gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading h11-0.14.0-py3-none-any.whl.metadata (8.2 kB)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich->-r requirements.txt (line 22)) (0.1.2)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.*->-r requirements.txt (line 8)) (1.2.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.*->-r requirements.txt (line 8)) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.*->-r requirements.txt (line 8)) (4.53.1)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.*->-r requirements.txt (line 8)) (1.4.5)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.*->-r requirements.txt (line 8)) (3.1.2)\n",
            "Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai>=0.6.4->lm_eval==0.3.0->-r requirements.txt (line 11)) (1.7.0)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2.0->gradio==4.26.*->-r requirements.txt (line 8)) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.20.1 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2.0->gradio==4.26.*->-r requirements.txt (line 8)) (2.20.1)\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from rouge-score>=0.0.4->lm_eval==0.3.0->-r requirements.txt (line 11)) (3.8.1)\n",
            "Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm_eval==0.3.0->-r requirements.txt (line 11)) (1.4.2)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm_eval==0.3.0->-r requirements.txt (line 11)) (3.5.0)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.32.*->-r requirements.txt (line 1)) (3.3)\n",
            "Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch>=1.10.0->accelerate==0.32.*->-r requirements.txt (line 1))\n",
            "  Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch>=1.10.0->accelerate==0.32.*->-r requirements.txt (line 1))\n",
            "  Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch>=1.10.0->accelerate==0.32.*->-r requirements.txt (line 1))\n",
            "  Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch>=1.10.0->accelerate==0.32.*->-r requirements.txt (line 1))\n",
            "  Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cublas-cu12==12.1.3.1 (from torch>=1.10.0->accelerate==0.32.*->-r requirements.txt (line 1))\n",
            "  Using cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cufft-cu12==11.0.2.54 (from torch>=1.10.0->accelerate==0.32.*->-r requirements.txt (line 1))\n",
            "  Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-curand-cu12==10.3.2.106 (from torch>=1.10.0->accelerate==0.32.*->-r requirements.txt (line 1))\n",
            "  Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch>=1.10.0->accelerate==0.32.*->-r requirements.txt (line 1))\n",
            "  Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch>=1.10.0->accelerate==0.32.*->-r requirements.txt (line 1))\n",
            "  Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-nccl-cu12==2.20.5 (from torch>=1.10.0->accelerate==0.32.*->-r requirements.txt (line 1))\n",
            "  Using cached nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl.metadata (1.8 kB)\n",
            "Collecting nvidia-nvtx-cu12==12.1.105 (from torch>=1.10.0->accelerate==0.32.*->-r requirements.txt (line 1))\n",
            "  Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)\n",
            "Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.10.0->accelerate==0.32.*->-r requirements.txt (line 1))\n",
            "  Downloading nvidia_nvjitlink_cu12-12.5.82-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)\n",
            "Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != \"emscripten\"->gradio==4.26.*->-r requirements.txt (line 8)) (1.5.4)\n",
            "\u001b[33mWARNING: typer 0.12.3 does not provide the extra 'all'\u001b[0m\u001b[33m\n",
            "\u001b[0mCollecting humanfriendly>=9.1 (from coloredlogs->optimum==1.17.*->-r requirements.txt (line 15))\n",
            "  Downloading humanfriendly-10.0-py2.py3-none-any.whl.metadata (9.2 kB)\n",
            "Collecting starlette (from sse-starlette==1.6.5->-r requirements.txt (line 34))\n",
            "  Downloading starlette-0.37.2-py3-none-any.whl.metadata (5.9 kB)\n",
            "Collecting fastapi-cli>=0.0.2 (from fastapi->gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading fastapi_cli-0.0.4-py3-none-any.whl.metadata (7.0 kB)\n",
            "Collecting email_validator>=2.0.0 (from fastapi->gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading email_validator-2.2.0-py3-none-any.whl.metadata (25 kB)\n",
            "Collecting cramjam>=2.3 (from fastparquet->exllamav2==0.1.7+cu121.torch2.2.2->-r requirements.txt (line 59))\n",
            "  Downloading cramjam-2.8.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)\n",
            "Collecting DataProperty<2,>=1.0.1 (from pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading DataProperty-1.0.1-py3-none-any.whl.metadata (11 kB)\n",
            "Collecting mbstrdecoder<2,>=1.0.0 (from pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading mbstrdecoder-1.1.3-py3-none-any.whl.metadata (4.0 kB)\n",
            "Collecting pathvalidate<4,>=2.3.0 (from pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading pathvalidate-3.2.0-py3-none-any.whl.metadata (11 kB)\n",
            "Collecting tabledata<2,>=1.3.1 (from pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading tabledata-1.3.3-py3-none-any.whl.metadata (3.7 kB)\n",
            "Collecting tcolorpy<1,>=0.0.5 (from pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading tcolorpy-0.1.6-py3-none-any.whl.metadata (6.4 kB)\n",
            "Collecting typepy<2,>=1.3.2 (from typepy[datetime]<2,>=1.3.2->pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 11))\n",
            "  Downloading typepy-1.3.2-py3-none-any.whl.metadata (9.3 kB)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->optimum==1.17.*->-r requirements.txt (line 15)) (1.3.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio->httpx>=0.24.1->gradio==4.26.*->-r requirements.txt (line 8)) (1.2.2)\n",
            "Collecting dnspython>=2.0.0 (from email_validator>=2.0.0->fastapi->gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading dnspython-2.6.1-py3-none-any.whl.metadata (5.8 kB)\n",
            "Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->gitpython!=3.1.29,>=1.0.0->wandb->-r requirements.txt (line 29))\n",
            "  Downloading smmap-5.0.1-py3-none-any.whl.metadata (4.3 kB)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 8)) (2023.12.1)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 8)) (0.35.1)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==4.26.*->-r requirements.txt (line 8)) (0.19.0)\n",
            "Requirement already satisfied: chardet<6,>=3.0.4 in /usr/local/lib/python3.10/dist-packages (from mbstrdecoder<2,>=1.0.0->pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 11)) (5.2.0)\n",
            "Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 26)) (0.6.0)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<2,>=0.5->tensorboard->-r requirements.txt (line 26)) (3.2.2)\n",
            "Collecting httptools>=0.5.0 (from uvicorn[standard]>=0.12.0->fastapi->gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading httptools-0.6.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.6 kB)\n",
            "Collecting python-dotenv>=0.13 (from uvicorn[standard]>=0.12.0->fastapi->gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB)\n",
            "Collecting uvloop!=0.15.0,!=0.15.1,>=0.14.0 (from uvicorn[standard]>=0.12.0->fastapi->gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading uvloop-0.19.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)\n",
            "Collecting watchfiles>=0.13 (from uvicorn[standard]>=0.12.0->fastapi->gradio==4.26.*->-r requirements.txt (line 8))\n",
            "  Downloading watchfiles-0.22.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)\n",
            "Downloading aqlm-1.1.6-py3-none-any.whl (14 kB)\n",
            "Downloading auto_gptq-0.7.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.5 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m23.5/23.5 MB\u001b[0m \u001b[31m83.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading bitsandbytes-0.43.2-py3-none-manylinux_2_24_x86_64.whl (137.5 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m137.5/137.5 MB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading gradio-4.26.0-py3-none-any.whl (17.1 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m17.1/17.1 MB\u001b[0m \u001b[31m96.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading lm_eval-0.3.0-py3-none-any.whl (178 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m178.7/178.7 kB\u001b[0m \u001b[31m18.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading numba-0.59.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.7 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.7/3.7 MB\u001b[0m \u001b[31m93.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m18.2/18.2 MB\u001b[0m \u001b[31m96.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading optimum-1.17.1-py3-none-any.whl (407 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m407.1/407.1 kB\u001b[0m \u001b[31m33.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading peft-0.8.2-py3-none-any.whl (183 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m183.4/183.4 kB\u001b[0m \u001b[31m18.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading SpeechRecognition-3.10.0-py2.py3-none-any.whl (32.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m32.8/32.8 MB\u001b[0m \u001b[31m16.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading flask_cloudflared-0.0.14-py3-none-any.whl (6.4 kB)\n",
            "Downloading sse_starlette-1.6.5-py3-none-any.whl (9.6 kB)\n",
            "Downloading gradio_client-0.15.1-py3-none-any.whl (313 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m313.6/313.6 kB\u001b[0m \u001b[31m30.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading sacrebleu-1.5.0-py3-none-any.whl (65 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m65.6/65.6 kB\u001b[0m \u001b[31m7.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading tomlkit-0.12.0-py3-none-any.whl (37 kB)\n",
            "Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n",
            "Downloading datasets-2.20.0-py3-none-any.whl (547 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m547.8/547.8 kB\u001b[0m \u001b[31m39.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading einops-0.8.0-py3-none-any.whl (43 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.2/43.2 kB\u001b[0m \u001b[31m4.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pillow-10.4.0-cp310-cp310-manylinux_2_28_x86_64.whl (4.5 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.5/4.5 MB\u001b[0m \u001b[31m106.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading requests-2.32.3-py3-none-any.whl (64 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m64.9/64.9 kB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading wandb-0.17.5-py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.8/6.8 MB\u001b[0m \u001b[31m113.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m56.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading aiofiles-23.2.1-py3-none-any.whl (15 kB)\n",
            "Downloading dill-0.3.8-py3-none-any.whl (116 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m12.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading diskcache-5.6.3-py3-none-any.whl (45 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.5/45.5 kB\u001b[0m \u001b[31m4.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)\n",
            "Downloading GitPython-3.1.43-py3-none-any.whl (207 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m207.3/207.3 kB\u001b[0m \u001b[31m21.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading httpx-0.27.0-py3-none-any.whl (75 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading httpcore-1.0.5-py3-none-any.whl (77 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.9/77.9 kB\u001b[0m \u001b[31m8.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading llvmlite-0.42.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (43.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.8/43.8 MB\u001b[0m \u001b[31m14.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading openai-1.37.0-py3-none-any.whl (337 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m337.0/337.0 kB\u001b[0m \u001b[31m31.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading orjson-3.10.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (141 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m141.1/141.1 kB\u001b[0m \u001b[31m15.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pyarrow-17.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (39.9 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m39.9/39.9 MB\u001b[0m \u001b[31m19.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pybind11-2.13.1-py3-none-any.whl (238 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m238.8/238.8 kB\u001b[0m \u001b[31m22.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading python_multipart-0.0.9-py3-none-any.whl (22 kB)\n",
            "Downloading ruff-0.5.4-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.1 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.1/10.1 MB\u001b[0m \u001b[31m105.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)\n",
            "Downloading sentry_sdk-2.11.0-py2.py3-none-any.whl (303 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m303.6/303.6 kB\u001b[0m \u001b[31m28.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hUsing cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)\n",
            "Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)\n",
            "Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)\n",
            "Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)\n",
            "Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)\n",
            "Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)\n",
            "Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)\n",
            "Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)\n",
            "Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)\n",
            "Using cached nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl (176.2 MB)\n",
            "Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)\n",
            "Downloading uvicorn-0.30.3-py3-none-any.whl (62 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.8/62.8 kB\u001b[0m \u001b[31m84.3 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading websockets-11.0.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (129 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m129.9/129.9 kB\u001b[0m \u001b[31m14.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m4.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading fastapi-0.111.1-py3-none-any.whl (92 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m92.2/92.2 kB\u001b[0m \u001b[31m10.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading starlette-0.37.2-py3-none-any.whl (71 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m71.9/71.9 kB\u001b[0m \u001b[31m7.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading fastparquet-2024.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m77.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading gekko-1.2.1-py3-none-any.whl (13.2 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.2/13.2 MB\u001b[0m \u001b[31m95.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading jsonlines-4.0.0-py3-none-any.whl (8.7 kB)\n",
            "Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m14.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m307.2/307.2 kB\u001b[0m \u001b[31m30.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pycountry-24.6.1-py3-none-any.whl (6.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.3/6.3 MB\u001b[0m \u001b[31m99.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)\n",
            "Downloading pytablewriter-1.2.0-py3-none-any.whl (111 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.1/111.1 kB\u001b[0m \u001b[31m12.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading rouge-1.0.1-py3-none-any.whl (13 kB)\n",
            "Downloading setproctitle-1.3.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30 kB)\n",
            "Downloading tqdm_multiprocess-0.0.11-py3-none-any.whl (9.8 kB)\n",
            "Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m16.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading zstandard-0.23.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.4/5.4 MB\u001b[0m \u001b[31m103.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading cramjam-2.8.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m68.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading DataProperty-1.0.1-py3-none-any.whl (27 kB)\n",
            "Downloading email_validator-2.2.0-py3-none-any.whl (33 kB)\n",
            "Downloading fastapi_cli-0.0.4-py3-none-any.whl (9.5 kB)\n",
            "Downloading gitdb-4.0.11-py3-none-any.whl (62 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 kB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m5.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.8/86.8 kB\u001b[0m \u001b[31m7.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading mbstrdecoder-1.1.3-py3-none-any.whl (7.8 kB)\n",
            "Downloading pathvalidate-3.2.0-py3-none-any.whl (23 kB)\n",
            "Downloading tabledata-1.3.3-py3-none-any.whl (11 kB)\n",
            "Downloading tcolorpy-0.1.6-py3-none-any.whl (8.1 kB)\n",
            "Downloading typepy-1.3.2-py3-none-any.whl (31 kB)\n",
            "Downloading portalocker-2.10.1-py3-none-any.whl (18 kB)\n",
            "Downloading dnspython-2.6.1-py3-none-any.whl (307 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m307.7/307.7 kB\u001b[0m \u001b[31m30.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading httptools-0.6.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (341 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m341.4/341.4 kB\u001b[0m \u001b[31m34.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)\n",
            "Downloading smmap-5.0.1-py3-none-any.whl (24 kB)\n",
            "Downloading uvloop-0.19.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.4/3.4 MB\u001b[0m \u001b[31m66.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading watchfiles-0.22.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m51.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_nvjitlink_cu12-12.5.82-py3-none-manylinux2014_x86_64.whl (21.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.3/21.3 MB\u001b[0m \u001b[31m58.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hBuilding wheels for collected packages: hqq, rouge-score, ffmpy, sqlitedict\n",
            "  Building wheel for hqq (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for hqq: filename=hqq-0.1.7.post3-py3-none-any.whl size=59171 sha256=10c37e3a9fe25694f58f5d8bf5de2d2342ec3972826202a1f7e4f584f8d1ab34\n",
            "  Stored in directory: /root/.cache/pip/wheels/e6/48/52/64097324429cfc0c8e47c763b060a132484cdb0a99483ecf82\n",
            "  Building wheel for rouge-score (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for rouge-score: filename=rouge_score-0.1.2-py3-none-any.whl size=24934 sha256=a1d7fde0fcd70586bac57e5233abbd8a058e2563cd92b6df6d2a25831c7c0d67\n",
            "  Stored in directory: /root/.cache/pip/wheels/5f/dd/89/461065a73be61a532ff8599a28e9beef17985c9e9c31e541b4\n",
            "  Building wheel for ffmpy (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ffmpy: filename=ffmpy-0.3.2-py3-none-any.whl size=5584 sha256=134c9fee02ce38c485799d280e075e755acfeaca0fd46301cd8c3c284f3e2476\n",
            "  Stored in directory: /root/.cache/pip/wheels/bd/65/9a/671fc6dcde07d4418df0c592f8df512b26d7a0029c2a23dd81\n",
            "  Building wheel for sqlitedict (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sqlitedict: filename=sqlitedict-2.1.0-py3-none-any.whl size=16863 sha256=9d54995095657a5e4b7dc944178b010c042a9e6347793372fbfdff0365694f91\n",
            "  Stored in directory: /root/.cache/pip/wheels/79/d6/e7/304e0e6cb2221022c26d8161f7c23cd4f259a9e41e8bbcfabd\n",
            "Successfully built hqq rouge-score ffmpy sqlitedict\n",
            "Installing collected packages: sqlitedict, pydub, ninja, ffmpy, zstandard, xxhash, websockets, uvloop, tomlkit, tcolorpy, smmap, setproctitle, sentry-sdk, semantic-version, ruff, rouge, requests, python-multipart, python-dotenv, pycountry, pybind11, portalocker, Pillow, pathvalidate, orjson, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, numpy, mbstrdecoder, llvmlite, jsonlines, humanfriendly, httptools, h11, einops, docker-pycreds, dnspython, diskcache, dill, cramjam, colorama, aiofiles, watchfiles, uvicorn, typepy, tqdm-multiprocess, tiktoken, starlette, SpeechRecognition, sacrebleu, rouge-score, pyarrow, nvidia-cusparse-cu12, nvidia-cudnn-cu12, numba, multiprocess, llama-cpp-python-cuda-tensorcores, llama-cpp-python-cuda, llama-cpp-python, httpcore, gitdb, gekko, email_validator, coloredlogs, sse-starlette, nvidia-cusolver-cu12, httpx, gitpython, flask_cloudflared, fastparquet, wandb, openai, gradio-client, fastapi-cli, exllamav2, datasets, DataProperty, tabledata, flash-attn, fastapi, bitsandbytes, pytablewriter, peft, optimum, hqq, gradio, aqlm, lm_eval, auto-gptq\n",
            "  Attempting uninstall: requests\n",
            "    Found existing installation: requests 2.31.0\n",
            "    Uninstalling requests-2.31.0:\n",
            "      Successfully uninstalled requests-2.31.0\n",
            "  Attempting uninstall: Pillow\n",
            "    Found existing installation: Pillow 9.4.0\n",
            "    Uninstalling Pillow-9.4.0:\n",
            "      Successfully uninstalled Pillow-9.4.0\n",
            "  Attempting uninstall: numpy\n",
            "    Found existing installation: numpy 1.25.2\n",
            "    Uninstalling numpy-1.25.2:\n",
            "      Successfully uninstalled numpy-1.25.2\n",
            "  Attempting uninstall: llvmlite\n",
            "    Found existing installation: llvmlite 0.41.1\n",
            "    Uninstalling llvmlite-0.41.1:\n",
            "      Successfully uninstalled llvmlite-0.41.1\n",
            "  Attempting uninstall: pyarrow\n",
            "    Found existing installation: pyarrow 14.0.2\n",
            "    Uninstalling pyarrow-14.0.2:\n",
            "      Successfully uninstalled pyarrow-14.0.2\n",
            "  Attempting uninstall: numba\n",
            "    Found existing installation: numba 0.58.1\n",
            "    Uninstalling numba-0.58.1:\n",
            "      Successfully uninstalled numba-0.58.1\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "cudf-cu12 24.4.1 requires pyarrow<15.0.0a0,>=14.0.1, but you have pyarrow 17.0.0 which is incompatible.\n",
            "google-colab 1.0.0 requires requests==2.31.0, but you have requests 2.32.3 which is incompatible.\n",
            "ibis-framework 8.0.0 requires pyarrow<16,>=2, but you have pyarrow 17.0.0 which is incompatible.\n",
            "imageio 2.31.6 requires pillow<10.1.0,>=8.3.2, but you have pillow 10.4.0 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed DataProperty-1.0.1 Pillow-10.4.0 SpeechRecognition-3.10.0 aiofiles-23.2.1 aqlm-1.1.6 auto-gptq-0.7.1 bitsandbytes-0.43.2 colorama-0.4.6 coloredlogs-15.0.1 cramjam-2.8.3 datasets-2.20.0 dill-0.3.8 diskcache-5.6.3 dnspython-2.6.1 docker-pycreds-0.4.0 einops-0.8.0 email_validator-2.2.0 exllamav2-0.1.7+cu121.torch2.2.2 fastapi-0.111.1 fastapi-cli-0.0.4 fastparquet-2024.5.0 ffmpy-0.3.2 flash-attn-2.6.1 flask_cloudflared-0.0.14 gekko-1.2.1 gitdb-4.0.11 gitpython-3.1.43 gradio-4.26.0 gradio-client-0.15.1 h11-0.14.0 hqq-0.1.7.post3 httpcore-1.0.5 httptools-0.6.1 httpx-0.27.0 humanfriendly-10.0 jsonlines-4.0.0 llama-cpp-python-0.2.83+cpuavx2 llama-cpp-python-cuda-0.2.83+cu121 llama-cpp-python-cuda-tensorcores-0.2.83+cu121 llvmlite-0.42.0 lm_eval-0.3.0 mbstrdecoder-1.1.3 multiprocess-0.70.16 ninja-1.11.1.1 numba-0.59.1 numpy-1.26.4 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.20.5 nvidia-nvjitlink-cu12-12.5.82 nvidia-nvtx-cu12-12.1.105 openai-1.37.0 optimum-1.17.1 orjson-3.10.6 pathvalidate-3.2.0 peft-0.8.2 portalocker-2.10.1 pyarrow-17.0.0 pybind11-2.13.1 pycountry-24.6.1 pydub-0.25.1 pytablewriter-1.2.0 python-dotenv-1.0.1 python-multipart-0.0.9 requests-2.32.3 rouge-1.0.1 rouge-score-0.1.2 ruff-0.5.4 sacrebleu-1.5.0 semantic-version-2.10.0 sentry-sdk-2.11.0 setproctitle-1.3.3 smmap-5.0.1 sqlitedict-2.1.0 sse-starlette-1.6.5 starlette-0.37.2 tabledata-1.3.3 tcolorpy-0.1.6 tiktoken-0.7.0 tomlkit-0.12.0 tqdm-multiprocess-0.0.11 typepy-1.3.2 uvicorn-0.30.3 uvloop-0.19.0 wandb-0.17.5 watchfiles-0.22.0 websockets-11.0.3 xxhash-3.4.1 zstandard-0.23.0\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "PIL"
                ]
              },
              "id": "890a673b87854a64a55952a7703322fd"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: gradio==4.26.0 in /usr/local/lib/python3.10/dist-packages (4.26.0)\n",
            "Requirement already satisfied: aiofiles<24.0,>=22.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (23.2.1)\n",
            "Requirement already satisfied: altair<6.0,>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (4.2.2)\n",
            "Requirement already satisfied: fastapi in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (0.111.1)\n",
            "Requirement already satisfied: ffmpy in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (0.3.2)\n",
            "Requirement already satisfied: gradio-client==0.15.1 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (0.15.1)\n",
            "Requirement already satisfied: httpx>=0.24.1 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (0.27.0)\n",
            "Requirement already satisfied: huggingface-hub>=0.19.3 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (0.23.5)\n",
            "Requirement already satisfied: importlib-resources<7.0,>=1.3 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (6.4.0)\n",
            "Requirement already satisfied: jinja2<4.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (3.1.4)\n",
            "Requirement already satisfied: markupsafe~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (2.1.5)\n",
            "Requirement already satisfied: matplotlib~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (3.7.1)\n",
            "Requirement already satisfied: numpy~=1.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (1.26.4)\n",
            "Requirement already satisfied: orjson~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (3.10.6)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (24.1)\n",
            "Requirement already satisfied: pandas<3.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (2.0.3)\n",
            "Requirement already satisfied: pillow<11.0,>=8.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (10.4.0)\n",
            "Requirement already satisfied: pydantic>=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (2.8.2)\n",
            "Requirement already satisfied: pydub in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (0.25.1)\n",
            "Requirement already satisfied: python-multipart>=0.0.9 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (0.0.9)\n",
            "Requirement already satisfied: pyyaml<7.0,>=5.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (6.0.1)\n",
            "Requirement already satisfied: ruff>=0.2.2 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (0.5.4)\n",
            "Requirement already satisfied: semantic-version~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (2.10.0)\n",
            "Requirement already satisfied: tomlkit==0.12.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (0.12.0)\n",
            "Requirement already satisfied: typer<1.0,>=0.9 in /usr/local/lib/python3.10/dist-packages (from typer[all]<1.0,>=0.9; sys_platform != \"emscripten\"->gradio==4.26.0) (0.12.3)\n",
            "Requirement already satisfied: typing-extensions~=4.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (4.12.2)\n",
            "Requirement already satisfied: uvicorn>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from gradio==4.26.0) (0.30.3)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio-client==0.15.1->gradio==4.26.0) (2023.6.0)\n",
            "Requirement already satisfied: websockets<12.0,>=10.0 in /usr/local/lib/python3.10/dist-packages (from gradio-client==0.15.1->gradio==4.26.0) (11.0.3)\n",
            "Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==4.26.0) (0.4)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==4.26.0) (4.19.2)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==4.26.0) (0.12.1)\n",
            "Requirement already satisfied: anyio in /usr/local/lib/python3.10/dist-packages (from httpx>=0.24.1->gradio==4.26.0) (3.7.1)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx>=0.24.1->gradio==4.26.0) (2024.7.4)\n",
            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx>=0.24.1->gradio==4.26.0) (1.0.5)\n",
            "Requirement already satisfied: idna in /usr/local/lib/python3.10/dist-packages (from httpx>=0.24.1->gradio==4.26.0) (3.7)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx>=0.24.1->gradio==4.26.0) (1.3.1)\n",
            "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx>=0.24.1->gradio==4.26.0) (0.14.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.19.3->gradio==4.26.0) (3.15.4)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.19.3->gradio==4.26.0) (2.32.3)\n",
            "Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.19.3->gradio==4.26.0) (4.66.4)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.0) (1.2.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.0) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.0) (4.53.1)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.0) (1.4.5)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.0) (3.1.2)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==4.26.0) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas<3.0,>=1.0->gradio==4.26.0) (2023.4)\n",
            "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas<3.0,>=1.0->gradio==4.26.0) (2024.1)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2.0->gradio==4.26.0) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.20.1 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2.0->gradio==4.26.0) (2.20.1)\n",
            "Requirement already satisfied: click>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != \"emscripten\"->gradio==4.26.0) (8.1.7)\n",
            "Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != \"emscripten\"->gradio==4.26.0) (1.5.4)\n",
            "Requirement already satisfied: rich>=10.11.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != \"emscripten\"->gradio==4.26.0) (13.7.1)\n",
            "\u001b[33mWARNING: typer 0.12.3 does not provide the extra 'all'\u001b[0m\u001b[33m\n",
            "\u001b[0mRequirement already satisfied: starlette<0.38.0,>=0.37.2 in /usr/local/lib/python3.10/dist-packages (from fastapi->gradio==4.26.0) (0.37.2)\n",
            "Requirement already satisfied: fastapi-cli>=0.0.2 in /usr/local/lib/python3.10/dist-packages (from fastapi->gradio==4.26.0) (0.0.4)\n",
            "Requirement already satisfied: email_validator>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from fastapi->gradio==4.26.0) (2.2.0)\n",
            "Requirement already satisfied: dnspython>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from email_validator>=2.0.0->fastapi->gradio==4.26.0) (2.6.1)\n",
            "Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==4.26.0) (23.2.0)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==4.26.0) (2023.12.1)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==4.26.0) (0.35.1)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==4.26.0) (0.19.0)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib~=3.0->gradio==4.26.0) (1.16.0)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.11.0->typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != \"emscripten\"->gradio==4.26.0) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.11.0->typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != \"emscripten\"->gradio==4.26.0) (2.16.1)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio->httpx>=0.24.1->gradio==4.26.0) (1.2.2)\n",
            "Requirement already satisfied: httptools>=0.5.0 in /usr/local/lib/python3.10/dist-packages (from uvicorn[standard]>=0.12.0->fastapi->gradio==4.26.0) (0.6.1)\n",
            "Requirement already satisfied: python-dotenv>=0.13 in /usr/local/lib/python3.10/dist-packages (from uvicorn[standard]>=0.12.0->fastapi->gradio==4.26.0) (1.0.1)\n",
            "Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from uvicorn[standard]>=0.12.0->fastapi->gradio==4.26.0) (0.19.0)\n",
            "Requirement already satisfied: watchfiles>=0.13 in /usr/local/lib/python3.10/dist-packages (from uvicorn[standard]>=0.12.0->fastapi->gradio==4.26.0) (0.22.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.19.3->gradio==4.26.0) (3.3.2)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.19.3->gradio==4.26.0) (2.0.7)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != \"emscripten\"->gradio==4.26.0) (0.1.2)\n",
            " *** Download Progress Summary as of Thu Jul 25 02:27:03 2024 *** \n",
            "=\n",
            "[#79bafc 7.0GiB/8.1GiB(86%) CN:16 DL:56MiB ETA:20s]\n",
            "FILE: /content/text-generation-webui/models//Mistral-Nemo-Instruct-2407-Q5_K_M.gguf\n",
            "-\n",
            "\n",
            "\u001b[0m\n",
            "Download Results:\n",
            "gid   |stat|avg speed  |path/URI\n",
            "======+====+===========+=======================================================\n",
            "79bafc|\u001b[1;32mOK\u001b[0m  |   108MiB/s|/content/text-generation-webui/models//Mistral-Nemo-Instruct-2407-Q5_K_M.gguf\n",
            "\n",
            "Status Legend:\n",
            "(OK):download completed.\n",
            "/content/text-generation-webui\n",
            "\u001b[2;36m02:27:28-796787\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Starting Text generation web UI                                            \n",
            "\u001b[2;36m02:27:29-231555\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Loading \u001b[32m\"Mistral-Nemo-Instruct-2407-Q5_K_M.gguf\"\u001b[0m                           \n",
            "\u001b[2;36m02:27:29-610241\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m llama.cpp weights detected: \u001b[32m\"models/Mistral-Nemo-Instruct-2407-Q5_K_M.gguf\"\u001b[0m\n",
            "llama_model_loader: loaded meta data with 35 key-value pairs and 363 tensors from models/Mistral-Nemo-Instruct-2407-Q5_K_M.gguf (version GGUF V3 (latest))\n",
            "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
            "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
            "llama_model_loader: - kv   1:                               general.type str              = model\n",
            "llama_model_loader: - kv   2:                               general.name str              = Mistral Nemo Instruct 2407\n",
            "llama_model_loader: - kv   3:                            general.version str              = 2407\n",
            "llama_model_loader: - kv   4:                           general.finetune str              = Instruct\n",
            "llama_model_loader: - kv   5:                           general.basename str              = Mistral-Nemo\n",
            "llama_model_loader: - kv   6:                         general.size_label str              = 12B\n",
            "llama_model_loader: - kv   7:                            general.license str              = apache-2.0\n",
            "llama_model_loader: - kv   8:                          general.languages arr[str,9]       = [\"en\", \"fr\", \"de\", \"es\", \"it\", \"pt\", ...\n",
            "llama_model_loader: - kv   9:                          llama.block_count u32              = 40\n",
            "llama_model_loader: - kv  10:                       llama.context_length u32              = 1024000\n",
            "llama_model_loader: - kv  11:                     llama.embedding_length u32              = 5120\n",
            "llama_model_loader: - kv  12:                  llama.feed_forward_length u32              = 14336\n",
            "llama_model_loader: - kv  13:                 llama.attention.head_count u32              = 32\n",
            "llama_model_loader: - kv  14:              llama.attention.head_count_kv u32              = 8\n",
            "llama_model_loader: - kv  15:                       llama.rope.freq_base f32              = 1000000.000000\n",
            "llama_model_loader: - kv  16:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
            "llama_model_loader: - kv  17:                 llama.attention.key_length u32              = 128\n",
            "llama_model_loader: - kv  18:               llama.attention.value_length u32              = 128\n",
            "llama_model_loader: - kv  19:                          general.file_type u32              = 17\n",
            "llama_model_loader: - kv  20:                           llama.vocab_size u32              = 131072\n",
            "llama_model_loader: - kv  21:                 llama.rope.dimension_count u32              = 128\n",
            "llama_model_loader: - kv  22:            tokenizer.ggml.add_space_prefix bool             = false\n",
            "llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2\n",
            "llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = tekken\n",
            "llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,131072]  = [\"<unk>\", \"<s>\", \"</s>\", \"[INST]\", \"[...\n",
            "llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,131072]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...\n",
            "Exception ignored on calling ctypes callback function: <function llama_log_callback at 0x7f8de3ef8040>\n",
            "Traceback (most recent call last):\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/llama_cpp_cuda/_logger.py\", line 30, in llama_log_callback\n",
            "    print(text.decode(\"utf-8\"), end=\"\", flush=True, file=sys.stderr)\n",
            "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc4 in position 128: invalid continuation byte\n",
            "llama_model_loader: - kv  28:                tokenizer.ggml.bos_token_id u32              = 1\n",
            "llama_model_loader: - kv  29:                tokenizer.ggml.eos_token_id u32              = 2\n",
            "llama_model_loader: - kv  30:            tokenizer.ggml.unknown_token_id u32              = 0\n",
            "llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = true\n",
            "llama_model_loader: - kv  32:               tokenizer.ggml.add_eos_token bool             = false\n",
            "llama_model_loader: - kv  33:                    tokenizer.chat_template str              = {%- if messages[0]['role'] == 'system...\n",
            "llama_model_loader: - kv  34:               general.quantization_version u32              = 2\n",
            "llama_model_loader: - type  f32:   81 tensors\n",
            "llama_model_loader: - type q5_K:  241 tensors\n",
            "llama_model_loader: - type q6_K:   41 tensors\n",
            "llm_load_vocab: special tokens cache size = 1000\n",
            "llm_load_vocab: token to piece cache size = 0.8498 MB\n",
            "llm_load_print_meta: format           = GGUF V3 (latest)\n",
            "llm_load_print_meta: arch             = llama\n",
            "llm_load_print_meta: vocab type       = BPE\n",
            "llm_load_print_meta: n_vocab          = 131072\n",
            "llm_load_print_meta: n_merges         = 269443\n",
            "llm_load_print_meta: vocab_only       = 0\n",
            "llm_load_print_meta: n_ctx_train      = 1024000\n",
            "llm_load_print_meta: n_embd           = 5120\n",
            "llm_load_print_meta: n_layer          = 40\n",
            "llm_load_print_meta: n_head           = 32\n",
            "llm_load_print_meta: n_head_kv        = 8\n",
            "llm_load_print_meta: n_rot            = 128\n",
            "llm_load_print_meta: n_swa            = 0\n",
            "llm_load_print_meta: n_embd_head_k    = 128\n",
            "llm_load_print_meta: n_embd_head_v    = 128\n",
            "llm_load_print_meta: n_gqa            = 4\n",
            "llm_load_print_meta: n_embd_k_gqa     = 1024\n",
            "llm_load_print_meta: n_embd_v_gqa     = 1024\n",
            "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
            "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
            "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
            "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
            "llm_load_print_meta: f_logit_scale    = 0.0e+00\n",
            "llm_load_print_meta: n_ff             = 14336\n",
            "llm_load_print_meta: n_expert         = 0\n",
            "llm_load_print_meta: n_expert_used    = 0\n",
            "llm_load_print_meta: causal attn      = 1\n",
            "llm_load_print_meta: pooling type     = 0\n",
            "llm_load_print_meta: rope type        = 0\n",
            "llm_load_print_meta: rope scaling     = linear\n",
            "llm_load_print_meta: freq_base_train  = 1000000.0\n",
            "llm_load_print_meta: freq_scale_train = 1\n",
            "llm_load_print_meta: n_ctx_orig_yarn  = 1024000\n",
            "llm_load_print_meta: rope_finetuned   = unknown\n",
            "llm_load_print_meta: ssm_d_conv       = 0\n",
            "llm_load_print_meta: ssm_d_inner      = 0\n",
            "llm_load_print_meta: ssm_d_state      = 0\n",
            "llm_load_print_meta: ssm_dt_rank      = 0\n",
            "llm_load_print_meta: model type       = 13B\n",
            "llm_load_print_meta: model ftype      = Q5_K - Medium\n",
            "llm_load_print_meta: model params     = 12.25 B\n",
            "llm_load_print_meta: model size       = 8.12 GiB (5.70 BPW) \n",
            "llm_load_print_meta: general.name     = Mistral Nemo Instruct 2407\n",
            "llm_load_print_meta: BOS token        = 1 '<s>'\n",
            "llm_load_print_meta: EOS token        = 2 '</s>'\n",
            "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
            "llm_load_print_meta: LF token         = 1196 'Ä'\n",
            "llm_load_print_meta: max token length = 150\n",
            "ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    yes\n",
            "ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no\n",
            "ggml_cuda_init: found 1 CUDA devices:\n",
            "  Device 0: Tesla T4, compute capability 7.5, VMM: yes\n",
            "llm_load_tensors: ggml ctx size =    0.34 MiB\n",
            "llm_load_tensors: offloading 40 repeating layers to GPU\n",
            "llm_load_tensors: offloading non-repeating layers to GPU\n",
            "llm_load_tensors: offloaded 41/41 layers to GPU\n",
            "llm_load_tensors:        CPU buffer size =   440.00 MiB\n",
            "llm_load_tensors:      CUDA0 buffer size =  7875.80 MiB\n",
            "...........................................................................................\n",
            "llama_new_context_with_model: n_ctx      = 8224\n",
            "llama_new_context_with_model: n_batch    = 512\n",
            "llama_new_context_with_model: n_ubatch   = 512\n",
            "llama_new_context_with_model: flash_attn = 0\n",
            "llama_new_context_with_model: freq_base  = 1000000.0\n",
            "llama_new_context_with_model: freq_scale = 1\n",
            "llama_kv_cache_init:      CUDA0 KV buffer size =  1285.00 MiB\n",
            "llama_new_context_with_model: KV self size  = 1285.00 MiB, K (f16):  642.50 MiB, V (f16):  642.50 MiB\n",
            "llama_new_context_with_model:  CUDA_Host  output buffer size =     0.50 MiB\n",
            "llama_new_context_with_model:      CUDA0 compute buffer size =   566.07 MiB\n",
            "llama_new_context_with_model:  CUDA_Host compute buffer size =    26.07 MiB\n",
            "llama_new_context_with_model: graph nodes  = 1286\n",
            "llama_new_context_with_model: graph splits = 2\n",
            "AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 0 | \n",
            "Model metadata: {'general.quantization_version': '2', 'tokenizer.chat_template': \"{%- if messages[0]['role'] == 'system' %}\\n    {%- set system_message = messages[0]['content'] %}\\n    {%- set loop_messages = messages[1:] %}\\n{%- else %}\\n    {%- set loop_messages = messages %}\\n{%- endif %}\\n\\n{{- bos_token }}\\n{%- for message in loop_messages %}\\n    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}\\n        {{- raise_exception('After the optional system message, conversation roles must alternate user/assistant/user/assistant/...') }}\\n    {%- endif %}\\n    {%- if message['role'] == 'user' %}\\n        {%- if loop.last and system_message is defined %}\\n            {{- '[INST] ' + system_message + '\\\\n\\\\n' + message['content'] + '[/INST]' }}\\n        {%- else %}\\n            {{- '[INST] ' + message['content'] + '[/INST]' }}\\n        {%- endif %}\\n    {%- elif message['role'] == 'assistant' %}\\n        {{- ' ' + message['content'] + eos_token}}\\n    {%- else %}\\n        {{- raise_exception('Only user and assistant roles are supported, with the exception of an initial optional system message!') }}\\n    {%- endif %}\\n{%- endfor %}\\n\", 'llama.embedding_length': '5120', 'llama.feed_forward_length': '14336', 'general.license': 'apache-2.0', 'llama.attention.value_length': '128', 'tokenizer.ggml.add_bos_token': 'true', 'general.size_label': '12B', 'general.type': 'model', 'general.version': '2407', 'llama.context_length': '1024000', 'general.name': 'Mistral Nemo Instruct 2407', 'tokenizer.ggml.bos_token_id': '1', 'general.basename': 'Mistral-Nemo', 'llama.attention.head_count_kv': '8', 'general.architecture': 'llama', 'llama.rope.freq_base': '1000000.000000', 'llama.attention.head_count': '32', 'llama.block_count': '40', 'llama.attention.key_length': '128', 'general.finetune': 'Instruct', 'general.file_type': '17', 'tokenizer.ggml.pre': 'tekken', 'llama.vocab_size': '131072', 'llama.rope.dimension_count': '128', 'tokenizer.ggml.add_space_prefix': 'false', 'tokenizer.ggml.add_eos_token': 'false', 'tokenizer.ggml.model': 'gpt2', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'tokenizer.ggml.eos_token_id': '2', 'tokenizer.ggml.unknown_token_id': '0'}\n",
            "Available chat formats from metadata: chat_template.default\n",
            "Using gguf chat template: {%- if messages[0]['role'] == 'system' %}\n",
            "    {%- set system_message = messages[0]['content'] %}\n",
            "    {%- set loop_messages = messages[1:] %}\n",
            "{%- else %}\n",
            "    {%- set loop_messages = messages %}\n",
            "{%- endif %}\n",
            "\n",
            "{{- bos_token }}\n",
            "{%- for message in loop_messages %}\n",
            "    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}\n",
            "        {{- raise_exception('After the optional system message, conversation roles must alternate user/assistant/user/assistant/...') }}\n",
            "    {%- endif %}\n",
            "    {%- if message['role'] == 'user' %}\n",
            "        {%- if loop.last and system_message is defined %}\n",
            "            {{- '[INST] ' + system_message + '\\n\\n' + message['content'] + '[/INST]' }}\n",
            "        {%- else %}\n",
            "            {{- '[INST] ' + message['content'] + '[/INST]' }}\n",
            "        {%- endif %}\n",
            "    {%- elif message['role'] == 'assistant' %}\n",
            "        {{- ' ' + message['content'] + eos_token}}\n",
            "    {%- else %}\n",
            "        {{- raise_exception('Only user and assistant roles are supported, with the exception of an initial optional system message!') }}\n",
            "    {%- endif %}\n",
            "{%- endfor %}\n",
            "\n",
            "Using chat eos_token: </s>\n",
            "Using chat bos_token: <s>\n",
            "\u001b[2;36m02:28:03-353998\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Loaded \u001b[32m\"Mistral-Nemo-Instruct-2407-Q5_K_M.gguf\"\u001b[0m in \u001b[1;36m34.12\u001b[0m seconds.          \n",
            "\u001b[2;36m02:28:03-355829\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m LOADER: \u001b[32m\"llama.cpp\"\u001b[0m                                                        \n",
            "\u001b[2;36m02:28:03-357139\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m TRUNCATION LENGTH: \u001b[1;36m8196\u001b[0m                                                    \n",
            "\u001b[2;36m02:28:03-358481\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m INSTRUCTION TEMPLATE: \u001b[32m\"Custom \u001b[0m\u001b[32m(\u001b[0m\u001b[32mobtained from model metadata\u001b[0m\u001b[32m)\u001b[0m\u001b[32m\"\u001b[0m              \n",
            "\n",
            "Running on local URL:  http://127.0.0.1:7860\n",
            "\n",
            "Running on public URL: https://859500d4cbb98b1a9e.gradio.live\n",
            "\n",
            "This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)\n",
            "\n",
            "llama_print_timings:        load time =     532.25 ms\n",
            "llama_print_timings:      sample time =    1006.57 ms /   104 runs   (    9.68 ms per token,   103.32 tokens per second)\n",
            "llama_print_timings: prompt eval time =     530.64 ms /    19 tokens (   27.93 ms per token,    35.81 tokens per second)\n",
            "llama_print_timings:        eval time =    4123.63 ms /   103 runs   (   40.04 ms per token,    24.98 tokens per second)\n",
            "llama_print_timings:       total time =    7323.23 ms /   122 tokens\n",
            "Output generated in 7.91 seconds (13.03 tokens/s, 103 tokens, context 26, seed 1494760513)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =     532.25 ms\n",
            "llama_print_timings:      sample time =    2327.26 ms /   290 runs   (    8.03 ms per token,   124.61 tokens per second)\n",
            "llama_print_timings: prompt eval time =     267.07 ms /   126 tokens (    2.12 ms per token,   471.78 tokens per second)\n",
            "llama_print_timings:        eval time =   11835.83 ms /   289 runs   (   40.95 ms per token,    24.42 tokens per second)\n",
            "llama_print_timings:       total time =   18233.58 ms /   415 tokens\n",
            "Output generated in 18.90 seconds (15.29 tokens/s, 289 tokens, context 158, seed 1590806321)\n",
            "\u001b[2;36m02:30:24-402476\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Deleted \u001b[32m\"logs/instruct/20240725-02-28-34.json\"\u001b[0m.                            \n",
            "Llama.generate: prefix-match hit\n"
          ]
        }
      ],
      "source": [
        "%cd /content\n",
        "!apt-get -y install -qq aria2\n",
        "\n",
        "!git clone -b V20240724 https://github.com/Troyanovsky/text-generation-webui\n",
        "%cd /content/text-generation-webui\n",
        "!pip install -r requirements.txt\n",
        "!pip install -U gradio==4.26.0\n",
        "\n",
        "!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/second-state/Mistral-Nemo-Instruct-2407-GGUF/resolve/main/Mistral-Nemo-Instruct-2407-Q5_K_M.gguf?download=true -d /content/text-generation-webui/models/ -o Mistral-Nemo-Instruct-2407-Q5_K_M.gguf\n",
        "\n",
        "%cd /content/text-generation-webui\n",
        "!python server.py --share --n-gpu-layers 100000 --model Mistral-Nemo-Instruct-2407-Q5_K_M.gguf --n_ctx 8196"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyP9JApd7YR1nHII5Wg0aoFp",
      "include_colab_link": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}