Mixtral_8x7B_Instruct_v0_1_GGUF.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Troyanovsky/Local-LLM-Comparison-Colab-UI/blob/main/Mixtral_8x7B_Instruct_v0_1_GGUF.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XNkpBtLvuTOp"
      },
      "source": [
        "## Mixtral-8x7B-Instruct-v0.1-GGUF WebUI\n",
        "1. Run the following cell, takes ~5 min\n",
        "(You may need to confirm to proceed by typing \"Y\")\n",
        "2. Click the gradio link at the bottom\n",
        "3. In Chat settings - Instruction Template: Mistral\n",
        "\n",
        "```\n",
        "<s>[INST] {prompt} [/INST]\n",
        "```\n",
        "\n",
        "Note:\n",
        "- this notebook is created at a time when Oobabooga doesn't support Mixtral 8x7B MOE natively\n",
        "- only run the cell with the quantization you need\n",
        "- only part of the layers are loaded to GPU so the generation speed will be slow (if you have colab pro, you can adjust the gpu layers to a higher number for faster speed)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!nvcc -V"
      ],
      "metadata": {
        "id": "Xltme_rDZwMC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "953cf207-40f0-40aa-8476-f88fb71b529c"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "nvcc: NVIDIA (R) Cuda compiler driver\n",
            "Copyright (c) 2005-2022 NVIDIA Corporation\n",
            "Built on Wed_Sep_21_10:33:58_PDT_2022\n",
            "Cuda compilation tools, release 11.8, V11.8.89\n",
            "Build cuda_11.8.r11.8/compiler.31833905_0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "D-MiHp_bveP6",
        "outputId": "359bb05b-c329-4903-98a8-55c5e96b1fe3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content\n",
            "Selecting previously unselected package libc-ares2:amd64.\n",
            "(Reading database ... 120903 files and directories currently installed.)\n",
            "Preparing to unpack .../libc-ares2_1.18.1-1ubuntu0.22.04.2_amd64.deb ...\n",
            "Unpacking libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.2) ...\n",
            "Selecting previously unselected package libaria2-0:amd64.\n",
            "Preparing to unpack .../libaria2-0_1.36.0-1_amd64.deb ...\n",
            "Unpacking libaria2-0:amd64 (1.36.0-1) ...\n",
            "Selecting previously unselected package aria2.\n",
            "Preparing to unpack .../aria2_1.36.0-1_amd64.deb ...\n",
            "Unpacking aria2 (1.36.0-1) ...\n",
            "Setting up libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.2) ...\n",
            "Setting up libaria2-0:amd64 (1.36.0-1) ...\n",
            "Setting up aria2 (1.36.0-1) ...\n",
            "Processing triggers for man-db (2.10.2-1) ...\n",
            "Processing triggers for libc-bin (2.35-0ubuntu3.4) ...\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link\n",
            "\n",
            "Cloning into 'text-generation-webui'...\n",
            "remote: Enumerating objects: 14570, done.\u001b[K\n",
            "remote: Total 14570 (delta 0), reused 0 (delta 0), pack-reused 14570\u001b[K\n",
            "Receiving objects: 100% (14570/14570), 25.77 MiB | 9.33 MiB/s, done.\n",
            "Resolving deltas: 100% (10066/10066), done.\n",
            "Note: switching to '705f04a0c9b09870fbe595dbba1d61db8289a07b'.\n",
            "\n",
            "You are in 'detached HEAD' state. You can look around, make experimental\n",
            "changes and commit them, and you can discard any commits you make in this\n",
            "state without impacting any branches by switching back to a branch.\n",
            "\n",
            "If you want to create a new branch to retain commits you create, you may\n",
            "do so (now or later) by using -c with the switch command. Example:\n",
            "\n",
            "  git switch -c <new-branch-name>\n",
            "\n",
            "Or undo this operation with:\n",
            "\n",
            "  git switch -\n",
            "\n",
            "Turn off this advice by setting config variable advice.detachedHead to false\n",
            "\n",
            "/content/text-generation-webui\n",
            "Ignoring exllamav2: markers 'platform_system != \"Darwin\" and platform_machine != \"x86_64\"' don't match your environment\n",
            "Collecting git+https://github.com/oobabooga/torch-grammar.git (from -r requirements.txt (line 23))\n",
            "  Cloning https://github.com/oobabooga/torch-grammar.git to /tmp/pip-req-build-2ib2helm\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/oobabooga/torch-grammar.git /tmp/pip-req-build-2ib2helm\n",
            "  Resolved https://github.com/oobabooga/torch-grammar.git to commit 82850b5383a629f3b0fa1fba7d8f2aba3185ddb2\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Ignoring bitsandbytes: markers 'platform_system == \"Windows\"' don't match your environment\n",
            "Ignoring llama-cpp-python: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting llama-cpp-python==0.2.19+cpuavx2 (from -r requirements.txt (line 31))\n",
            "  Downloading https://github.com/jllllll/llama-cpp-python-cuBLAS-wheels/releases/download/cpu/llama_cpp_python-0.2.19+cpuavx2-cp310-cp310-manylinux_2_31_x86_64.whl (2.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m25.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring llama-cpp-python: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.9\"' don't match your environment\n",
            "Ignoring llama-cpp-python: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.8\"' don't match your environment\n",
            "Ignoring llama-cpp-python: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring llama-cpp-python: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring llama-cpp-python: markers 'platform_system == \"Windows\" and python_version == \"3.9\"' don't match your environment\n",
            "Ignoring llama-cpp-python: markers 'platform_system == \"Windows\" and python_version == \"3.8\"' don't match your environment\n",
            "Ignoring auto-gptq: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring auto-gptq: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring auto-gptq: markers 'platform_system == \"Windows\" and python_version == \"3.9\"' don't match your environment\n",
            "Ignoring auto-gptq: markers 'platform_system == \"Windows\" and python_version == \"3.8\"' don't match your environment\n",
            "Ignoring auto-gptq: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting auto-gptq==0.5.1+cu121 (from -r requirements.txt (line 45))\n",
            "  Downloading https://github.com/jllllll/AutoGPTQ/releases/download/v0.5.1/auto_gptq-0.5.1+cu121-cp310-cp310-linux_x86_64.whl (4.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.8/4.8 MB\u001b[0m \u001b[31m70.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring auto-gptq: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.9\"' don't match your environment\n",
            "Ignoring auto-gptq: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.8\"' don't match your environment\n",
            "Ignoring exllama: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring exllama: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring exllama: markers 'platform_system == \"Windows\" and python_version == \"3.9\"' don't match your environment\n",
            "Ignoring exllama: markers 'platform_system == \"Windows\" and python_version == \"3.8\"' don't match your environment\n",
            "Ignoring exllama: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting exllama==0.0.18+cu121 (from -r requirements.txt (line 53))\n",
            "  Downloading https://github.com/jllllll/exllama/releases/download/0.0.18/exllama-0.0.18+cu121-cp310-cp310-linux_x86_64.whl (434 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m434.6/434.6 kB\u001b[0m \u001b[31m42.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring exllama: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.9\"' don't match your environment\n",
            "Ignoring exllama: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.8\"' don't match your environment\n",
            "Ignoring exllamav2: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring exllamav2: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring exllamav2: markers 'platform_system == \"Windows\" and python_version == \"3.9\"' don't match your environment\n",
            "Ignoring exllamav2: markers 'platform_system == \"Windows\" and python_version == \"3.8\"' don't match your environment\n",
            "Ignoring exllamav2: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting exllamav2==0.0.10+cu121 (from -r requirements.txt (line 61))\n",
            "  Downloading https://github.com/turboderp/exllamav2/releases/download/v0.0.10/exllamav2-0.0.10+cu121-cp310-cp310-linux_x86_64.whl (12.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.1/12.1 MB\u001b[0m \u001b[31m68.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring exllamav2: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.9\"' don't match your environment\n",
            "Ignoring exllamav2: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.8\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Windows\" and python_version == \"3.9\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Windows\" and python_version == \"3.8\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting flash-attn==2.3.4+cu122torch2.1cxx11abiFALSE (from -r requirements.txt (line 69))\n",
            "  Downloading https://github.com/Dao-AILab/flash-attention/releases/download/v2.3.4/flash_attn-2.3.4+cu122torch2.1cxx11abiFALSE-cp310-cp310-linux_x86_64.whl (57.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.4/57.4 MB\u001b[0m \u001b[31m9.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring flash-attn: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.9\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.8\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Windows\" and python_version == \"3.9\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Windows\" and python_version == \"3.8\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting llama-cpp-python-cuda==0.2.19+cu121 (from -r requirements.txt (line 77))\n",
            "  Downloading https://github.com/jllllll/llama-cpp-python-cuBLAS-wheels/releases/download/textgen-webui/llama_cpp_python_cuda-0.2.19+cu121-cp310-cp310-manylinux_2_31_x86_64.whl (25.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m25.3/25.3 MB\u001b[0m \u001b[31m66.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring llama-cpp-python-cuda: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.9\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.8\"' don't match your environment\n",
            "Ignoring gptq-for-llama: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring gptq-for-llama: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring gptq-for-llama: markers 'platform_system == \"Windows\" and python_version == \"3.9\"' don't match your environment\n",
            "Ignoring gptq-for-llama: markers 'platform_system == \"Windows\" and python_version == \"3.8\"' don't match your environment\n",
            "Ignoring gptq-for-llama: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting gptq-for-llama==0.1.1+cu121 (from -r requirements.txt (line 85))\n",
            "  Downloading https://github.com/jllllll/GPTQ-for-LLaMa-CUDA/releases/download/0.1.1/gptq_for_llama-0.1.1+cu121-cp310-cp310-linux_x86_64.whl (739 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m739.5/739.5 kB\u001b[0m \u001b[31m59.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring gptq-for-llama: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.9\"' don't match your environment\n",
            "Ignoring gptq-for-llama: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.8\"' don't match your environment\n",
            "Collecting ctransformers==0.2.27+cu121 (from -r requirements.txt (line 88))\n",
            "  Downloading https://github.com/jllllll/ctransformers-cuBLAS-wheels/releases/download/AVX2/ctransformers-0.2.27+cu121-py3-none-any.whl (15.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m15.5/15.5 MB\u001b[0m \u001b[31m86.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting accelerate==0.25.* (from -r requirements.txt (line 1))\n",
            "  Downloading accelerate-0.25.0-py3-none-any.whl (265 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m265.7/265.7 kB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting colorama (from -r requirements.txt (line 2))\n",
            "  Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n",
            "Collecting datasets (from -r requirements.txt (line 3))\n",
            "  Downloading datasets-2.15.0-py3-none-any.whl (521 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m521.2/521.2 kB\u001b[0m \u001b[31m48.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting einops (from -r requirements.txt (line 4))\n",
            "  Downloading einops-0.7.0-py3-none-any.whl (44 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m44.6/44.6 kB\u001b[0m \u001b[31m5.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting gradio==3.50.* (from -r requirements.txt (line 6))\n",
            "  Downloading gradio-3.50.2-py3-none-any.whl (20.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m20.3/20.3 MB\u001b[0m \u001b[31m16.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: markdown in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 7)) (3.5.1)\n",
            "Collecting numpy==1.24.* (from -r requirements.txt (line 8))\n",
            "  Downloading numpy-1.24.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m17.3/17.3 MB\u001b[0m \u001b[31m60.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting optimum==1.14.0 (from -r requirements.txt (line 9))\n",
            "  Downloading optimum-1.14.0-py3-none-any.whl (398 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m398.9/398.9 kB\u001b[0m \u001b[31m41.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 10)) (1.5.3)\n",
            "Collecting peft==0.6.* (from -r requirements.txt (line 11))\n",
            "  Downloading peft-0.6.2-py3-none-any.whl (174 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m174.7/174.7 kB\u001b[0m \u001b[31m21.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting Pillow>=9.5.0 (from -r requirements.txt (line 12))\n",
            "  Downloading Pillow-10.1.0-cp310-cp310-manylinux_2_28_x86_64.whl (3.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.6/3.6 MB\u001b[0m \u001b[31m76.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 13)) (6.0.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 14)) (2.31.0)\n",
            "Requirement already satisfied: safetensors==0.4.1 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 15)) (0.4.1)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 16)) (1.11.4)\n",
            "Collecting sentencepiece (from -r requirements.txt (line 17))\n",
            "  Downloading sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m58.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tensorboard in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 18)) (2.14.1)\n",
            "Requirement already satisfied: transformers==4.35.* in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 19)) (4.35.2)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 20)) (4.66.1)\n",
            "Collecting wandb (from -r requirements.txt (line 21))\n",
            "  Downloading wandb-0.16.1-py3-none-any.whl (2.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m73.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting bitsandbytes==0.41.1 (from -r requirements.txt (line 26))\n",
            "  Downloading bitsandbytes-0.41.1-py3-none-any.whl (92.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m92.6/92.6 MB\u001b[0m \u001b[31m8.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting autoawq==0.1.7 (from -r requirements.txt (line 89))\n",
            "  Downloading autoawq-0.1.7-cp310-cp310-manylinux2014_x86_64.whl (20.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m20.5/20.5 MB\u001b[0m \u001b[31m49.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.25.*->-r requirements.txt (line 1)) (23.2)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate==0.25.*->-r requirements.txt (line 1)) (5.9.5)\n",
            "Requirement already satisfied: torch>=1.10.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.25.*->-r requirements.txt (line 1)) (2.1.0+cu118)\n",
            "Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (from accelerate==0.25.*->-r requirements.txt (line 1)) (0.19.4)\n",
            "Collecting aiofiles<24.0,>=22.0 (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading aiofiles-23.2.1-py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: altair<6.0,>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.*->-r requirements.txt (line 6)) (4.2.2)\n",
            "Collecting fastapi (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading fastapi-0.105.0-py3-none-any.whl (93 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m93.1/93.1 kB\u001b[0m \u001b[31m12.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting ffmpy (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading ffmpy-0.3.1.tar.gz (5.5 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting gradio-client==0.6.1 (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading gradio_client-0.6.1-py3-none-any.whl (299 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m299.2/299.2 kB\u001b[0m \u001b[31m33.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting httpx (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading httpx-0.25.2-py3-none-any.whl (74 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.0/75.0 kB\u001b[0m \u001b[31m9.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: importlib-resources<7.0,>=1.3 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.*->-r requirements.txt (line 6)) (6.1.1)\n",
            "Requirement already satisfied: jinja2<4.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.*->-r requirements.txt (line 6)) (3.1.2)\n",
            "Requirement already satisfied: markupsafe~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.*->-r requirements.txt (line 6)) (2.1.3)\n",
            "Requirement already satisfied: matplotlib~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.*->-r requirements.txt (line 6)) (3.7.1)\n",
            "Collecting orjson~=3.0 (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading orjson-3.9.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (138 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m138.7/138.7 kB\u001b[0m \u001b[31m18.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.*->-r requirements.txt (line 6)) (1.10.13)\n",
            "Collecting pydub (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)\n",
            "Collecting python-multipart (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading python_multipart-0.0.6-py3-none-any.whl (45 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.7/45.7 kB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting semantic-version~=2.0 (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: typing-extensions~=4.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.*->-r requirements.txt (line 6)) (4.5.0)\n",
            "Collecting uvicorn>=0.14.0 (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading uvicorn-0.24.0.post1-py3-none-any.whl (59 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m59.7/59.7 kB\u001b[0m \u001b[31m8.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting websockets<12.0,>=10.0 (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading websockets-11.0.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (129 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m129.9/129.9 kB\u001b[0m \u001b[31m16.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting coloredlogs (from optimum==1.14.0->-r requirements.txt (line 9))\n",
            "  Downloading coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from optimum==1.14.0->-r requirements.txt (line 9)) (1.12)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.35.*->-r requirements.txt (line 19)) (3.13.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.35.*->-r requirements.txt (line 19)) (2023.6.3)\n",
            "Requirement already satisfied: tokenizers<0.19,>=0.14 in /usr/local/lib/python3.10/dist-packages (from transformers==4.35.*->-r requirements.txt (line 19)) (0.15.0)\n",
            "Collecting lm-eval (from autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading lm_eval-0.3.0-py3-none-any.whl (178 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m178.7/178.7 kB\u001b[0m \u001b[31m20.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting texttable (from autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading texttable-1.7.0-py2.py3-none-any.whl (10 kB)\n",
            "Requirement already satisfied: toml in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.7->-r requirements.txt (line 89)) (0.10.2)\n",
            "Collecting attributedict (from autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading attributedict-0.3.0-py3-none-any.whl (14 kB)\n",
            "Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.7->-r requirements.txt (line 89)) (3.20.3)\n",
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.7->-r requirements.txt (line 89)) (0.16.0+cu118)\n",
            "Requirement already satisfied: tabulate in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.7->-r requirements.txt (line 89)) (0.9.0)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio-client==0.6.1->gradio==3.50.*->-r requirements.txt (line 6)) (2023.6.0)\n",
            "Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 3)) (10.0.1)\n",
            "Collecting pyarrow-hotfix (from datasets->-r requirements.txt (line 3))\n",
            "  Downloading pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB)\n",
            "Collecting dill<0.3.8,>=0.3.0 (from datasets->-r requirements.txt (line 3))\n",
            "  Downloading dill-0.3.7-py3-none-any.whl (115 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m115.3/115.3 kB\u001b[0m \u001b[31m15.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 3)) (3.4.1)\n",
            "Collecting multiprocess (from datasets->-r requirements.txt (line 3))\n",
            "  Downloading multiprocess-0.70.15-py310-none-any.whl (134 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m18.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 3)) (3.9.1)\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 10)) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 10)) (2023.3.post1)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 14)) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 14)) (3.6)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 14)) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 14)) (2023.11.17)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 18)) (1.4.0)\n",
            "Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 18)) (1.59.3)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 18)) (2.17.3)\n",
            "Requirement already satisfied: google-auth-oauthlib<1.1,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 18)) (1.0.0)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 18)) (67.7.2)\n",
            "Requirement already satisfied: six>1.9 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 18)) (1.16.0)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 18)) (0.7.2)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 18)) (3.0.1)\n",
            "Requirement already satisfied: Click!=8.0.0,>=7.1 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 21)) (8.1.7)\n",
            "Collecting GitPython!=3.1.29,>=1.0.0 (from wandb->-r requirements.txt (line 21))\n",
            "  Downloading GitPython-3.1.40-py3-none-any.whl (190 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m190.6/190.6 kB\u001b[0m \u001b[31m26.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting sentry-sdk>=1.0.0 (from wandb->-r requirements.txt (line 21))\n",
            "  Downloading sentry_sdk-1.39.0-py2.py3-none-any.whl (254 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m254.0/254.0 kB\u001b[0m \u001b[31m32.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting docker-pycreds>=0.4.0 (from wandb->-r requirements.txt (line 21))\n",
            "  Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)\n",
            "Collecting setproctitle (from wandb->-r requirements.txt (line 21))\n",
            "  Downloading setproctitle-1.3.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30 kB)\n",
            "Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 21)) (1.4.4)\n",
            "Requirement already satisfied: diskcache>=5.6.1 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python==0.2.19+cpuavx2->-r requirements.txt (line 31)) (5.6.3)\n",
            "Collecting rouge (from auto-gptq==0.5.1+cu121->-r requirements.txt (line 45))\n",
            "  Downloading rouge-1.0.1-py3-none-any.whl (13 kB)\n",
            "Collecting gekko (from auto-gptq==0.5.1+cu121->-r requirements.txt (line 45))\n",
            "  Downloading gekko-1.0.6-py3-none-any.whl (12.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.2/12.2 MB\u001b[0m \u001b[31m31.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting ninja (from exllamav2==0.0.10+cu121->-r requirements.txt (line 61))\n",
            "  Downloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m307.2/307.2 kB\u001b[0m \u001b[31m32.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting fastparquet (from exllamav2==0.0.10+cu121->-r requirements.txt (line 61))\n",
            "  Downloading fastparquet-2023.10.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m87.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pygments in /usr/local/lib/python3.10/dist-packages (from exllamav2==0.0.10+cu121->-r requirements.txt (line 61)) (2.16.1)\n",
            "Requirement already satisfied: py-cpuinfo<10.0.0,>=9.0.0 in /usr/local/lib/python3.10/dist-packages (from ctransformers==0.2.27+cu121->-r requirements.txt (line 88)) (9.0.0)\n",
            "Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==3.50.*->-r requirements.txt (line 6)) (0.4)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==3.50.*->-r requirements.txt (line 6)) (4.19.2)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==3.50.*->-r requirements.txt (line 6)) (0.12.0)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (23.1.0)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (6.0.4)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (1.9.3)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (1.4.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (1.3.1)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (4.0.3)\n",
            "Collecting gitdb<5,>=4.0.1 (from GitPython!=3.1.29,>=1.0.0->wandb->-r requirements.txt (line 21))\n",
            "  Downloading gitdb-4.0.11-py3-none-any.whl (62 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 kB\u001b[0m \u001b[31m8.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 18)) (5.3.2)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 18)) (0.3.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 18)) (4.9)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard->-r requirements.txt (line 18)) (1.3.1)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.*->-r requirements.txt (line 6)) (1.2.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.*->-r requirements.txt (line 6)) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.*->-r requirements.txt (line 6)) (4.46.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.*->-r requirements.txt (line 6)) (1.4.5)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.*->-r requirements.txt (line 6)) (3.1.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.*->-r requirements.txt (line 1)) (3.2.1)\n",
            "Requirement already satisfied: triton==2.1.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.*->-r requirements.txt (line 1)) (2.1.0)\n",
            "Collecting h11>=0.8 (from uvicorn>=0.14.0->gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m7.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting rootpath>=0.1.0 (from attributedict->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading rootpath-0.1.1-py3-none-any.whl (15 kB)\n",
            "Collecting inspecta>=0.1.0 (from attributedict->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading inspecta-0.1.3-py3-none-any.whl (9.2 kB)\n",
            "Collecting colour-runner>=0.0.5 (from attributedict->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading colour_runner-0.1.1-py2.py3-none-any.whl (3.7 kB)\n",
            "Collecting deepdiff>=3.3.0 (from attributedict->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading deepdiff-6.7.1-py3-none-any.whl (76 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m76.6/76.6 kB\u001b[0m \u001b[31m9.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tox>=3.0.0 (from attributedict->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading tox-4.11.4-py3-none-any.whl (153 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m154.0/154.0 kB\u001b[0m \u001b[31m17.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting coverage>=4.5.2 (from attributedict->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading coverage-7.3.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (227 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m227.5/227.5 kB\u001b[0m \u001b[31m25.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting codecov>=2.0.15 (from attributedict->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading codecov-2.1.13-py2.py3-none-any.whl (16 kB)\n",
            "Collecting humanfriendly>=9.1 (from coloredlogs->optimum==1.14.0->-r requirements.txt (line 9))\n",
            "  Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.8/86.8 kB\u001b[0m \u001b[31m11.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: anyio<4.0.0,>=3.7.1 in /usr/local/lib/python3.10/dist-packages (from fastapi->gradio==3.50.*->-r requirements.txt (line 6)) (3.7.1)\n",
            "Collecting starlette<0.28.0,>=0.27.0 (from fastapi->gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading starlette-0.27.0-py3-none-any.whl (66 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m67.0/67.0 kB\u001b[0m \u001b[31m8.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting typing-extensions~=4.0 (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading typing_extensions-4.9.0-py3-none-any.whl (32 kB)\n",
            "Collecting cramjam>=2.3 (from fastparquet->exllamav2==0.0.10+cu121->-r requirements.txt (line 61))\n",
            "  Downloading cramjam-2.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m31.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting httpcore==1.* (from httpx->gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading httpcore-1.0.2-py3-none-any.whl (76 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m76.9/76.9 kB\u001b[0m \u001b[31m10.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx->gradio==3.50.*->-r requirements.txt (line 6)) (1.3.0)\n",
            "Collecting jsonlines (from lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading jsonlines-4.0.0-py3-none-any.whl (8.7 kB)\n",
            "Requirement already satisfied: numexpr in /usr/local/lib/python3.10/dist-packages (from lm-eval->autoawq==0.1.7->-r requirements.txt (line 89)) (2.8.7)\n",
            "Collecting openai>=0.6.4 (from lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading openai-1.3.9-py3-none-any.whl (221 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m221.4/221.4 kB\u001b[0m \u001b[31m27.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pybind11>=2.6.2 (from lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading pybind11-2.11.1-py3-none-any.whl (227 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m227.7/227.7 kB\u001b[0m \u001b[31m26.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pycountry (from lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading pycountry-23.12.11-py3-none-any.whl (6.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.2/6.2 MB\u001b[0m \u001b[31m45.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pytablewriter (from lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading pytablewriter-1.2.0-py3-none-any.whl (111 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.1/111.1 kB\u001b[0m \u001b[31m14.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting rouge-score>=0.0.4 (from lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading rouge_score-0.1.2.tar.gz (17 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting sacrebleu==1.5.0 (from lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading sacrebleu-1.5.0-py3-none-any.whl (65 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m65.6/65.6 kB\u001b[0m \u001b[31m9.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: scikit-learn>=0.24.1 in /usr/local/lib/python3.10/dist-packages (from lm-eval->autoawq==0.1.7->-r requirements.txt (line 89)) (1.2.2)\n",
            "Collecting sqlitedict (from lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading sqlitedict-2.1.0.tar.gz (21 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting tqdm-multiprocess (from lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading tqdm_multiprocess-0.0.11-py3-none-any.whl (9.8 kB)\n",
            "Collecting zstandard (from lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading zstandard-0.22.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.4/5.4 MB\u001b[0m \u001b[31m32.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting portalocker (from sacrebleu==1.5.0->lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading portalocker-2.8.2-py3-none-any.whl (17 kB)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->optimum==1.14.0->-r requirements.txt (line 9)) (1.3.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<4.0.0,>=3.7.1->fastapi->gradio==3.50.*->-r requirements.txt (line 6)) (1.2.0)\n",
            "Collecting blessings (from colour-runner>=0.0.5->attributedict->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading blessings-1.7-py3-none-any.whl (18 kB)\n",
            "Collecting ordered-set<4.2.0,>=4.0.2 (from deepdiff>=3.3.0->attributedict->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading ordered_set-4.1.0-py3-none-any.whl (7.6 kB)\n",
            "Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython!=3.1.29,>=1.0.0->wandb->-r requirements.txt (line 21))\n",
            "  Downloading smmap-5.0.1-py3-none-any.whl (24 kB)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from inspecta>=0.1.0->attributedict->autoawq==0.1.7->-r requirements.txt (line 89)) (2.4.0)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.*->-r requirements.txt (line 6)) (2023.11.2)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.*->-r requirements.txt (line 6)) (0.31.1)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.*->-r requirements.txt (line 6)) (0.13.2)\n",
            "Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai>=0.6.4->lm-eval->autoawq==0.1.7->-r requirements.txt (line 89)) (1.7.0)\n",
            "Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 18)) (0.5.1)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard->-r requirements.txt (line 18)) (3.2.2)\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from rouge-score>=0.0.4->lm-eval->autoawq==0.1.7->-r requirements.txt (line 89)) (3.8.1)\n",
            "Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm-eval->autoawq==0.1.7->-r requirements.txt (line 89)) (1.3.2)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm-eval->autoawq==0.1.7->-r requirements.txt (line 89)) (3.2.0)\n",
            "Requirement already satisfied: chardet>=5.2 in /usr/local/lib/python3.10/dist-packages (from tox>=3.0.0->attributedict->autoawq==0.1.7->-r requirements.txt (line 89)) (5.2.0)\n",
            "Requirement already satisfied: platformdirs>=3.10 in /usr/local/lib/python3.10/dist-packages (from tox>=3.0.0->attributedict->autoawq==0.1.7->-r requirements.txt (line 89)) (4.1.0)\n",
            "Requirement already satisfied: pluggy>=1.3 in /usr/local/lib/python3.10/dist-packages (from tox>=3.0.0->attributedict->autoawq==0.1.7->-r requirements.txt (line 89)) (1.3.0)\n",
            "Collecting pyproject-api>=1.6.1 (from tox>=3.0.0->attributedict->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading pyproject_api-1.6.1-py3-none-any.whl (12 kB)\n",
            "Requirement already satisfied: tomli>=2.0.1 in /usr/local/lib/python3.10/dist-packages (from tox>=3.0.0->attributedict->autoawq==0.1.7->-r requirements.txt (line 89)) (2.0.1)\n",
            "Collecting virtualenv>=20.24.3 (from tox>=3.0.0->attributedict->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading virtualenv-20.25.0-py3-none-any.whl (3.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.8/3.8 MB\u001b[0m \u001b[31m103.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting DataProperty<2,>=1.0.1 (from pytablewriter->lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading DataProperty-1.0.1-py3-none-any.whl (27 kB)\n",
            "Collecting mbstrdecoder<2,>=1.0.0 (from pytablewriter->lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading mbstrdecoder-1.1.3-py3-none-any.whl (7.8 kB)\n",
            "Collecting pathvalidate<4,>=2.3.0 (from pytablewriter->lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading pathvalidate-3.2.0-py3-none-any.whl (23 kB)\n",
            "Collecting tabledata<2,>=1.3.1 (from pytablewriter->lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading tabledata-1.3.3-py3-none-any.whl (11 kB)\n",
            "Collecting tcolorpy<1,>=0.0.5 (from pytablewriter->lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading tcolorpy-0.1.4-py3-none-any.whl (7.9 kB)\n",
            "Collecting typepy[datetime]<2,>=1.3.2 (from pytablewriter->lm-eval->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading typepy-1.3.2-py3-none-any.whl (31 kB)\n",
            "Collecting distlib<1,>=0.3.7 (from virtualenv>=20.24.3->tox>=3.0.0->attributedict->autoawq==0.1.7->-r requirements.txt (line 89))\n",
            "  Downloading distlib-0.3.8-py2.py3-none-any.whl (468 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m468.9/468.9 kB\u001b[0m \u001b[31m49.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hBuilding wheels for collected packages: torch-grammar, ffmpy, rouge-score, sqlitedict\n",
            "  Building wheel for torch-grammar (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for torch-grammar: filename=torch_grammar-0.3.3-py3-none-any.whl size=8259 sha256=43be34776231fd2667ba9ad3979c115c48a2c24c75a4393f0ef4f4ac2379d364\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-_okiz313/wheels/dd/00/c9/08292a4496a8b7e174aefc6e0a8194cbb53977b7cbdd3006b7\n",
            "  Building wheel for ffmpy (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ffmpy: filename=ffmpy-0.3.1-py3-none-any.whl size=5579 sha256=e4e2ae4e7b928090ac89be1cf9933975b00b6bedd8c1a909afc8841568ae25f2\n",
            "  Stored in directory: /root/.cache/pip/wheels/01/a6/d1/1c0828c304a4283b2c1639a09ad86f83d7c487ef34c6b4a1bf\n",
            "  Building wheel for rouge-score (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for rouge-score: filename=rouge_score-0.1.2-py3-none-any.whl size=24933 sha256=14b9b55e789f27c62f81b4e20f6926a69ba30b0e624eaacea1d3eba28185777f\n",
            "  Stored in directory: /root/.cache/pip/wheels/5f/dd/89/461065a73be61a532ff8599a28e9beef17985c9e9c31e541b4\n",
            "  Building wheel for sqlitedict (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sqlitedict: filename=sqlitedict-2.1.0-py3-none-any.whl size=16863 sha256=102fa165cbab13ba0a2198aa2f6f58a7590e9909e5dc4507210ad9e09f060a05\n",
            "  Stored in directory: /root/.cache/pip/wheels/79/d6/e7/304e0e6cb2221022c26d8161f7c23cd4f259a9e41e8bbcfabd\n",
            "Successfully built torch-grammar ffmpy rouge-score sqlitedict\n",
            "Installing collected packages: texttable, sqlitedict, sentencepiece, pydub, ninja, gptq-for-llama, ffmpy, distlib, bitsandbytes, zstandard, websockets, virtualenv, typing-extensions, tcolorpy, smmap, setproctitle, sentry-sdk, semantic-version, rouge, python-multipart, pyproject-api, pycountry, pybind11, pyarrow-hotfix, portalocker, Pillow, pathvalidate, orjson, ordered-set, numpy, mbstrdecoder, jsonlines, humanfriendly, h11, einops, docker-pycreds, dill, cramjam, coverage, colorama, blessings, aiofiles, uvicorn, typepy, tqdm-multiprocess, tox, starlette, sacrebleu, rouge-score, multiprocess, llama-cpp-python-cuda, llama-cpp-python, httpcore, gitdb, gekko, deepdiff, colour-runner, coloredlogs, codecov, rootpath, httpx, GitPython, flash-attn, fastparquet, fastapi, exllama, ctransformers, accelerate, wandb, openai, inspecta, gradio-client, exllamav2, datasets, DataProperty, torch-grammar, tabledata, peft, gradio, attributedict, pytablewriter, optimum, auto-gptq, lm-eval, autoawq\n",
            "  Attempting uninstall: typing-extensions\n",
            "    Found existing installation: typing_extensions 4.5.0\n",
            "    Uninstalling typing_extensions-4.5.0:\n",
            "      Successfully uninstalled typing_extensions-4.5.0\n",
            "  Attempting uninstall: Pillow\n",
            "    Found existing installation: Pillow 9.4.0\n",
            "    Uninstalling Pillow-9.4.0:\n",
            "      Successfully uninstalled Pillow-9.4.0\n",
            "  Attempting uninstall: numpy\n",
            "    Found existing installation: numpy 1.23.5\n",
            "    Uninstalling numpy-1.23.5:\n",
            "      Successfully uninstalled numpy-1.23.5\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "lida 0.0.10 requires kaleido, which is not installed.\n",
            "llmx 0.0.15a0 requires cohere, which is not installed.\n",
            "llmx 0.0.15a0 requires tiktoken, which is not installed.\n",
            "imageio 2.31.6 requires pillow<10.1.0,>=8.3.2, but you have pillow 10.1.0 which is incompatible.\n",
            "tensorflow-probability 0.22.0 requires typing-extensions<4.6.0, but you have typing-extensions 4.9.0 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed DataProperty-1.0.1 GitPython-3.1.40 Pillow-10.1.0 accelerate-0.25.0 aiofiles-23.2.1 attributedict-0.3.0 auto-gptq-0.5.1+cu121 autoawq-0.1.7 bitsandbytes-0.41.1 blessings-1.7 codecov-2.1.13 colorama-0.4.6 coloredlogs-15.0.1 colour-runner-0.1.1 coverage-7.3.2 cramjam-2.7.0 ctransformers-0.2.27+cu121 datasets-2.15.0 deepdiff-6.7.1 dill-0.3.7 distlib-0.3.8 docker-pycreds-0.4.0 einops-0.7.0 exllama-0.0.18+cu121 exllamav2-0.0.10+cu121 fastapi-0.105.0 fastparquet-2023.10.1 ffmpy-0.3.1 flash-attn-2.3.4 gekko-1.0.6 gitdb-4.0.11 gptq-for-llama-0.1.1+cu121 gradio-3.50.2 gradio-client-0.6.1 h11-0.14.0 httpcore-1.0.2 httpx-0.25.2 humanfriendly-10.0 inspecta-0.1.3 jsonlines-4.0.0 llama-cpp-python-0.2.19+cpuavx2 llama-cpp-python-cuda-0.2.19+cu121 lm-eval-0.3.0 mbstrdecoder-1.1.3 multiprocess-0.70.15 ninja-1.11.1.1 numpy-1.24.4 openai-1.3.9 optimum-1.14.0 ordered-set-4.1.0 orjson-3.9.10 pathvalidate-3.2.0 peft-0.6.2 portalocker-2.8.2 pyarrow-hotfix-0.6 pybind11-2.11.1 pycountry-23.12.11 pydub-0.25.1 pyproject-api-1.6.1 pytablewriter-1.2.0 python-multipart-0.0.6 rootpath-0.1.1 rouge-1.0.1 rouge-score-0.1.2 sacrebleu-1.5.0 semantic-version-2.10.0 sentencepiece-0.1.99 sentry-sdk-1.39.0 setproctitle-1.3.3 smmap-5.0.1 sqlitedict-2.1.0 starlette-0.27.0 tabledata-1.3.3 tcolorpy-0.1.4 texttable-1.7.0 torch-grammar-0.3.3 tox-4.11.4 tqdm-multiprocess-0.0.11 typepy-1.3.2 typing-extensions-4.9.0 uvicorn-0.24.0.post1 virtualenv-20.25.0 wandb-0.16.1 websockets-11.0.3 zstandard-0.22.0\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "PIL",
                  "numpy"
                ]
              }
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: gradio==3.50.2 in /usr/local/lib/python3.10/dist-packages (3.50.2)\n",
            "Requirement already satisfied: aiofiles<24.0,>=22.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (23.2.1)\n",
            "Requirement already satisfied: altair<6.0,>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (4.2.2)\n",
            "Requirement already satisfied: fastapi in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (0.105.0)\n",
            "Requirement already satisfied: ffmpy in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (0.3.1)\n",
            "Requirement already satisfied: gradio-client==0.6.1 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (0.6.1)\n",
            "Requirement already satisfied: httpx in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (0.25.2)\n",
            "Requirement already satisfied: huggingface-hub>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (0.19.4)\n",
            "Requirement already satisfied: importlib-resources<7.0,>=1.3 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (6.1.1)\n",
            "Requirement already satisfied: jinja2<4.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (3.1.2)\n",
            "Requirement already satisfied: markupsafe~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (2.1.3)\n",
            "Requirement already satisfied: matplotlib~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (3.7.1)\n",
            "Requirement already satisfied: numpy~=1.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (1.24.4)\n",
            "Requirement already satisfied: orjson~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (3.9.10)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (23.2)\n",
            "Requirement already satisfied: pandas<3.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (1.5.3)\n",
            "Requirement already satisfied: pillow<11.0,>=8.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (10.1.0)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (1.10.13)\n",
            "Requirement already satisfied: pydub in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (0.25.1)\n",
            "Requirement already satisfied: python-multipart in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (0.0.6)\n",
            "Requirement already satisfied: pyyaml<7.0,>=5.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (6.0.1)\n",
            "Requirement already satisfied: requests~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (2.31.0)\n",
            "Requirement already satisfied: semantic-version~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (2.10.0)\n",
            "Requirement already satisfied: typing-extensions~=4.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (4.9.0)\n",
            "Requirement already satisfied: uvicorn>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (0.24.0.post1)\n",
            "Requirement already satisfied: websockets<12.0,>=10.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (11.0.3)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio-client==0.6.1->gradio==3.50.2) (2023.6.0)\n",
            "Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==3.50.2) (0.4)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==3.50.2) (4.19.2)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==3.50.2) (0.12.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.14.0->gradio==3.50.2) (3.13.1)\n",
            "Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.14.0->gradio==3.50.2) (4.66.1)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (1.2.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (4.46.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (1.4.5)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (3.1.1)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas<3.0,>=1.0->gradio==3.50.2) (2023.3.post1)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio==3.50.2) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio==3.50.2) (3.6)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio==3.50.2) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio==3.50.2) (2023.11.17)\n",
            "Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/dist-packages (from uvicorn>=0.14.0->gradio==3.50.2) (8.1.7)\n",
            "Requirement already satisfied: h11>=0.8 in /usr/local/lib/python3.10/dist-packages (from uvicorn>=0.14.0->gradio==3.50.2) (0.14.0)\n",
            "Requirement already satisfied: anyio<4.0.0,>=3.7.1 in /usr/local/lib/python3.10/dist-packages (from fastapi->gradio==3.50.2) (3.7.1)\n",
            "Requirement already satisfied: starlette<0.28.0,>=0.27.0 in /usr/local/lib/python3.10/dist-packages (from fastapi->gradio==3.50.2) (0.27.0)\n",
            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx->gradio==3.50.2) (1.0.2)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx->gradio==3.50.2) (1.3.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<4.0.0,>=3.7.1->fastapi->gradio==3.50.2) (1.2.0)\n",
            "Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.2) (23.1.0)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.2) (2023.11.2)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.2) (0.31.1)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.2) (0.13.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib~=3.0->gradio==3.50.2) (1.16.0)\n",
            "/bin/bash: line 1: conda: command not found\n",
            "[Errno 2] No such file or directory: '/content/text-generation-webui/repositories'\n",
            "/content/text-generation-webui\n",
            "Cloning into 'llama-cpp-python'...\n",
            "remote: Enumerating objects: 5590, done.\u001b[K\n",
            "remote: Counting objects: 100% (1984/1984), done.\u001b[K\n",
            "remote: Compressing objects: 100% (543/543), done.\u001b[K\n",
            "remote: Total 5590 (delta 1638), reused 1646 (delta 1411), pack-reused 3606\u001b[K\n",
            "Receiving objects: 100% (5590/5590), 2.66 MiB | 20.50 MiB/s, done.\n",
            "Resolving deltas: 100% (3514/3514), done.\n",
            "/content/text-generation-webui/llama-cpp-python/vendor\n",
            "Cloning into 'llama.cpp'...\n",
            "remote: Enumerating objects: 13783, done.\u001b[K\n",
            "remote: Counting objects: 100% (4277/4277), done.\u001b[K\n",
            "remote: Compressing objects: 100% (422/422), done.\u001b[K\n",
            "remote: Total 13783 (delta 4078), reused 3910 (delta 3855), pack-reused 9506\u001b[K\n",
            "Receiving objects: 100% (13783/13783), 15.86 MiB | 10.67 MiB/s, done.\n",
            "Resolving deltas: 100% (9623/9623), done.\n",
            "/content/text-generation-webui/llama-cpp-python\n",
            "exllama                          0.0.18+cu121\n",
            "exllamav2                        0.0.10+cu121\n",
            "gptq-for-llama                   0.1.1+cu121\n",
            "llama_cpp_python                 0.2.19+cpuavx2\n",
            "llama_cpp_python_cuda            0.2.19+cu121\n",
            "Found existing installation: llama_cpp_python 0.2.19+cpuavx2\n",
            "Uninstalling llama_cpp_python-0.2.19+cpuavx2:\n",
            "  Successfully uninstalled llama_cpp_python-0.2.19+cpuavx2\n",
            "Found existing installation: llama_cpp_python_cuda 0.2.19+cu121\n",
            "Uninstalling llama_cpp_python_cuda-0.2.19+cu121:\n",
            "  Successfully uninstalled llama_cpp_python_cuda-0.2.19+cu121\n",
            "Processing /content/text-generation-webui/llama-cpp-python\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Installing backend dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: typing-extensions>=4.5.0 in /usr/local/lib/python3.10/dist-packages (from llama_cpp_python==0.2.23) (4.9.0)\n",
            "Requirement already satisfied: numpy>=1.20.0 in /usr/local/lib/python3.10/dist-packages (from llama_cpp_python==0.2.23) (1.24.4)\n",
            "Requirement already satisfied: diskcache>=5.6.1 in /usr/local/lib/python3.10/dist-packages (from llama_cpp_python==0.2.23) (5.6.3)\n",
            "Building wheels for collected packages: llama_cpp_python\n",
            "  Building wheel for llama_cpp_python (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for llama_cpp_python: filename=llama_cpp_python-0.2.23-cp310-cp310-manylinux_2_35_x86_64.whl size=7942570 sha256=f07f61cedc2c32a580629c6ad0ab9c45a16b1b7c8265ca0feb743bf3886f641f\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-grm9dqhy/wheels/eb/72/de/806ff773f742e835fa3e0a2b5745c3034198a5eb055e644426\n",
            "Successfully built llama_cpp_python\n",
            "Installing collected packages: llama_cpp_python\n",
            "Successfully installed llama_cpp_python-0.2.23\n",
            "Found existing installation: flash-attn 2.3.4\n",
            "Uninstalling flash-attn-2.3.4:\n",
            "  Successfully uninstalled flash-attn-2.3.4\n",
            "Collecting flash-attn==2.3.4\n",
            "  Downloading flash_attn-2.3.4.tar.gz (2.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.3/2.3 MB\u001b[0m \u001b[31m30.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.3.4) (2.1.0+cu118)\n",
            "Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.3.4) (0.7.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.3.4) (23.2)\n",
            "Requirement already satisfied: ninja in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.3.4) (1.11.1.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.3.4) (3.13.1)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.3.4) (4.9.0)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.3.4) (1.12)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.3.4) (3.2.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.3.4) (3.1.2)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.3.4) (2023.6.0)\n",
            "Requirement already satisfied: triton==2.1.0 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.3.4) (2.1.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->flash-attn==2.3.4) (2.1.3)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch->flash-attn==2.3.4) (1.3.0)\n",
            "Building wheels for collected packages: flash-attn\n",
            "  Building wheel for flash-attn (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for flash-attn: filename=flash_attn-2.3.4-cp310-cp310-linux_x86_64.whl size=57449248 sha256=4ceff7f9ad6a6747975b568f3193b2416f598808b5e860fb09555dbfa32b42cf\n",
            "  Stored in directory: /root/.cache/pip/wheels/32/19/1b/bf8147431c43f8b8d2b2e5d66be1ebf586f2fca96d72203f53\n",
            "Successfully built flash-attn\n",
            "Installing collected packages: flash-attn\n",
            "Successfully installed flash-attn-2.3.4\n"
          ]
        }
      ],
      "source": [
        "%cd /content\n",
        "!apt-get -y install -qq aria2\n",
        "\n",
        "!git clone -b V20231211 https://github.com/Troyanovsky/text-generation-webui\n",
        "%cd /content/text-generation-webui\n",
        "!pip install -r requirements.txt\n",
        "!pip install -U gradio==3.50.2\n",
        "\n",
        "!conda activate textgen\n",
        "\n",
        "%cd /content/text-generation-webui/repositories\n",
        "\n",
        "!git clone https://github.com/abetlen/llama-cpp-python\n",
        "%cd llama-cpp-python/vendor\n",
        "!rm -R llama.cpp\n",
        "!git clone --branch=mixtral https://github.com/ggerganov/llama.cpp.git\n",
        "%cd ..\n",
        "!pip list | grep llama\n",
        "!pip uninstall -y llama_cpp_python\n",
        "!pip uninstall -y llama_cpp_python_cuda\n",
        "!CMAKE_ARGS=\"-DLLAMA_CUDA=on\" FORCE_CMAKE=1 pip install . --no-cache-dir\n",
        "\n",
        "!pip uninstall flash-attn -y\n",
        "!pip install --no-build-isolation flash-attn==2.3.4"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Q2_K, some on GPU, faster generation (1.68 T/s)\n",
        "!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF/resolve/main/mixtral-8x7b-instruct-v0.1.Q2_K.gguf?download=true -d /content/text-generation-webui/models/ -o mixtral-8x7b-instruct-v0.1.Q2_K.gguf\n",
        "\n",
        "%cd /content/text-generation-webui\n",
        "!python server.py --share --n-gpu-layers 20 --n_ctx 2048 --model mixtral-8x7b-instruct-v0.1.Q2_K.gguf"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BzXpIHFQG9gD",
        "outputId": "ab4b1176-9c0e-49ff-e32f-eb38b528d439"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/text-generation-webui\n",
            "2023-12-14 07:38:49 INFO:\u001b[32mLoading mixtral-8x7b-instruct-v0.1.Q2_K.gguf...\u001b[0m\n",
            "2023-12-14 07:38:49 INFO:\u001b[32mllama.cpp weights detected: models/mixtral-8x7b-instruct-v0.1.Q2_K.gguf\u001b[0m\n",
            "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
            "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
            "ggml_init_cublas: found 1 CUDA devices:\n",
            "  Device 0: Tesla T4, compute capability 7.5\n",
            "llama_model_loader: loaded meta data with 26 key-value pairs and 995 tensors from models/mixtral-8x7b-instruct-v0.1.Q2_K.gguf (version GGUF V3 (latest))\n",
            "llama_model_loader: - tensor    0:                token_embd.weight q2_K     [  4096, 32000,     1,     1 ]\n",
            "llama_model_loader: - tensor    1:          blk.0.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor    2:          blk.0.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor    3:            blk.0.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor    4:          blk.0.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor    5:          blk.0.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor    6:            blk.0.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor    7:          blk.0.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor    8:          blk.0.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor    9:            blk.0.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   10:          blk.0.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   11:          blk.0.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   12:            blk.0.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   13:          blk.0.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   14:          blk.0.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   15:            blk.0.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   16:          blk.0.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   17:          blk.0.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   18:            blk.0.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   19:          blk.0.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   20:          blk.0.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   21:            blk.0.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   22:          blk.0.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   23:          blk.0.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   24:            blk.0.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   25:        blk.0.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor   26:           blk.0.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   27:            blk.0.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   28:              blk.0.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   29:         blk.0.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   30:              blk.0.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   31:              blk.0.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   32:          blk.1.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   33:          blk.1.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   34:            blk.1.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   35:          blk.1.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   36:          blk.1.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   37:            blk.1.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   38:          blk.1.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   39:          blk.1.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   40:            blk.1.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   41:          blk.1.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   42:          blk.1.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   43:            blk.1.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   44:          blk.1.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   45:          blk.1.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   46:        blk.1.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor   47:              blk.1.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   48:         blk.1.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   49:              blk.1.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   50:              blk.1.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   51:            blk.1.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   52:          blk.1.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   53:          blk.1.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   54:            blk.1.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   55:          blk.1.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   56:          blk.1.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   57:            blk.1.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   58:          blk.1.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   59:          blk.1.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   60:            blk.1.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   61:           blk.1.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   62:            blk.1.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   63:          blk.2.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   64:          blk.2.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   65:            blk.2.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   66:          blk.2.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   67:          blk.2.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   68:            blk.2.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   69:          blk.2.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   70:          blk.2.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   71:            blk.2.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   72:          blk.2.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   73:          blk.2.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   74:            blk.2.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   75:          blk.2.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   76:          blk.2.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   77:            blk.2.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   78:          blk.2.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   79:          blk.2.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   80:            blk.2.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   81:          blk.2.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   82:          blk.2.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   83:            blk.2.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   84:          blk.2.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   85:          blk.2.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   86:            blk.2.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   87:        blk.2.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor   88:           blk.2.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   89:            blk.2.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   90:              blk.2.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   91:         blk.2.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   92:              blk.2.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   93:              blk.2.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   94:          blk.3.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   95:          blk.3.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   96:            blk.3.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   97:          blk.3.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   98:          blk.3.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   99:            blk.3.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  100:          blk.3.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  101:        blk.3.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  102:              blk.3.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  103:         blk.3.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  104:              blk.3.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  105:              blk.3.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  106:          blk.3.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  107:            blk.3.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  108:          blk.3.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  109:          blk.3.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  110:            blk.3.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  111:          blk.3.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  112:          blk.3.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  113:            blk.3.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  114:          blk.3.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  115:          blk.3.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  116:            blk.3.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  117:          blk.3.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  118:          blk.3.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  119:            blk.3.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  120:          blk.3.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  121:          blk.3.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  122:            blk.3.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  123:           blk.3.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  124:            blk.3.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  125:          blk.4.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  126:          blk.4.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  127:            blk.4.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  128:          blk.4.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  129:          blk.4.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  130:            blk.4.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  131:          blk.4.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  132:          blk.4.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  133:            blk.4.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  134:          blk.4.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  135:          blk.4.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  136:            blk.4.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  137:          blk.4.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  138:          blk.4.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  139:            blk.4.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  140:          blk.4.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  141:          blk.4.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  142:            blk.4.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  143:          blk.4.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  144:          blk.4.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  145:            blk.4.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  146:          blk.4.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  147:          blk.4.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  148:            blk.4.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  149:        blk.4.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  150:           blk.4.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  151:            blk.4.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  152:              blk.4.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  153:         blk.4.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  154:              blk.4.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  155:              blk.4.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  156:        blk.5.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  157:              blk.5.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  158:         blk.5.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  159:              blk.5.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  160:              blk.5.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  161:          blk.5.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  162:          blk.5.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  163:            blk.5.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  164:          blk.5.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  165:          blk.5.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  166:            blk.5.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  167:          blk.5.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  168:          blk.5.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  169:            blk.5.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  170:          blk.5.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  171:          blk.5.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  172:            blk.5.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  173:          blk.5.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  174:          blk.5.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  175:            blk.5.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  176:          blk.5.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  177:          blk.5.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  178:            blk.5.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  179:          blk.5.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  180:          blk.5.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  181:            blk.5.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  182:          blk.5.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  183:          blk.5.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  184:            blk.5.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  185:           blk.5.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  186:            blk.5.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  187:          blk.6.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  188:          blk.6.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  189:            blk.6.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  190:          blk.6.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  191:          blk.6.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  192:            blk.6.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  193:          blk.6.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  194:          blk.6.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  195:            blk.6.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  196:          blk.6.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  197:          blk.6.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  198:            blk.6.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  199:          blk.6.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  200:          blk.6.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  201:            blk.6.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  202:          blk.6.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  203:          blk.6.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  204:        blk.6.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  205:              blk.6.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  206:         blk.6.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  207:              blk.6.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  208:              blk.6.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  209:            blk.6.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  210:          blk.6.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  211:          blk.6.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  212:            blk.6.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  213:          blk.6.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  214:          blk.6.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  215:            blk.6.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  216:           blk.6.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  217:            blk.6.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  218:          blk.7.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  219:          blk.7.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  220:            blk.7.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  221:          blk.7.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  222:          blk.7.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  223:            blk.7.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  224:          blk.7.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  225:          blk.7.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  226:            blk.7.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  227:          blk.7.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  228:          blk.7.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  229:            blk.7.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  230:          blk.7.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  231:          blk.7.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  232:            blk.7.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  233:          blk.7.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  234:          blk.7.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  235:            blk.7.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  236:          blk.7.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  237:          blk.7.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  238:            blk.7.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  239:          blk.7.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  240:          blk.7.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  241:            blk.7.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  242:        blk.7.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  243:           blk.7.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  244:            blk.7.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  245:              blk.7.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  246:         blk.7.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  247:              blk.7.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  248:              blk.7.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  249:          blk.8.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  250:          blk.8.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  251:            blk.8.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  252:          blk.8.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  253:          blk.8.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  254:            blk.8.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  255:          blk.8.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  256:          blk.8.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  257:            blk.8.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  258:          blk.8.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  259:        blk.8.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  260:              blk.8.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  261:         blk.8.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  262:              blk.8.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  263:              blk.8.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  264:         blk.10.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  265:         blk.10.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  266:           blk.10.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  267:       blk.10.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  268:             blk.10.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  269:        blk.10.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  270:             blk.10.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  271:             blk.10.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  272:          blk.8.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  273:            blk.8.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  274:          blk.8.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  275:          blk.8.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  276:            blk.8.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  277:          blk.8.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  278:          blk.8.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  279:            blk.8.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  280:          blk.8.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  281:          blk.8.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  282:            blk.8.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  283:          blk.8.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  284:          blk.8.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  285:            blk.8.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  286:           blk.8.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  287:            blk.8.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  288:          blk.9.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  289:          blk.9.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  290:            blk.9.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  291:          blk.9.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  292:          blk.9.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  293:            blk.9.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  294:          blk.9.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  295:          blk.9.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  296:            blk.9.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  297:          blk.9.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  298:          blk.9.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  299:            blk.9.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  300:          blk.9.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  301:          blk.9.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  302:            blk.9.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  303:          blk.9.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  304:          blk.9.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  305:            blk.9.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  306:          blk.9.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  307:          blk.9.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  308:            blk.9.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  309:          blk.9.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  310:          blk.9.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  311:            blk.9.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  312:        blk.9.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  313:           blk.9.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  314:            blk.9.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  315:              blk.9.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  316:         blk.9.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  317:              blk.9.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  318:              blk.9.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  319:         blk.10.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  320:         blk.10.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  321:           blk.10.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  322:         blk.10.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  323:         blk.10.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  324:           blk.10.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  325:         blk.10.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  326:         blk.10.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  327:           blk.10.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  328:         blk.10.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  329:         blk.10.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  330:           blk.10.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  331:         blk.10.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  332:         blk.10.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  333:           blk.10.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  334:         blk.10.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  335:         blk.10.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  336:           blk.10.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  337:         blk.10.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  338:         blk.10.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  339:           blk.10.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  340:          blk.10.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  341:           blk.10.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  342:         blk.11.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  343:         blk.11.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  344:           blk.11.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  345:         blk.11.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  346:         blk.11.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  347:           blk.11.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  348:         blk.11.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  349:         blk.11.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  350:           blk.11.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  351:         blk.11.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  352:         blk.11.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  353:           blk.11.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  354:         blk.11.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  355:         blk.11.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  356:           blk.11.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  357:         blk.11.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  358:         blk.11.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  359:           blk.11.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  360:         blk.11.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  361:         blk.11.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  362:       blk.11.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  363:             blk.11.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  364:        blk.11.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  365:             blk.11.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  366:             blk.11.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  367:           blk.11.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  368:         blk.11.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  369:         blk.11.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  370:           blk.11.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  371:          blk.11.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  372:           blk.11.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  373:         blk.12.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  374:         blk.12.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  375:           blk.12.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  376:         blk.12.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  377:         blk.12.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  378:           blk.12.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  379:         blk.12.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  380:         blk.12.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  381:           blk.12.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  382:         blk.12.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  383:         blk.12.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  384:           blk.12.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  385:         blk.12.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  386:         blk.12.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  387:           blk.12.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  388:         blk.12.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  389:         blk.12.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  390:           blk.12.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  391:         blk.12.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  392:         blk.12.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  393:           blk.12.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  394:         blk.12.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  395:         blk.12.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  396:           blk.12.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  397:       blk.12.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  398:          blk.12.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  399:           blk.12.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  400:             blk.12.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  401:        blk.12.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  402:             blk.12.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  403:             blk.12.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  404:         blk.13.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  405:         blk.13.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  406:           blk.13.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  407:         blk.13.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  408:         blk.13.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  409:           blk.13.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  410:         blk.13.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  411:         blk.13.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  412:           blk.13.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  413:         blk.13.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  414:         blk.13.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  415:           blk.13.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  416:         blk.13.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  417:       blk.13.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  418:             blk.13.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  419:        blk.13.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  420:             blk.13.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  421:             blk.13.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  422:         blk.13.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  423:           blk.13.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  424:         blk.13.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  425:         blk.13.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  426:           blk.13.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  427:         blk.13.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  428:         blk.13.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  429:           blk.13.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  430:         blk.13.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  431:         blk.13.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  432:           blk.13.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  433:          blk.13.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  434:           blk.13.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  435:         blk.14.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  436:         blk.14.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  437:           blk.14.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  438:         blk.14.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  439:         blk.14.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  440:           blk.14.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  441:         blk.14.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  442:         blk.14.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  443:           blk.14.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  444:         blk.14.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  445:         blk.14.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  446:           blk.14.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  447:         blk.14.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  448:         blk.14.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  449:           blk.14.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  450:         blk.14.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  451:         blk.14.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  452:           blk.14.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  453:         blk.14.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  454:         blk.14.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  455:           blk.14.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  456:         blk.14.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  457:         blk.14.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  458:           blk.14.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  459:       blk.14.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  460:          blk.14.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  461:           blk.14.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  462:             blk.14.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  463:        blk.14.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  464:             blk.14.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  465:             blk.14.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  466:         blk.15.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  467:         blk.15.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  468:           blk.15.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  469:         blk.15.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  470:         blk.15.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  471:           blk.15.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  472:       blk.15.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  473:             blk.15.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  474:        blk.15.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  475:             blk.15.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  476:             blk.15.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  477:         blk.15.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  478:         blk.15.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  479:           blk.15.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  480:         blk.15.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  481:         blk.15.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  482:           blk.15.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  483:         blk.15.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  484:         blk.15.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  485:           blk.15.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  486:         blk.15.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  487:         blk.15.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  488:           blk.15.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  489:         blk.15.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  490:         blk.15.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  491:           blk.15.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  492:         blk.15.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  493:         blk.15.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  494:           blk.15.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  495:          blk.15.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  496:           blk.15.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  497:         blk.16.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  498:         blk.16.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  499:           blk.16.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  500:         blk.16.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  501:         blk.16.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  502:           blk.16.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  503:         blk.16.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  504:         blk.16.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  505:           blk.16.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  506:         blk.16.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  507:         blk.16.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  508:           blk.16.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  509:         blk.16.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  510:         blk.16.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  511:           blk.16.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  512:         blk.16.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  513:         blk.16.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  514:           blk.16.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  515:         blk.16.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  516:         blk.16.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  517:           blk.16.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  518:         blk.16.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  519:         blk.16.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  520:       blk.16.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  521:             blk.16.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  522:        blk.16.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  523:             blk.16.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  524:             blk.16.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  525:           blk.16.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  526:          blk.16.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  527:           blk.16.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  528:         blk.17.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  529:         blk.17.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  530:           blk.17.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  531:         blk.17.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  532:         blk.17.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  533:           blk.17.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  534:         blk.17.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  535:         blk.17.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  536:           blk.17.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  537:         blk.17.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  538:         blk.17.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  539:           blk.17.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  540:         blk.17.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  541:         blk.17.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  542:           blk.17.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  543:         blk.17.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  544:         blk.17.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  545:           blk.17.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  546:         blk.17.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  547:         blk.17.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  548:           blk.17.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  549:         blk.17.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  550:         blk.17.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  551:           blk.17.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  552:       blk.17.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  553:          blk.17.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  554:           blk.17.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  555:             blk.17.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  556:        blk.17.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  557:             blk.17.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  558:             blk.17.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  559:         blk.18.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  560:         blk.18.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  561:           blk.18.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  562:         blk.18.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  563:         blk.18.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  564:           blk.18.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  565:         blk.18.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  566:         blk.18.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  567:           blk.18.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  568:         blk.18.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  569:         blk.18.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  570:           blk.18.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  571:         blk.18.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  572:         blk.18.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  573:           blk.18.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  574:         blk.18.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  575:       blk.18.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  576:             blk.18.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  577:        blk.18.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  578:             blk.18.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  579:             blk.18.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  580:         blk.18.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  581:           blk.18.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  582:         blk.18.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  583:         blk.18.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  584:           blk.18.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  585:         blk.18.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  586:         blk.18.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  587:           blk.18.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  588:          blk.18.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  589:           blk.18.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  590:         blk.19.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  591:         blk.19.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  592:           blk.19.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  593:         blk.19.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  594:         blk.19.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  595:           blk.19.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  596:         blk.19.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  597:         blk.19.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  598:           blk.19.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  599:         blk.19.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  600:         blk.19.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  601:           blk.19.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  602:         blk.19.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  603:         blk.19.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  604:           blk.19.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  605:         blk.19.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  606:         blk.19.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  607:           blk.19.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  608:         blk.19.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  609:         blk.19.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  610:           blk.19.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  611:         blk.19.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  612:         blk.19.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  613:           blk.19.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  614:       blk.19.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  615:          blk.19.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  616:           blk.19.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  617:             blk.19.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  618:        blk.19.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  619:             blk.19.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  620:             blk.19.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  621:         blk.20.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  622:         blk.20.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  623:           blk.20.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  624:         blk.20.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  625:         blk.20.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  626:           blk.20.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  627:         blk.20.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  628:         blk.20.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  629:           blk.20.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  630:       blk.20.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  631:             blk.20.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  632:        blk.20.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  633:             blk.20.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  634:             blk.20.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  635:         blk.20.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  636:         blk.20.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  637:           blk.20.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  638:         blk.20.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  639:         blk.20.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  640:           blk.20.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  641:         blk.20.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  642:         blk.20.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  643:           blk.20.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  644:         blk.20.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  645:         blk.20.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  646:           blk.20.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  647:         blk.20.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  648:         blk.20.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  649:           blk.20.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  650:          blk.20.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  651:           blk.20.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  652:         blk.21.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  653:         blk.21.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  654:           blk.21.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  655:         blk.21.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  656:         blk.21.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  657:           blk.21.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  658:         blk.21.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  659:         blk.21.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  660:           blk.21.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  661:         blk.21.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  662:         blk.21.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  663:           blk.21.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  664:         blk.21.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  665:         blk.21.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  666:           blk.21.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  667:         blk.21.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  668:         blk.21.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  669:           blk.21.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  670:         blk.21.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  671:         blk.21.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  672:           blk.21.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  673:         blk.21.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  674:         blk.21.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  675:           blk.21.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  676:       blk.21.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  677:          blk.21.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  678:           blk.21.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  679:             blk.21.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  680:        blk.21.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  681:             blk.21.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  682:             blk.21.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  683:         blk.22.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  684:         blk.22.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  685:       blk.22.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  686:             blk.22.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  687:        blk.22.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  688:             blk.22.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  689:             blk.22.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  690:           blk.22.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  691:         blk.22.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  692:         blk.22.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  693:           blk.22.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  694:         blk.22.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  695:         blk.22.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  696:           blk.22.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  697:         blk.22.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  698:         blk.22.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  699:           blk.22.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  700:         blk.22.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  701:         blk.22.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  702:           blk.22.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  703:         blk.22.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  704:         blk.22.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  705:           blk.22.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  706:         blk.22.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  707:         blk.22.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  708:           blk.22.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  709:         blk.22.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  710:         blk.22.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  711:           blk.22.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  712:          blk.22.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  713:           blk.22.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  714:         blk.23.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  715:         blk.23.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  716:           blk.23.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  717:         blk.23.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  718:         blk.23.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  719:           blk.23.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  720:         blk.23.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  721:         blk.23.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  722:           blk.23.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  723:         blk.23.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  724:         blk.23.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  725:           blk.23.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  726:         blk.23.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  727:         blk.23.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  728:           blk.23.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  729:         blk.23.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  730:         blk.23.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  731:           blk.23.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  732:         blk.23.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  733:       blk.23.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  734:             blk.23.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  735:        blk.23.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  736:             blk.23.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  737:             blk.23.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  738:         blk.23.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  739:           blk.23.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  740:         blk.23.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  741:         blk.23.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  742:           blk.23.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  743:          blk.23.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  744:           blk.23.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  745:         blk.24.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  746:         blk.24.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  747:           blk.24.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  748:         blk.24.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  749:         blk.24.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  750:           blk.24.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  751:         blk.24.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  752:         blk.24.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  753:           blk.24.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  754:         blk.24.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  755:         blk.24.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  756:           blk.24.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  757:         blk.24.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  758:         blk.24.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  759:           blk.24.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  760:         blk.24.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  761:         blk.24.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  762:           blk.24.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  763:         blk.24.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  764:         blk.24.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  765:           blk.24.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  766:         blk.24.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  767:         blk.24.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  768:           blk.24.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  769:       blk.24.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  770:          blk.24.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  771:           blk.24.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  772:             blk.24.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  773:        blk.24.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  774:             blk.24.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  775:             blk.24.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  776:         blk.25.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  777:         blk.25.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  778:           blk.25.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  779:         blk.25.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  780:         blk.25.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  781:           blk.25.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  782:         blk.25.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  783:         blk.25.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  784:           blk.25.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  785:         blk.25.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  786:         blk.25.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  787:           blk.25.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  788:       blk.25.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  789:             blk.25.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  790:        blk.25.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  791:             blk.25.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  792:             blk.25.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  793:         blk.25.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  794:         blk.25.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  795:           blk.25.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  796:         blk.25.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  797:         blk.25.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  798:           blk.25.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  799:         blk.25.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  800:         blk.25.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  801:           blk.25.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  802:         blk.25.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  803:         blk.25.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  804:           blk.25.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  805:          blk.25.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  806:           blk.25.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  807:         blk.26.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  808:         blk.26.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  809:           blk.26.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  810:         blk.26.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  811:         blk.26.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  812:           blk.26.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  813:         blk.26.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  814:         blk.26.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  815:           blk.26.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  816:         blk.26.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  817:         blk.26.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  818:           blk.26.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  819:         blk.26.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  820:         blk.26.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  821:           blk.26.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  822:         blk.26.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  823:         blk.26.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  824:           blk.26.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  825:         blk.26.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  826:         blk.26.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  827:           blk.26.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  828:         blk.26.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  829:         blk.26.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  830:           blk.26.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  831:       blk.26.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  832:          blk.26.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  833:           blk.26.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  834:             blk.26.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  835:        blk.26.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  836:             blk.26.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  837:             blk.26.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  838:         blk.27.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  839:         blk.27.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  840:           blk.27.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  841:         blk.27.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  842:         blk.27.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  843:       blk.27.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  844:             blk.27.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  845:        blk.27.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  846:             blk.27.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  847:             blk.27.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  848:           blk.27.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  849:         blk.27.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  850:         blk.27.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  851:           blk.27.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  852:         blk.27.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  853:         blk.27.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  854:           blk.27.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  855:         blk.27.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  856:         blk.27.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  857:           blk.27.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  858:         blk.27.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  859:         blk.27.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  860:           blk.27.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  861:         blk.27.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  862:         blk.27.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  863:           blk.27.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  864:         blk.27.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  865:         blk.27.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  866:           blk.27.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  867:          blk.27.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  868:           blk.27.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  869:         blk.28.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  870:         blk.28.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  871:           blk.28.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  872:         blk.28.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  873:         blk.28.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  874:           blk.28.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  875:         blk.28.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  876:         blk.28.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  877:           blk.28.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  878:         blk.28.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  879:         blk.28.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  880:           blk.28.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  881:         blk.28.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  882:         blk.28.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  883:           blk.28.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  884:         blk.28.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  885:         blk.28.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  886:           blk.28.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  887:         blk.28.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  888:         blk.28.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  889:           blk.28.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  890:         blk.28.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  891:       blk.28.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  892:             blk.28.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  893:        blk.28.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  894:             blk.28.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  895:             blk.28.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  896:         blk.28.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  897:           blk.28.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  898:          blk.28.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  899:           blk.28.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  900:         blk.29.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  901:         blk.29.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  902:           blk.29.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  903:         blk.29.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  904:         blk.29.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  905:           blk.29.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  906:         blk.29.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  907:         blk.29.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  908:           blk.29.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  909:         blk.29.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  910:         blk.29.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  911:           blk.29.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  912:         blk.29.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  913:         blk.29.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  914:           blk.29.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  915:         blk.29.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  916:         blk.29.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  917:           blk.29.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  918:         blk.29.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  919:         blk.29.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  920:           blk.29.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  921:         blk.29.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  922:         blk.29.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  923:           blk.29.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  924:       blk.29.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  925:          blk.29.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  926:           blk.29.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  927:             blk.29.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  928:        blk.29.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  929:             blk.29.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  930:             blk.29.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  931:         blk.30.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  932:         blk.30.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  933:           blk.30.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  934:         blk.30.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  935:         blk.30.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  936:           blk.30.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  937:         blk.30.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  938:         blk.30.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  939:           blk.30.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  940:         blk.30.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  941:         blk.30.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  942:           blk.30.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  943:         blk.30.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  944:         blk.30.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  945:           blk.30.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  946:       blk.30.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  947:             blk.30.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  948:        blk.30.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  949:             blk.30.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  950:             blk.30.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  951:                    output.weight q6_K     [  4096, 32000,     1,     1 ]\n",
            "llama_model_loader: - tensor  952:         blk.30.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  953:         blk.30.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  954:           blk.30.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  955:         blk.30.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  956:         blk.30.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  957:           blk.30.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  958:         blk.30.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  959:         blk.30.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  960:           blk.30.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  961:          blk.30.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  962:           blk.30.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  963:         blk.31.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  964:         blk.31.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  965:           blk.31.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  966:         blk.31.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  967:         blk.31.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  968:           blk.31.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  969:         blk.31.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  970:         blk.31.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  971:           blk.31.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  972:         blk.31.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  973:         blk.31.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  974:           blk.31.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  975:         blk.31.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  976:         blk.31.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  977:           blk.31.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  978:         blk.31.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  979:         blk.31.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  980:           blk.31.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  981:         blk.31.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  982:         blk.31.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  983:           blk.31.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  984:         blk.31.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  985:         blk.31.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  986:           blk.31.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  987:       blk.31.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  988:          blk.31.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  989:           blk.31.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  990:             blk.31.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  991:        blk.31.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  992:             blk.31.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  993:             blk.31.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  994:               output_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
            "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
            "llama_model_loader: - kv   1:                               general.name str              = mistralai_mixtral-8x7b-instruct-v0.1\n",
            "llama_model_loader: - kv   2:                       llama.context_length u32              = 32768\n",
            "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
            "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
            "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336\n",
            "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
            "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
            "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
            "llama_model_loader: - kv   9:                         llama.expert_count u32              = 8\n",
            "llama_model_loader: - kv  10:                    llama.expert_used_count u32              = 2\n",
            "llama_model_loader: - kv  11:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
            "llama_model_loader: - kv  12:                       llama.rope.freq_base f32              = 1000000.000000\n",
            "llama_model_loader: - kv  13:                          general.file_type u32              = 10\n",
            "llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = llama\n",
            "llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
            "llama_model_loader: - kv  16:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
            "llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
            "llama_model_loader: - kv  18:                tokenizer.ggml.bos_token_id u32              = 1\n",
            "llama_model_loader: - kv  19:                tokenizer.ggml.eos_token_id u32              = 2\n",
            "llama_model_loader: - kv  20:            tokenizer.ggml.unknown_token_id u32              = 0\n",
            "llama_model_loader: - kv  21:            tokenizer.ggml.padding_token_id u32              = 0\n",
            "llama_model_loader: - kv  22:               tokenizer.ggml.add_bos_token bool             = true\n",
            "llama_model_loader: - kv  23:               tokenizer.ggml.add_eos_token bool             = false\n",
            "llama_model_loader: - kv  24:                    tokenizer.chat_template str              = {{ bos_token }}{% for message in mess...\n",
            "llama_model_loader: - kv  25:               general.quantization_version u32              = 2\n",
            "llama_model_loader: - type  f32:   65 tensors\n",
            "llama_model_loader: - type  f16:   32 tensors\n",
            "llama_model_loader: - type q8_0:   64 tensors\n",
            "llama_model_loader: - type q2_K:  801 tensors\n",
            "llama_model_loader: - type q3_K:   32 tensors\n",
            "llama_model_loader: - type q6_K:    1 tensors\n",
            "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
            "llm_load_print_meta: format           = GGUF V3 (latest)\n",
            "llm_load_print_meta: arch             = llama\n",
            "llm_load_print_meta: vocab type       = SPM\n",
            "llm_load_print_meta: n_vocab          = 32000\n",
            "llm_load_print_meta: n_merges         = 0\n",
            "llm_load_print_meta: n_ctx_train      = 32768\n",
            "llm_load_print_meta: n_embd           = 4096\n",
            "llm_load_print_meta: n_head           = 32\n",
            "llm_load_print_meta: n_head_kv        = 8\n",
            "llm_load_print_meta: n_layer          = 32\n",
            "llm_load_print_meta: n_rot            = 128\n",
            "llm_load_print_meta: n_gqa            = 4\n",
            "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
            "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
            "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
            "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
            "llm_load_print_meta: n_ff             = 14336\n",
            "llm_load_print_meta: n_expert         = 8\n",
            "llm_load_print_meta: n_expert_used    = 2\n",
            "llm_load_print_meta: rope scaling     = linear\n",
            "llm_load_print_meta: freq_base_train  = 1000000.0\n",
            "llm_load_print_meta: freq_scale_train = 1\n",
            "llm_load_print_meta: n_yarn_orig_ctx  = 32768\n",
            "llm_load_print_meta: rope_finetuned   = unknown\n",
            "llm_load_print_meta: model type       = 7B\n",
            "llm_load_print_meta: model ftype      = mostly Q2_K\n",
            "llm_load_print_meta: model params     = 46.70 B\n",
            "llm_load_print_meta: model size       = 14.57 GiB (2.68 BPW) \n",
            "llm_load_print_meta: general.name     = mistralai_mixtral-8x7b-instruct-v0.1\n",
            "llm_load_print_meta: BOS token        = 1 '<s>'\n",
            "llm_load_print_meta: EOS token        = 2 '</s>'\n",
            "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
            "llm_load_print_meta: PAD token        = 0 '<unk>'\n",
            "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
            "llm_load_tensors: ggml ctx size =    0.39 MiB\n",
            "llm_load_tensors: using CUDA for GPU acceleration\n",
            "llm_load_tensors: mem required  = 5684.59 MiB\n",
            "llm_load_tensors: offloading 20 repeating layers to GPU\n",
            "llm_load_tensors: offloaded 20/33 layers to GPU\n",
            "llm_load_tensors: VRAM used: 9234.38 MiB\n",
            "....................................................................................................\n",
            "llama_new_context_with_model: n_ctx      = 2048\n",
            "llama_new_context_with_model: freq_base  = 1000000.0\n",
            "llama_new_context_with_model: freq_scale = 1\n",
            "llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB\n",
            "llama_build_graph: non-view tensors processed: 1124/1124\n",
            "llama_new_context_with_model: compute buffer total size = 187.35 MiB\n",
            "llama_new_context_with_model: VRAM scratch buffer: 184.04 MiB\n",
            "llama_new_context_with_model: total VRAM used: 9418.41 MiB (model: 9234.38 MiB, context: 184.04 MiB)\n",
            "AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
            "2023-12-14 07:40:52 INFO:\u001b[32mLOADER: llama.cpp\u001b[0m\n",
            "2023-12-14 07:40:52 INFO:\u001b[32mTRUNCATION LENGTH: 2048\u001b[0m\n",
            "2023-12-14 07:40:52 INFO:\u001b[32mINSTRUCTION TEMPLATE: Alpaca\u001b[0m\n",
            "2023-12-14 07:40:52 INFO:\u001b[32mLoaded the model in 123.69 seconds.\u001b[0m\n",
            "2023-12-14 07:40:52 INFO:\u001b[32mLoading the extension \"gallery\"...\u001b[0m\n",
            "Running on local URL:  http://127.0.0.1:7860\n",
            "Running on public URL: https://f14565e45ce53884a9.gradio.live\n",
            "\n",
            "This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)\n",
            "\n",
            "llama_print_timings:        load time =   35097.15 ms\n",
            "llama_print_timings:      sample time =      57.54 ms /    97 runs   (    0.59 ms per token,  1685.87 tokens per second)\n",
            "llama_print_timings: prompt eval time =   35097.06 ms /    26 tokens ( 1349.89 ms per token,     0.74 tokens per second)\n",
            "llama_print_timings:        eval time =   57976.70 ms /    96 runs   (  603.92 ms per token,     1.66 tokens per second)\n",
            "llama_print_timings:       total time =   93572.83 ms\n",
            "Output generated in 94.26 seconds (1.02 tokens/s, 96 tokens, context 26, seed 226568912)\n",
            "2023-12-14 07:44:46 INFO:\u001b[32mLoading mixtral-8x7b-instruct-v0.1.Q2_K.gguf...\u001b[0m\n",
            "2023-12-14 07:44:46 INFO:\u001b[32mllama.cpp weights detected: models/mixtral-8x7b-instruct-v0.1.Q2_K.gguf\u001b[0m\n",
            "llama_model_loader: loaded meta data with 26 key-value pairs and 995 tensors from models/mixtral-8x7b-instruct-v0.1.Q2_K.gguf (version GGUF V3 (latest))\n",
            "llama_model_loader: - tensor    0:                token_embd.weight q2_K     [  4096, 32000,     1,     1 ]\n",
            "llama_model_loader: - tensor    1:          blk.0.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor    2:          blk.0.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor    3:            blk.0.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor    4:          blk.0.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor    5:          blk.0.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor    6:            blk.0.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor    7:          blk.0.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor    8:          blk.0.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor    9:            blk.0.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   10:          blk.0.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   11:          blk.0.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   12:            blk.0.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   13:          blk.0.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   14:          blk.0.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   15:            blk.0.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   16:          blk.0.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   17:          blk.0.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   18:            blk.0.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   19:          blk.0.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   20:          blk.0.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   21:            blk.0.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   22:          blk.0.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   23:          blk.0.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   24:            blk.0.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   25:        blk.0.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor   26:           blk.0.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   27:            blk.0.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   28:              blk.0.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   29:         blk.0.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   30:              blk.0.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   31:              blk.0.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   32:          blk.1.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   33:          blk.1.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   34:            blk.1.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   35:          blk.1.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   36:          blk.1.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   37:            blk.1.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   38:          blk.1.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   39:          blk.1.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   40:            blk.1.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   41:          blk.1.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   42:          blk.1.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   43:            blk.1.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   44:          blk.1.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   45:          blk.1.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   46:        blk.1.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor   47:              blk.1.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   48:         blk.1.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   49:              blk.1.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   50:              blk.1.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   51:            blk.1.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   52:          blk.1.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   53:          blk.1.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   54:            blk.1.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   55:          blk.1.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   56:          blk.1.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   57:            blk.1.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   58:          blk.1.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   59:          blk.1.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   60:            blk.1.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   61:           blk.1.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   62:            blk.1.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   63:          blk.2.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   64:          blk.2.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   65:            blk.2.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   66:          blk.2.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   67:          blk.2.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   68:            blk.2.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   69:          blk.2.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   70:          blk.2.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   71:            blk.2.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   72:          blk.2.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   73:          blk.2.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   74:            blk.2.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   75:          blk.2.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   76:          blk.2.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   77:            blk.2.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   78:          blk.2.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   79:          blk.2.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   80:            blk.2.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   81:          blk.2.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   82:          blk.2.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   83:            blk.2.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   84:          blk.2.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   85:          blk.2.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   86:            blk.2.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   87:        blk.2.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor   88:           blk.2.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   89:            blk.2.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   90:              blk.2.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   91:         blk.2.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   92:              blk.2.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   93:              blk.2.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   94:          blk.3.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   95:          blk.3.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   96:            blk.3.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   97:          blk.3.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   98:          blk.3.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   99:            blk.3.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  100:          blk.3.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  101:        blk.3.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  102:              blk.3.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  103:         blk.3.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  104:              blk.3.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  105:              blk.3.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  106:          blk.3.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  107:            blk.3.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  108:          blk.3.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  109:          blk.3.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  110:            blk.3.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  111:          blk.3.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  112:          blk.3.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  113:            blk.3.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  114:          blk.3.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  115:          blk.3.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  116:            blk.3.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  117:          blk.3.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  118:          blk.3.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  119:            blk.3.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  120:          blk.3.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  121:          blk.3.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  122:            blk.3.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  123:           blk.3.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  124:            blk.3.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  125:          blk.4.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  126:          blk.4.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  127:            blk.4.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  128:          blk.4.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  129:          blk.4.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  130:            blk.4.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  131:          blk.4.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  132:          blk.4.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  133:            blk.4.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  134:          blk.4.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  135:          blk.4.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  136:            blk.4.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  137:          blk.4.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  138:          blk.4.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  139:            blk.4.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  140:          blk.4.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  141:          blk.4.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  142:            blk.4.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  143:          blk.4.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  144:          blk.4.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  145:            blk.4.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  146:          blk.4.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  147:          blk.4.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  148:            blk.4.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  149:        blk.4.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  150:           blk.4.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  151:            blk.4.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  152:              blk.4.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  153:         blk.4.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  154:              blk.4.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  155:              blk.4.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  156:        blk.5.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  157:              blk.5.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  158:         blk.5.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  159:              blk.5.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  160:              blk.5.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  161:          blk.5.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  162:          blk.5.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  163:            blk.5.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  164:          blk.5.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  165:          blk.5.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  166:            blk.5.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  167:          blk.5.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  168:          blk.5.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  169:            blk.5.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  170:          blk.5.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  171:          blk.5.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  172:            blk.5.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  173:          blk.5.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  174:          blk.5.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  175:            blk.5.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  176:          blk.5.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  177:          blk.5.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  178:            blk.5.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  179:          blk.5.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  180:          blk.5.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  181:            blk.5.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  182:          blk.5.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  183:          blk.5.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  184:            blk.5.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  185:           blk.5.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  186:            blk.5.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  187:          blk.6.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  188:          blk.6.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  189:            blk.6.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  190:          blk.6.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  191:          blk.6.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  192:            blk.6.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  193:          blk.6.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  194:          blk.6.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  195:            blk.6.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  196:          blk.6.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  197:          blk.6.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  198:            blk.6.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  199:          blk.6.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  200:          blk.6.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  201:            blk.6.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  202:          blk.6.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  203:          blk.6.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  204:        blk.6.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  205:              blk.6.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  206:         blk.6.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  207:              blk.6.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  208:              blk.6.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  209:            blk.6.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  210:          blk.6.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  211:          blk.6.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  212:            blk.6.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  213:          blk.6.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  214:          blk.6.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  215:            blk.6.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  216:           blk.6.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  217:            blk.6.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  218:          blk.7.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  219:          blk.7.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  220:            blk.7.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  221:          blk.7.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  222:          blk.7.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  223:            blk.7.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  224:          blk.7.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  225:          blk.7.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  226:            blk.7.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  227:          blk.7.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  228:          blk.7.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  229:            blk.7.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  230:          blk.7.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  231:          blk.7.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  232:            blk.7.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  233:          blk.7.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  234:          blk.7.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  235:            blk.7.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  236:          blk.7.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  237:          blk.7.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  238:            blk.7.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  239:          blk.7.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  240:          blk.7.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  241:            blk.7.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  242:        blk.7.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  243:           blk.7.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  244:            blk.7.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  245:              blk.7.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  246:         blk.7.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  247:              blk.7.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  248:              blk.7.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  249:          blk.8.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  250:          blk.8.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  251:            blk.8.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  252:          blk.8.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  253:          blk.8.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  254:            blk.8.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  255:          blk.8.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  256:          blk.8.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  257:            blk.8.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  258:          blk.8.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  259:        blk.8.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  260:              blk.8.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  261:         blk.8.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  262:              blk.8.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  263:              blk.8.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  264:         blk.10.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  265:         blk.10.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  266:           blk.10.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  267:       blk.10.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  268:             blk.10.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  269:        blk.10.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  270:             blk.10.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  271:             blk.10.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  272:          blk.8.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  273:            blk.8.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  274:          blk.8.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  275:          blk.8.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  276:            blk.8.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  277:          blk.8.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  278:          blk.8.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  279:            blk.8.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  280:          blk.8.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  281:          blk.8.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  282:            blk.8.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  283:          blk.8.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  284:          blk.8.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  285:            blk.8.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  286:           blk.8.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  287:            blk.8.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  288:          blk.9.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  289:          blk.9.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  290:            blk.9.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  291:          blk.9.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  292:          blk.9.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  293:            blk.9.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  294:          blk.9.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  295:          blk.9.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  296:            blk.9.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  297:          blk.9.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  298:          blk.9.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  299:            blk.9.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  300:          blk.9.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  301:          blk.9.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  302:            blk.9.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  303:          blk.9.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  304:          blk.9.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  305:            blk.9.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  306:          blk.9.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  307:          blk.9.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  308:            blk.9.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  309:          blk.9.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  310:          blk.9.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  311:            blk.9.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  312:        blk.9.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  313:           blk.9.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  314:            blk.9.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  315:              blk.9.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  316:         blk.9.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  317:              blk.9.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  318:              blk.9.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  319:         blk.10.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  320:         blk.10.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  321:           blk.10.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  322:         blk.10.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  323:         blk.10.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  324:           blk.10.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  325:         blk.10.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  326:         blk.10.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  327:           blk.10.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  328:         blk.10.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  329:         blk.10.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  330:           blk.10.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  331:         blk.10.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  332:         blk.10.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  333:           blk.10.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  334:         blk.10.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  335:         blk.10.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  336:           blk.10.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  337:         blk.10.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  338:         blk.10.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  339:           blk.10.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  340:          blk.10.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  341:           blk.10.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  342:         blk.11.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  343:         blk.11.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  344:           blk.11.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  345:         blk.11.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  346:         blk.11.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  347:           blk.11.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  348:         blk.11.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  349:         blk.11.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  350:           blk.11.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  351:         blk.11.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  352:         blk.11.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  353:           blk.11.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  354:         blk.11.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  355:         blk.11.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  356:           blk.11.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  357:         blk.11.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  358:         blk.11.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  359:           blk.11.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  360:         blk.11.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  361:         blk.11.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  362:       blk.11.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  363:             blk.11.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  364:        blk.11.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  365:             blk.11.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  366:             blk.11.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  367:           blk.11.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  368:         blk.11.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  369:         blk.11.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  370:           blk.11.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  371:          blk.11.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  372:           blk.11.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  373:         blk.12.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  374:         blk.12.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  375:           blk.12.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  376:         blk.12.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  377:         blk.12.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  378:           blk.12.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  379:         blk.12.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  380:         blk.12.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  381:           blk.12.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  382:         blk.12.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  383:         blk.12.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  384:           blk.12.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  385:         blk.12.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  386:         blk.12.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  387:           blk.12.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  388:         blk.12.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  389:         blk.12.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  390:           blk.12.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  391:         blk.12.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  392:         blk.12.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  393:           blk.12.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  394:         blk.12.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  395:         blk.12.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  396:           blk.12.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  397:       blk.12.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  398:          blk.12.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  399:           blk.12.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  400:             blk.12.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  401:        blk.12.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  402:             blk.12.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  403:             blk.12.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  404:         blk.13.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  405:         blk.13.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  406:           blk.13.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  407:         blk.13.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  408:         blk.13.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  409:           blk.13.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  410:         blk.13.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  411:         blk.13.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  412:           blk.13.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  413:         blk.13.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  414:         blk.13.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  415:           blk.13.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  416:         blk.13.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  417:       blk.13.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  418:             blk.13.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  419:        blk.13.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  420:             blk.13.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  421:             blk.13.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  422:         blk.13.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  423:           blk.13.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  424:         blk.13.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  425:         blk.13.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  426:           blk.13.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  427:         blk.13.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  428:         blk.13.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  429:           blk.13.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  430:         blk.13.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  431:         blk.13.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  432:           blk.13.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  433:          blk.13.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  434:           blk.13.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  435:         blk.14.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  436:         blk.14.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  437:           blk.14.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  438:         blk.14.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  439:         blk.14.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  440:           blk.14.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  441:         blk.14.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  442:         blk.14.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  443:           blk.14.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  444:         blk.14.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  445:         blk.14.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  446:           blk.14.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  447:         blk.14.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  448:         blk.14.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  449:           blk.14.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  450:         blk.14.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  451:         blk.14.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  452:           blk.14.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  453:         blk.14.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  454:         blk.14.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  455:           blk.14.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  456:         blk.14.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  457:         blk.14.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  458:           blk.14.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  459:       blk.14.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  460:          blk.14.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  461:           blk.14.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  462:             blk.14.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  463:        blk.14.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  464:             blk.14.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  465:             blk.14.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  466:         blk.15.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  467:         blk.15.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  468:           blk.15.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  469:         blk.15.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  470:         blk.15.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  471:           blk.15.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  472:       blk.15.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  473:             blk.15.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  474:        blk.15.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  475:             blk.15.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  476:             blk.15.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  477:         blk.15.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  478:         blk.15.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  479:           blk.15.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  480:         blk.15.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  481:         blk.15.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  482:           blk.15.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  483:         blk.15.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  484:         blk.15.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  485:           blk.15.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  486:         blk.15.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  487:         blk.15.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  488:           blk.15.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  489:         blk.15.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  490:         blk.15.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  491:           blk.15.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  492:         blk.15.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  493:         blk.15.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  494:           blk.15.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  495:          blk.15.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  496:           blk.15.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  497:         blk.16.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  498:         blk.16.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  499:           blk.16.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  500:         blk.16.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  501:         blk.16.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  502:           blk.16.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  503:         blk.16.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  504:         blk.16.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  505:           blk.16.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  506:         blk.16.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  507:         blk.16.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  508:           blk.16.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  509:         blk.16.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  510:         blk.16.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  511:           blk.16.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  512:         blk.16.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  513:         blk.16.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  514:           blk.16.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  515:         blk.16.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  516:         blk.16.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  517:           blk.16.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  518:         blk.16.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  519:         blk.16.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  520:       blk.16.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  521:             blk.16.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  522:        blk.16.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  523:             blk.16.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  524:             blk.16.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  525:           blk.16.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  526:          blk.16.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  527:           blk.16.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  528:         blk.17.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  529:         blk.17.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  530:           blk.17.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  531:         blk.17.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  532:         blk.17.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  533:           blk.17.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  534:         blk.17.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  535:         blk.17.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  536:           blk.17.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  537:         blk.17.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  538:         blk.17.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  539:           blk.17.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  540:         blk.17.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  541:         blk.17.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  542:           blk.17.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  543:         blk.17.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  544:         blk.17.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  545:           blk.17.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  546:         blk.17.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  547:         blk.17.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  548:           blk.17.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  549:         blk.17.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  550:         blk.17.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  551:           blk.17.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  552:       blk.17.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  553:          blk.17.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  554:           blk.17.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  555:             blk.17.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  556:        blk.17.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  557:             blk.17.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  558:             blk.17.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  559:         blk.18.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  560:         blk.18.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  561:           blk.18.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  562:         blk.18.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  563:         blk.18.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  564:           blk.18.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  565:         blk.18.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  566:         blk.18.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  567:           blk.18.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  568:         blk.18.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  569:         blk.18.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  570:           blk.18.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  571:         blk.18.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  572:         blk.18.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  573:           blk.18.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  574:         blk.18.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  575:       blk.18.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  576:             blk.18.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  577:        blk.18.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  578:             blk.18.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  579:             blk.18.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  580:         blk.18.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  581:           blk.18.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  582:         blk.18.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  583:         blk.18.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  584:           blk.18.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  585:         blk.18.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  586:         blk.18.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  587:           blk.18.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  588:          blk.18.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  589:           blk.18.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  590:         blk.19.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  591:         blk.19.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  592:           blk.19.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  593:         blk.19.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  594:         blk.19.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  595:           blk.19.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  596:         blk.19.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  597:         blk.19.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  598:           blk.19.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  599:         blk.19.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  600:         blk.19.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  601:           blk.19.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  602:         blk.19.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  603:         blk.19.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  604:           blk.19.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  605:         blk.19.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  606:         blk.19.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  607:           blk.19.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  608:         blk.19.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  609:         blk.19.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  610:           blk.19.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  611:         blk.19.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  612:         blk.19.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  613:           blk.19.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  614:       blk.19.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  615:          blk.19.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  616:           blk.19.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  617:             blk.19.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  618:        blk.19.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  619:             blk.19.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  620:             blk.19.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  621:         blk.20.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  622:         blk.20.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  623:           blk.20.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  624:         blk.20.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  625:         blk.20.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  626:           blk.20.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  627:         blk.20.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  628:         blk.20.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  629:           blk.20.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  630:       blk.20.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  631:             blk.20.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  632:        blk.20.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  633:             blk.20.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  634:             blk.20.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  635:         blk.20.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  636:         blk.20.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  637:           blk.20.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  638:         blk.20.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  639:         blk.20.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  640:           blk.20.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  641:         blk.20.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  642:         blk.20.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  643:           blk.20.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  644:         blk.20.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  645:         blk.20.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  646:           blk.20.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  647:         blk.20.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  648:         blk.20.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  649:           blk.20.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  650:          blk.20.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  651:           blk.20.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  652:         blk.21.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  653:         blk.21.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  654:           blk.21.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  655:         blk.21.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  656:         blk.21.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  657:           blk.21.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  658:         blk.21.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  659:         blk.21.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  660:           blk.21.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  661:         blk.21.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  662:         blk.21.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  663:           blk.21.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  664:         blk.21.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  665:         blk.21.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  666:           blk.21.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  667:         blk.21.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  668:         blk.21.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  669:           blk.21.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  670:         blk.21.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  671:         blk.21.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  672:           blk.21.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  673:         blk.21.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  674:         blk.21.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  675:           blk.21.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  676:       blk.21.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  677:          blk.21.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  678:           blk.21.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  679:             blk.21.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  680:        blk.21.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  681:             blk.21.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  682:             blk.21.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  683:         blk.22.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  684:         blk.22.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  685:       blk.22.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  686:             blk.22.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  687:        blk.22.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  688:             blk.22.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  689:             blk.22.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  690:           blk.22.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  691:         blk.22.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  692:         blk.22.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  693:           blk.22.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  694:         blk.22.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  695:         blk.22.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  696:           blk.22.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  697:         blk.22.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  698:         blk.22.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  699:           blk.22.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  700:         blk.22.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  701:         blk.22.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  702:           blk.22.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  703:         blk.22.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  704:         blk.22.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  705:           blk.22.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  706:         blk.22.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  707:         blk.22.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  708:           blk.22.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  709:         blk.22.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  710:         blk.22.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  711:           blk.22.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  712:          blk.22.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  713:           blk.22.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  714:         blk.23.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  715:         blk.23.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  716:           blk.23.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  717:         blk.23.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  718:         blk.23.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  719:           blk.23.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  720:         blk.23.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  721:         blk.23.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  722:           blk.23.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  723:         blk.23.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  724:         blk.23.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  725:           blk.23.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  726:         blk.23.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  727:         blk.23.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  728:           blk.23.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  729:         blk.23.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  730:         blk.23.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  731:           blk.23.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  732:         blk.23.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  733:       blk.23.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  734:             blk.23.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  735:        blk.23.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  736:             blk.23.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  737:             blk.23.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  738:         blk.23.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  739:           blk.23.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  740:         blk.23.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  741:         blk.23.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  742:           blk.23.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  743:          blk.23.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  744:           blk.23.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  745:         blk.24.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  746:         blk.24.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  747:           blk.24.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  748:         blk.24.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  749:         blk.24.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  750:           blk.24.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  751:         blk.24.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  752:         blk.24.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  753:           blk.24.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  754:         blk.24.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  755:         blk.24.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  756:           blk.24.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  757:         blk.24.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  758:         blk.24.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  759:           blk.24.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  760:         blk.24.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  761:         blk.24.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  762:           blk.24.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  763:         blk.24.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  764:         blk.24.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  765:           blk.24.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  766:         blk.24.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  767:         blk.24.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  768:           blk.24.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  769:       blk.24.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  770:          blk.24.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  771:           blk.24.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  772:             blk.24.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  773:        blk.24.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  774:             blk.24.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  775:             blk.24.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  776:         blk.25.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  777:         blk.25.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  778:           blk.25.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  779:         blk.25.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  780:         blk.25.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  781:           blk.25.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  782:         blk.25.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  783:         blk.25.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  784:           blk.25.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  785:         blk.25.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  786:         blk.25.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  787:           blk.25.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  788:       blk.25.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  789:             blk.25.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  790:        blk.25.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  791:             blk.25.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  792:             blk.25.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  793:         blk.25.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  794:         blk.25.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  795:           blk.25.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  796:         blk.25.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  797:         blk.25.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  798:           blk.25.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  799:         blk.25.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  800:         blk.25.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  801:           blk.25.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  802:         blk.25.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  803:         blk.25.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  804:           blk.25.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  805:          blk.25.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  806:           blk.25.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  807:         blk.26.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  808:         blk.26.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  809:           blk.26.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  810:         blk.26.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  811:         blk.26.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  812:           blk.26.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  813:         blk.26.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  814:         blk.26.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  815:           blk.26.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  816:         blk.26.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  817:         blk.26.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  818:           blk.26.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  819:         blk.26.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  820:         blk.26.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  821:           blk.26.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  822:         blk.26.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  823:         blk.26.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  824:           blk.26.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  825:         blk.26.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  826:         blk.26.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  827:           blk.26.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  828:         blk.26.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  829:         blk.26.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  830:           blk.26.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  831:       blk.26.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  832:          blk.26.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  833:           blk.26.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  834:             blk.26.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  835:        blk.26.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  836:             blk.26.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  837:             blk.26.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  838:         blk.27.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  839:         blk.27.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  840:           blk.27.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  841:         blk.27.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  842:         blk.27.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  843:       blk.27.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  844:             blk.27.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  845:        blk.27.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  846:             blk.27.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  847:             blk.27.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  848:           blk.27.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  849:         blk.27.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  850:         blk.27.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  851:           blk.27.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  852:         blk.27.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  853:         blk.27.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  854:           blk.27.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  855:         blk.27.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  856:         blk.27.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  857:           blk.27.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  858:         blk.27.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  859:         blk.27.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  860:           blk.27.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  861:         blk.27.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  862:         blk.27.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  863:           blk.27.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  864:         blk.27.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  865:         blk.27.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  866:           blk.27.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  867:          blk.27.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  868:           blk.27.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  869:         blk.28.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  870:         blk.28.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  871:           blk.28.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  872:         blk.28.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  873:         blk.28.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  874:           blk.28.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  875:         blk.28.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  876:         blk.28.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  877:           blk.28.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  878:         blk.28.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  879:         blk.28.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  880:           blk.28.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  881:         blk.28.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  882:         blk.28.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  883:           blk.28.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  884:         blk.28.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  885:         blk.28.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  886:           blk.28.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  887:         blk.28.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  888:         blk.28.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  889:           blk.28.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  890:         blk.28.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  891:       blk.28.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  892:             blk.28.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  893:        blk.28.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  894:             blk.28.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  895:             blk.28.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  896:         blk.28.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  897:           blk.28.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  898:          blk.28.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  899:           blk.28.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  900:         blk.29.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  901:         blk.29.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  902:           blk.29.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  903:         blk.29.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  904:         blk.29.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  905:           blk.29.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  906:         blk.29.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  907:         blk.29.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  908:           blk.29.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  909:         blk.29.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  910:         blk.29.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  911:           blk.29.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  912:         blk.29.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  913:         blk.29.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  914:           blk.29.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  915:         blk.29.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  916:         blk.29.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  917:           blk.29.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  918:         blk.29.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  919:         blk.29.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  920:           blk.29.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  921:         blk.29.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  922:         blk.29.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  923:           blk.29.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  924:       blk.29.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  925:          blk.29.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  926:           blk.29.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  927:             blk.29.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  928:        blk.29.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  929:             blk.29.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  930:             blk.29.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  931:         blk.30.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  932:         blk.30.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  933:           blk.30.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  934:         blk.30.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  935:         blk.30.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  936:           blk.30.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  937:         blk.30.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  938:         blk.30.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  939:           blk.30.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  940:         blk.30.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  941:         blk.30.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  942:           blk.30.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  943:         blk.30.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  944:         blk.30.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  945:           blk.30.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  946:       blk.30.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  947:             blk.30.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  948:        blk.30.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  949:             blk.30.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  950:             blk.30.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  951:                    output.weight q6_K     [  4096, 32000,     1,     1 ]\n",
            "llama_model_loader: - tensor  952:         blk.30.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  953:         blk.30.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  954:           blk.30.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  955:         blk.30.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  956:         blk.30.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  957:           blk.30.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  958:         blk.30.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  959:         blk.30.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  960:           blk.30.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  961:          blk.30.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  962:           blk.30.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  963:         blk.31.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  964:         blk.31.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  965:           blk.31.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  966:         blk.31.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  967:         blk.31.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  968:           blk.31.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  969:         blk.31.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  970:         blk.31.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  971:           blk.31.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  972:         blk.31.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  973:         blk.31.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  974:           blk.31.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  975:         blk.31.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  976:         blk.31.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  977:           blk.31.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  978:         blk.31.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  979:         blk.31.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  980:           blk.31.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  981:         blk.31.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  982:         blk.31.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  983:           blk.31.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  984:         blk.31.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  985:         blk.31.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  986:           blk.31.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  987:       blk.31.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  988:          blk.31.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  989:           blk.31.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  990:             blk.31.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  991:        blk.31.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  992:             blk.31.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  993:             blk.31.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  994:               output_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
            "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
            "llama_model_loader: - kv   1:                               general.name str              = mistralai_mixtral-8x7b-instruct-v0.1\n",
            "llama_model_loader: - kv   2:                       llama.context_length u32              = 32768\n",
            "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
            "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
            "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336\n",
            "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
            "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
            "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
            "llama_model_loader: - kv   9:                         llama.expert_count u32              = 8\n",
            "llama_model_loader: - kv  10:                    llama.expert_used_count u32              = 2\n",
            "llama_model_loader: - kv  11:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
            "llama_model_loader: - kv  12:                       llama.rope.freq_base f32              = 1000000.000000\n",
            "llama_model_loader: - kv  13:                          general.file_type u32              = 10\n",
            "llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = llama\n",
            "llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
            "llama_model_loader: - kv  16:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
            "llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
            "llama_model_loader: - kv  18:                tokenizer.ggml.bos_token_id u32              = 1\n",
            "llama_model_loader: - kv  19:                tokenizer.ggml.eos_token_id u32              = 2\n",
            "llama_model_loader: - kv  20:            tokenizer.ggml.unknown_token_id u32              = 0\n",
            "llama_model_loader: - kv  21:            tokenizer.ggml.padding_token_id u32              = 0\n",
            "llama_model_loader: - kv  22:               tokenizer.ggml.add_bos_token bool             = true\n",
            "llama_model_loader: - kv  23:               tokenizer.ggml.add_eos_token bool             = false\n",
            "llama_model_loader: - kv  24:                    tokenizer.chat_template str              = {{ bos_token }}{% for message in mess...\n",
            "llama_model_loader: - kv  25:               general.quantization_version u32              = 2\n",
            "llama_model_loader: - type  f32:   65 tensors\n",
            "llama_model_loader: - type  f16:   32 tensors\n",
            "llama_model_loader: - type q8_0:   64 tensors\n",
            "llama_model_loader: - type q2_K:  801 tensors\n",
            "llama_model_loader: - type q3_K:   32 tensors\n",
            "llama_model_loader: - type q6_K:    1 tensors\n",
            "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
            "llm_load_print_meta: format           = GGUF V3 (latest)\n",
            "llm_load_print_meta: arch             = llama\n",
            "llm_load_print_meta: vocab type       = SPM\n",
            "llm_load_print_meta: n_vocab          = 32000\n",
            "llm_load_print_meta: n_merges         = 0\n",
            "llm_load_print_meta: n_ctx_train      = 32768\n",
            "llm_load_print_meta: n_embd           = 4096\n",
            "llm_load_print_meta: n_head           = 32\n",
            "llm_load_print_meta: n_head_kv        = 8\n",
            "llm_load_print_meta: n_layer          = 32\n",
            "llm_load_print_meta: n_rot            = 128\n",
            "llm_load_print_meta: n_gqa            = 4\n",
            "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
            "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
            "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
            "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
            "llm_load_print_meta: n_ff             = 14336\n",
            "llm_load_print_meta: n_expert         = 8\n",
            "llm_load_print_meta: n_expert_used    = 2\n",
            "llm_load_print_meta: rope scaling     = linear\n",
            "llm_load_print_meta: freq_base_train  = 1000000.0\n",
            "llm_load_print_meta: freq_scale_train = 1\n",
            "llm_load_print_meta: n_yarn_orig_ctx  = 32768\n",
            "llm_load_print_meta: rope_finetuned   = unknown\n",
            "llm_load_print_meta: model type       = 7B\n",
            "llm_load_print_meta: model ftype      = mostly Q2_K\n",
            "llm_load_print_meta: model params     = 46.70 B\n",
            "llm_load_print_meta: model size       = 14.57 GiB (2.68 BPW) \n",
            "llm_load_print_meta: general.name     = mistralai_mixtral-8x7b-instruct-v0.1\n",
            "llm_load_print_meta: BOS token        = 1 '<s>'\n",
            "llm_load_print_meta: EOS token        = 2 '</s>'\n",
            "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
            "llm_load_print_meta: PAD token        = 0 '<unk>'\n",
            "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
            "llm_load_tensors: ggml ctx size =    0.39 MiB\n",
            "llm_load_tensors: using CUDA for GPU acceleration\n",
            "llm_load_tensors: mem required  = 3376.00 MiB\n",
            "llm_load_tensors: offloading 25 repeating layers to GPU\n",
            "llm_load_tensors: offloaded 25/33 layers to GPU\n",
            "llm_load_tensors: VRAM used: 11542.97 MiB\n",
            "....................................................................................................\n",
            "llama_new_context_with_model: n_ctx      = 2048\n",
            "llama_new_context_with_model: freq_base  = 1000000.0\n",
            "llama_new_context_with_model: freq_scale = 1\n",
            "llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB\n",
            "llama_build_graph: non-view tensors processed: 1124/1124\n",
            "llama_new_context_with_model: compute buffer total size = 187.35 MiB\n",
            "llama_new_context_with_model: VRAM scratch buffer: 184.04 MiB\n",
            "llama_new_context_with_model: total VRAM used: 11727.00 MiB (model: 11542.97 MiB, context: 184.04 MiB)\n",
            "AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
            "2023-12-14 07:46:30 INFO:\u001b[32mLOADER: llama.cpp\u001b[0m\n",
            "2023-12-14 07:46:30 INFO:\u001b[32mTRUNCATION LENGTH: 2048\u001b[0m\n",
            "2023-12-14 07:46:30 INFO:\u001b[32mINSTRUCTION TEMPLATE: Alpaca\u001b[0m\n",
            "2023-12-14 07:46:30 INFO:\u001b[32mLoaded the model in 104.43 seconds.\u001b[0m\n",
            "\n",
            "llama_print_timings:        load time =   41908.42 ms\n",
            "llama_print_timings:      sample time =      77.84 ms /   143 runs   (    0.54 ms per token,  1837.10 tokens per second)\n",
            "llama_print_timings: prompt eval time =   41908.08 ms /   140 tokens (  299.34 ms per token,     3.34 tokens per second)\n",
            "llama_print_timings:        eval time =   40886.17 ms /   142 runs   (  287.93 ms per token,     3.47 tokens per second)\n",
            "llama_print_timings:       total time =   83750.51 ms\n",
            "Output generated in 84.53 seconds (1.68 tokens/s, 142 tokens, context 141, seed 139966707)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Q4_K_M, half on GPU, very slow generation\n",
        "!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF/resolve/main/mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf?download=true -d /content/text-generation-webui/models/ -o mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf\n",
        "\n",
        "%cd /content/text-generation-webui\n",
        "!python server.py --share --n-gpu-layers 12 --n_ctx 2048 --model mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf"
      ],
      "metadata": {
        "id": "OLuj0BHLG7G-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Q2_K, CPU only, very slow generation (0.03T/s)\n",
        "!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF/resolve/main/mixtral-8x7b-instruct-v0.1.Q2_K.gguf?download=true -d /content/text-generation-webui/models/ -o mixtral-8x7b-instruct-v0.1.Q2_K.gguf\n",
        "\n",
        "%cd /content/text-generation-webui\n",
        "!python server.py --share --model mixtral-8x7b-instruct-v0.1.Q2_K.gguf"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T8Zy4LCSI7B2",
        "outputId": "66cda241-0c32-4be8-b0df-76df2c9c612e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " *** Download Progress Summary as of Thu Dec 14 07:09:02 2023 *** \n",
            "=\n",
            "[#494504 11GiB/14GiB(77%) CN:16 DL:183MiB ETA:18s]\n",
            "FILE: /content/text-generation-webui/models//mixtral-8x7b-instruct-v0.1.Q2_K.gguf\n",
            "-\n",
            "\n",
            "\u001b[0m\n",
            "Download Results:\n",
            "gid   |stat|avg speed  |path/URI\n",
            "======+====+===========+=======================================================\n",
            "494504|\u001b[1;32mOK\u001b[0m  |   193MiB/s|/content/text-generation-webui/models//mixtral-8x7b-instruct-v0.1.Q2_K.gguf\n",
            "\n",
            "Status Legend:\n",
            "(OK):download completed.\n",
            "/content/text-generation-webui\n",
            "2023-12-14 07:09:29 INFO:\u001b[32mLoading mixtral-8x7b-instruct-v0.1.Q2_K.gguf...\u001b[0m\n",
            "2023-12-14 07:09:29 INFO:\u001b[32mllama.cpp weights detected: models/mixtral-8x7b-instruct-v0.1.Q2_K.gguf\u001b[0m\n",
            "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
            "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
            "ggml_init_cublas: found 1 CUDA devices:\n",
            "  Device 0: Tesla T4, compute capability 7.5\n",
            "llama_model_loader: loaded meta data with 26 key-value pairs and 995 tensors from models/mixtral-8x7b-instruct-v0.1.Q2_K.gguf (version GGUF V3 (latest))\n",
            "llama_model_loader: - tensor    0:                token_embd.weight q2_K     [  4096, 32000,     1,     1 ]\n",
            "llama_model_loader: - tensor    1:          blk.0.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor    2:          blk.0.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor    3:            blk.0.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor    4:          blk.0.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor    5:          blk.0.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor    6:            blk.0.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor    7:          blk.0.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor    8:          blk.0.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor    9:            blk.0.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   10:          blk.0.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   11:          blk.0.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   12:            blk.0.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   13:          blk.0.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   14:          blk.0.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   15:            blk.0.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   16:          blk.0.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   17:          blk.0.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   18:            blk.0.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   19:          blk.0.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   20:          blk.0.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   21:            blk.0.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   22:          blk.0.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   23:          blk.0.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   24:            blk.0.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   25:        blk.0.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor   26:           blk.0.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   27:            blk.0.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   28:              blk.0.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   29:         blk.0.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   30:              blk.0.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   31:              blk.0.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   32:          blk.1.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   33:          blk.1.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   34:            blk.1.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   35:          blk.1.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   36:          blk.1.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   37:            blk.1.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   38:          blk.1.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   39:          blk.1.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   40:            blk.1.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   41:          blk.1.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   42:          blk.1.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   43:            blk.1.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   44:          blk.1.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   45:          blk.1.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   46:        blk.1.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor   47:              blk.1.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   48:         blk.1.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   49:              blk.1.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   50:              blk.1.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   51:            blk.1.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   52:          blk.1.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   53:          blk.1.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   54:            blk.1.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   55:          blk.1.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   56:          blk.1.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   57:            blk.1.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   58:          blk.1.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   59:          blk.1.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   60:            blk.1.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   61:           blk.1.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   62:            blk.1.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   63:          blk.2.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   64:          blk.2.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   65:            blk.2.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   66:          blk.2.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   67:          blk.2.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   68:            blk.2.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   69:          blk.2.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   70:          blk.2.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   71:            blk.2.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   72:          blk.2.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   73:          blk.2.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   74:            blk.2.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   75:          blk.2.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   76:          blk.2.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   77:            blk.2.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   78:          blk.2.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   79:          blk.2.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   80:            blk.2.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   81:          blk.2.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   82:          blk.2.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   83:            blk.2.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   84:          blk.2.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   85:          blk.2.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   86:            blk.2.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   87:        blk.2.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor   88:           blk.2.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   89:            blk.2.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor   90:              blk.2.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   91:         blk.2.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   92:              blk.2.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   93:              blk.2.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor   94:          blk.3.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   95:          blk.3.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   96:            blk.3.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   97:          blk.3.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor   98:          blk.3.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor   99:            blk.3.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  100:          blk.3.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  101:        blk.3.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  102:              blk.3.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  103:         blk.3.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  104:              blk.3.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  105:              blk.3.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  106:          blk.3.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  107:            blk.3.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  108:          blk.3.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  109:          blk.3.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  110:            blk.3.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  111:          blk.3.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  112:          blk.3.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  113:            blk.3.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  114:          blk.3.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  115:          blk.3.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  116:            blk.3.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  117:          blk.3.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  118:          blk.3.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  119:            blk.3.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  120:          blk.3.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  121:          blk.3.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  122:            blk.3.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  123:           blk.3.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  124:            blk.3.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  125:          blk.4.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  126:          blk.4.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  127:            blk.4.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  128:          blk.4.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  129:          blk.4.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  130:            blk.4.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  131:          blk.4.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  132:          blk.4.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  133:            blk.4.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  134:          blk.4.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  135:          blk.4.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  136:            blk.4.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  137:          blk.4.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  138:          blk.4.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  139:            blk.4.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  140:          blk.4.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  141:          blk.4.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  142:            blk.4.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  143:          blk.4.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  144:          blk.4.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  145:            blk.4.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  146:          blk.4.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  147:          blk.4.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  148:            blk.4.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  149:        blk.4.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  150:           blk.4.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  151:            blk.4.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  152:              blk.4.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  153:         blk.4.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  154:              blk.4.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  155:              blk.4.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  156:        blk.5.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  157:              blk.5.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  158:         blk.5.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  159:              blk.5.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  160:              blk.5.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  161:          blk.5.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  162:          blk.5.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  163:            blk.5.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  164:          blk.5.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  165:          blk.5.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  166:            blk.5.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  167:          blk.5.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  168:          blk.5.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  169:            blk.5.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  170:          blk.5.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  171:          blk.5.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  172:            blk.5.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  173:          blk.5.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  174:          blk.5.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  175:            blk.5.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  176:          blk.5.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  177:          blk.5.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  178:            blk.5.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  179:          blk.5.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  180:          blk.5.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  181:            blk.5.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  182:          blk.5.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  183:          blk.5.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  184:            blk.5.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  185:           blk.5.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  186:            blk.5.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  187:          blk.6.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  188:          blk.6.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  189:            blk.6.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  190:          blk.6.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  191:          blk.6.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  192:            blk.6.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  193:          blk.6.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  194:          blk.6.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  195:            blk.6.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  196:          blk.6.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  197:          blk.6.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  198:            blk.6.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  199:          blk.6.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  200:          blk.6.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  201:            blk.6.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  202:          blk.6.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  203:          blk.6.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  204:        blk.6.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  205:              blk.6.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  206:         blk.6.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  207:              blk.6.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  208:              blk.6.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  209:            blk.6.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  210:          blk.6.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  211:          blk.6.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  212:            blk.6.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  213:          blk.6.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  214:          blk.6.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  215:            blk.6.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  216:           blk.6.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  217:            blk.6.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  218:          blk.7.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  219:          blk.7.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  220:            blk.7.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  221:          blk.7.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  222:          blk.7.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  223:            blk.7.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  224:          blk.7.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  225:          blk.7.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  226:            blk.7.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  227:          blk.7.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  228:          blk.7.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  229:            blk.7.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  230:          blk.7.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  231:          blk.7.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  232:            blk.7.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  233:          blk.7.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  234:          blk.7.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  235:            blk.7.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  236:          blk.7.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  237:          blk.7.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  238:            blk.7.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  239:          blk.7.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  240:          blk.7.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  241:            blk.7.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  242:        blk.7.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  243:           blk.7.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  244:            blk.7.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  245:              blk.7.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  246:         blk.7.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  247:              blk.7.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  248:              blk.7.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  249:          blk.8.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  250:          blk.8.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  251:            blk.8.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  252:          blk.8.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  253:          blk.8.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  254:            blk.8.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  255:          blk.8.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  256:          blk.8.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  257:            blk.8.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  258:          blk.8.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  259:        blk.8.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  260:              blk.8.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  261:         blk.8.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  262:              blk.8.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  263:              blk.8.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  264:         blk.10.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  265:         blk.10.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  266:           blk.10.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  267:       blk.10.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  268:             blk.10.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  269:        blk.10.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  270:             blk.10.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  271:             blk.10.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  272:          blk.8.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  273:            blk.8.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  274:          blk.8.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  275:          blk.8.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  276:            blk.8.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  277:          blk.8.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  278:          blk.8.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  279:            blk.8.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  280:          blk.8.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  281:          blk.8.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  282:            blk.8.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  283:          blk.8.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  284:          blk.8.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  285:            blk.8.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  286:           blk.8.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  287:            blk.8.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  288:          blk.9.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  289:          blk.9.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  290:            blk.9.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  291:          blk.9.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  292:          blk.9.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  293:            blk.9.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  294:          blk.9.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  295:          blk.9.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  296:            blk.9.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  297:          blk.9.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  298:          blk.9.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  299:            blk.9.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  300:          blk.9.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  301:          blk.9.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  302:            blk.9.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  303:          blk.9.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  304:          blk.9.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  305:            blk.9.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  306:          blk.9.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  307:          blk.9.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  308:            blk.9.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  309:          blk.9.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  310:          blk.9.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  311:            blk.9.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  312:        blk.9.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  313:           blk.9.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  314:            blk.9.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  315:              blk.9.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  316:         blk.9.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  317:              blk.9.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  318:              blk.9.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  319:         blk.10.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  320:         blk.10.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  321:           blk.10.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  322:         blk.10.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  323:         blk.10.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  324:           blk.10.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  325:         blk.10.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  326:         blk.10.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  327:           blk.10.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  328:         blk.10.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  329:         blk.10.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  330:           blk.10.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  331:         blk.10.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  332:         blk.10.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  333:           blk.10.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  334:         blk.10.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  335:         blk.10.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  336:           blk.10.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  337:         blk.10.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  338:         blk.10.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  339:           blk.10.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  340:          blk.10.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  341:           blk.10.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  342:         blk.11.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  343:         blk.11.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  344:           blk.11.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  345:         blk.11.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  346:         blk.11.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  347:           blk.11.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  348:         blk.11.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  349:         blk.11.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  350:           blk.11.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  351:         blk.11.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  352:         blk.11.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  353:           blk.11.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  354:         blk.11.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  355:         blk.11.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  356:           blk.11.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  357:         blk.11.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  358:         blk.11.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  359:           blk.11.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  360:         blk.11.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  361:         blk.11.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  362:       blk.11.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  363:             blk.11.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  364:        blk.11.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  365:             blk.11.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  366:             blk.11.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  367:           blk.11.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  368:         blk.11.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  369:         blk.11.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  370:           blk.11.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  371:          blk.11.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  372:           blk.11.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  373:         blk.12.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  374:         blk.12.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  375:           blk.12.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  376:         blk.12.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  377:         blk.12.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  378:           blk.12.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  379:         blk.12.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  380:         blk.12.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  381:           blk.12.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  382:         blk.12.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  383:         blk.12.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  384:           blk.12.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  385:         blk.12.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  386:         blk.12.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  387:           blk.12.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  388:         blk.12.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  389:         blk.12.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  390:           blk.12.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  391:         blk.12.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  392:         blk.12.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  393:           blk.12.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  394:         blk.12.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  395:         blk.12.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  396:           blk.12.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  397:       blk.12.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  398:          blk.12.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  399:           blk.12.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  400:             blk.12.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  401:        blk.12.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  402:             blk.12.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  403:             blk.12.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  404:         blk.13.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  405:         blk.13.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  406:           blk.13.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  407:         blk.13.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  408:         blk.13.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  409:           blk.13.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  410:         blk.13.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  411:         blk.13.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  412:           blk.13.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  413:         blk.13.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  414:         blk.13.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  415:           blk.13.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  416:         blk.13.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  417:       blk.13.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  418:             blk.13.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  419:        blk.13.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  420:             blk.13.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  421:             blk.13.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  422:         blk.13.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  423:           blk.13.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  424:         blk.13.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  425:         blk.13.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  426:           blk.13.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  427:         blk.13.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  428:         blk.13.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  429:           blk.13.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  430:         blk.13.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  431:         blk.13.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  432:           blk.13.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  433:          blk.13.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  434:           blk.13.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  435:         blk.14.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  436:         blk.14.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  437:           blk.14.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  438:         blk.14.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  439:         blk.14.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  440:           blk.14.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  441:         blk.14.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  442:         blk.14.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  443:           blk.14.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  444:         blk.14.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  445:         blk.14.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  446:           blk.14.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  447:         blk.14.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  448:         blk.14.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  449:           blk.14.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  450:         blk.14.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  451:         blk.14.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  452:           blk.14.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  453:         blk.14.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  454:         blk.14.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  455:           blk.14.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  456:         blk.14.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  457:         blk.14.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  458:           blk.14.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  459:       blk.14.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  460:          blk.14.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  461:           blk.14.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  462:             blk.14.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  463:        blk.14.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  464:             blk.14.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  465:             blk.14.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  466:         blk.15.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  467:         blk.15.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  468:           blk.15.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  469:         blk.15.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  470:         blk.15.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  471:           blk.15.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  472:       blk.15.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  473:             blk.15.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  474:        blk.15.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  475:             blk.15.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  476:             blk.15.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  477:         blk.15.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  478:         blk.15.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  479:           blk.15.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  480:         blk.15.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  481:         blk.15.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  482:           blk.15.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  483:         blk.15.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  484:         blk.15.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  485:           blk.15.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  486:         blk.15.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  487:         blk.15.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  488:           blk.15.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  489:         blk.15.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  490:         blk.15.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  491:           blk.15.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  492:         blk.15.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  493:         blk.15.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  494:           blk.15.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  495:          blk.15.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  496:           blk.15.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  497:         blk.16.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  498:         blk.16.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  499:           blk.16.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  500:         blk.16.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  501:         blk.16.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  502:           blk.16.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  503:         blk.16.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  504:         blk.16.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  505:           blk.16.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  506:         blk.16.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  507:         blk.16.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  508:           blk.16.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  509:         blk.16.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  510:         blk.16.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  511:           blk.16.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  512:         blk.16.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  513:         blk.16.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  514:           blk.16.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  515:         blk.16.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  516:         blk.16.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  517:           blk.16.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  518:         blk.16.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  519:         blk.16.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  520:       blk.16.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  521:             blk.16.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  522:        blk.16.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  523:             blk.16.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  524:             blk.16.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  525:           blk.16.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  526:          blk.16.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  527:           blk.16.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  528:         blk.17.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  529:         blk.17.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  530:           blk.17.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  531:         blk.17.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  532:         blk.17.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  533:           blk.17.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  534:         blk.17.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  535:         blk.17.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  536:           blk.17.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  537:         blk.17.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  538:         blk.17.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  539:           blk.17.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  540:         blk.17.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  541:         blk.17.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  542:           blk.17.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  543:         blk.17.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  544:         blk.17.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  545:           blk.17.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  546:         blk.17.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  547:         blk.17.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  548:           blk.17.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  549:         blk.17.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  550:         blk.17.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  551:           blk.17.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  552:       blk.17.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  553:          blk.17.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  554:           blk.17.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  555:             blk.17.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  556:        blk.17.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  557:             blk.17.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  558:             blk.17.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  559:         blk.18.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  560:         blk.18.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  561:           blk.18.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  562:         blk.18.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  563:         blk.18.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  564:           blk.18.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  565:         blk.18.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  566:         blk.18.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  567:           blk.18.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  568:         blk.18.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  569:         blk.18.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  570:           blk.18.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  571:         blk.18.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  572:         blk.18.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  573:           blk.18.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  574:         blk.18.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  575:       blk.18.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  576:             blk.18.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  577:        blk.18.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  578:             blk.18.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  579:             blk.18.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  580:         blk.18.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  581:           blk.18.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  582:         blk.18.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  583:         blk.18.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  584:           blk.18.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  585:         blk.18.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  586:         blk.18.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  587:           blk.18.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  588:          blk.18.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  589:           blk.18.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  590:         blk.19.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  591:         blk.19.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  592:           blk.19.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  593:         blk.19.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  594:         blk.19.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  595:           blk.19.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  596:         blk.19.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  597:         blk.19.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  598:           blk.19.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  599:         blk.19.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  600:         blk.19.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  601:           blk.19.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  602:         blk.19.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  603:         blk.19.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  604:           blk.19.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  605:         blk.19.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  606:         blk.19.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  607:           blk.19.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  608:         blk.19.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  609:         blk.19.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  610:           blk.19.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  611:         blk.19.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  612:         blk.19.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  613:           blk.19.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  614:       blk.19.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  615:          blk.19.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  616:           blk.19.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  617:             blk.19.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  618:        blk.19.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  619:             blk.19.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  620:             blk.19.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  621:         blk.20.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  622:         blk.20.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  623:           blk.20.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  624:         blk.20.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  625:         blk.20.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  626:           blk.20.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  627:         blk.20.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  628:         blk.20.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  629:           blk.20.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  630:       blk.20.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  631:             blk.20.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  632:        blk.20.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  633:             blk.20.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  634:             blk.20.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  635:         blk.20.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  636:         blk.20.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  637:           blk.20.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  638:         blk.20.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  639:         blk.20.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  640:           blk.20.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  641:         blk.20.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  642:         blk.20.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  643:           blk.20.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  644:         blk.20.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  645:         blk.20.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  646:           blk.20.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  647:         blk.20.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  648:         blk.20.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  649:           blk.20.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  650:          blk.20.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  651:           blk.20.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  652:         blk.21.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  653:         blk.21.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  654:           blk.21.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  655:         blk.21.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  656:         blk.21.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  657:           blk.21.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  658:         blk.21.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  659:         blk.21.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  660:           blk.21.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  661:         blk.21.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  662:         blk.21.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  663:           blk.21.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  664:         blk.21.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  665:         blk.21.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  666:           blk.21.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  667:         blk.21.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  668:         blk.21.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  669:           blk.21.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  670:         blk.21.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  671:         blk.21.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  672:           blk.21.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  673:         blk.21.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  674:         blk.21.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  675:           blk.21.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  676:       blk.21.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  677:          blk.21.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  678:           blk.21.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  679:             blk.21.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  680:        blk.21.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  681:             blk.21.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  682:             blk.21.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  683:         blk.22.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  684:         blk.22.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  685:       blk.22.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  686:             blk.22.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  687:        blk.22.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  688:             blk.22.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  689:             blk.22.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  690:           blk.22.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  691:         blk.22.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  692:         blk.22.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  693:           blk.22.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  694:         blk.22.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  695:         blk.22.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  696:           blk.22.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  697:         blk.22.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  698:         blk.22.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  699:           blk.22.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  700:         blk.22.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  701:         blk.22.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  702:           blk.22.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  703:         blk.22.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  704:         blk.22.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  705:           blk.22.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  706:         blk.22.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  707:         blk.22.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  708:           blk.22.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  709:         blk.22.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  710:         blk.22.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  711:           blk.22.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  712:          blk.22.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  713:           blk.22.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  714:         blk.23.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  715:         blk.23.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  716:           blk.23.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  717:         blk.23.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  718:         blk.23.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  719:           blk.23.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  720:         blk.23.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  721:         blk.23.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  722:           blk.23.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  723:         blk.23.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  724:         blk.23.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  725:           blk.23.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  726:         blk.23.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  727:         blk.23.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  728:           blk.23.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  729:         blk.23.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  730:         blk.23.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  731:           blk.23.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  732:         blk.23.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  733:       blk.23.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  734:             blk.23.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  735:        blk.23.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  736:             blk.23.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  737:             blk.23.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  738:         blk.23.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  739:           blk.23.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  740:         blk.23.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  741:         blk.23.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  742:           blk.23.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  743:          blk.23.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  744:           blk.23.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  745:         blk.24.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  746:         blk.24.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  747:           blk.24.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  748:         blk.24.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  749:         blk.24.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  750:           blk.24.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  751:         blk.24.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  752:         blk.24.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  753:           blk.24.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  754:         blk.24.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  755:         blk.24.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  756:           blk.24.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  757:         blk.24.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  758:         blk.24.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  759:           blk.24.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  760:         blk.24.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  761:         blk.24.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  762:           blk.24.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  763:         blk.24.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  764:         blk.24.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  765:           blk.24.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  766:         blk.24.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  767:         blk.24.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  768:           blk.24.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  769:       blk.24.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  770:          blk.24.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  771:           blk.24.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  772:             blk.24.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  773:        blk.24.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  774:             blk.24.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  775:             blk.24.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  776:         blk.25.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  777:         blk.25.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  778:           blk.25.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  779:         blk.25.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  780:         blk.25.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  781:           blk.25.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  782:         blk.25.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  783:         blk.25.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  784:           blk.25.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  785:         blk.25.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  786:         blk.25.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  787:           blk.25.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  788:       blk.25.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  789:             blk.25.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  790:        blk.25.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  791:             blk.25.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  792:             blk.25.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  793:         blk.25.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  794:         blk.25.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  795:           blk.25.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  796:         blk.25.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  797:         blk.25.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  798:           blk.25.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  799:         blk.25.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  800:         blk.25.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  801:           blk.25.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  802:         blk.25.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  803:         blk.25.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  804:           blk.25.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  805:          blk.25.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  806:           blk.25.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  807:         blk.26.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  808:         blk.26.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  809:           blk.26.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  810:         blk.26.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  811:         blk.26.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  812:           blk.26.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  813:         blk.26.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  814:         blk.26.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  815:           blk.26.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  816:         blk.26.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  817:         blk.26.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  818:           blk.26.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  819:         blk.26.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  820:         blk.26.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  821:           blk.26.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  822:         blk.26.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  823:         blk.26.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  824:           blk.26.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  825:         blk.26.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  826:         blk.26.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  827:           blk.26.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  828:         blk.26.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  829:         blk.26.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  830:           blk.26.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  831:       blk.26.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  832:          blk.26.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  833:           blk.26.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  834:             blk.26.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  835:        blk.26.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  836:             blk.26.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  837:             blk.26.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  838:         blk.27.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  839:         blk.27.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  840:           blk.27.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  841:         blk.27.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  842:         blk.27.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  843:       blk.27.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  844:             blk.27.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  845:        blk.27.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  846:             blk.27.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  847:             blk.27.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  848:           blk.27.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  849:         blk.27.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  850:         blk.27.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  851:           blk.27.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  852:         blk.27.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  853:         blk.27.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  854:           blk.27.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  855:         blk.27.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  856:         blk.27.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  857:           blk.27.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  858:         blk.27.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  859:         blk.27.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  860:           blk.27.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  861:         blk.27.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  862:         blk.27.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  863:           blk.27.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  864:         blk.27.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  865:         blk.27.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  866:           blk.27.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  867:          blk.27.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  868:           blk.27.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  869:         blk.28.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  870:         blk.28.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  871:           blk.28.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  872:         blk.28.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  873:         blk.28.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  874:           blk.28.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  875:         blk.28.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  876:         blk.28.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  877:           blk.28.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  878:         blk.28.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  879:         blk.28.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  880:           blk.28.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  881:         blk.28.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  882:         blk.28.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  883:           blk.28.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  884:         blk.28.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  885:         blk.28.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  886:           blk.28.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  887:         blk.28.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  888:         blk.28.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  889:           blk.28.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  890:         blk.28.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  891:       blk.28.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  892:             blk.28.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  893:        blk.28.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  894:             blk.28.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  895:             blk.28.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  896:         blk.28.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  897:           blk.28.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  898:          blk.28.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  899:           blk.28.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  900:         blk.29.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  901:         blk.29.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  902:           blk.29.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  903:         blk.29.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  904:         blk.29.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  905:           blk.29.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  906:         blk.29.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  907:         blk.29.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  908:           blk.29.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  909:         blk.29.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  910:         blk.29.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  911:           blk.29.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  912:         blk.29.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  913:         blk.29.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  914:           blk.29.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  915:         blk.29.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  916:         blk.29.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  917:           blk.29.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  918:         blk.29.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  919:         blk.29.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  920:           blk.29.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  921:         blk.29.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  922:         blk.29.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  923:           blk.29.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  924:       blk.29.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  925:          blk.29.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  926:           blk.29.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  927:             blk.29.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  928:        blk.29.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  929:             blk.29.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  930:             blk.29.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  931:         blk.30.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  932:         blk.30.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  933:           blk.30.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  934:         blk.30.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  935:         blk.30.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  936:           blk.30.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  937:         blk.30.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  938:         blk.30.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  939:           blk.30.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  940:         blk.30.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  941:         blk.30.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  942:           blk.30.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  943:         blk.30.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  944:         blk.30.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  945:           blk.30.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  946:       blk.30.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  947:             blk.30.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  948:        blk.30.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  949:             blk.30.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  950:             blk.30.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  951:                    output.weight q6_K     [  4096, 32000,     1,     1 ]\n",
            "llama_model_loader: - tensor  952:         blk.30.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  953:         blk.30.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  954:           blk.30.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  955:         blk.30.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  956:         blk.30.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  957:           blk.30.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  958:         blk.30.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  959:         blk.30.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  960:           blk.30.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  961:          blk.30.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  962:           blk.30.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  963:         blk.31.ffn_gate.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  964:         blk.31.ffn_down.0.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  965:           blk.31.ffn_up.0.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  966:         blk.31.ffn_gate.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  967:         blk.31.ffn_down.1.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  968:           blk.31.ffn_up.1.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  969:         blk.31.ffn_gate.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  970:         blk.31.ffn_down.2.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  971:           blk.31.ffn_up.2.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  972:         blk.31.ffn_gate.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  973:         blk.31.ffn_down.3.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  974:           blk.31.ffn_up.3.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  975:         blk.31.ffn_gate.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  976:         blk.31.ffn_down.4.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  977:           blk.31.ffn_up.4.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  978:         blk.31.ffn_gate.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  979:         blk.31.ffn_down.5.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  980:           blk.31.ffn_up.5.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  981:         blk.31.ffn_gate.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  982:         blk.31.ffn_down.6.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  983:           blk.31.ffn_up.6.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  984:         blk.31.ffn_gate.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  985:         blk.31.ffn_down.7.weight q2_K     [ 14336,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  986:           blk.31.ffn_up.7.weight q2_K     [  4096, 14336,     1,     1 ]\n",
            "llama_model_loader: - tensor  987:       blk.31.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]\n",
            "llama_model_loader: - tensor  988:          blk.31.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  989:           blk.31.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: - tensor  990:             blk.31.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  991:        blk.31.attn_output.weight q3_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  992:             blk.31.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]\n",
            "llama_model_loader: - tensor  993:             blk.31.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]\n",
            "llama_model_loader: - tensor  994:               output_norm.weight f32      [  4096,     1,     1,     1 ]\n",
            "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
            "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
            "llama_model_loader: - kv   1:                               general.name str              = mistralai_mixtral-8x7b-instruct-v0.1\n",
            "llama_model_loader: - kv   2:                       llama.context_length u32              = 32768\n",
            "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
            "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
            "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336\n",
            "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
            "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
            "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
            "llama_model_loader: - kv   9:                         llama.expert_count u32              = 8\n",
            "llama_model_loader: - kv  10:                    llama.expert_used_count u32              = 2\n",
            "llama_model_loader: - kv  11:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
            "llama_model_loader: - kv  12:                       llama.rope.freq_base f32              = 1000000.000000\n",
            "llama_model_loader: - kv  13:                          general.file_type u32              = 10\n",
            "llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = llama\n",
            "llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
            "llama_model_loader: - kv  16:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
            "llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
            "llama_model_loader: - kv  18:                tokenizer.ggml.bos_token_id u32              = 1\n",
            "llama_model_loader: - kv  19:                tokenizer.ggml.eos_token_id u32              = 2\n",
            "llama_model_loader: - kv  20:            tokenizer.ggml.unknown_token_id u32              = 0\n",
            "llama_model_loader: - kv  21:            tokenizer.ggml.padding_token_id u32              = 0\n",
            "llama_model_loader: - kv  22:               tokenizer.ggml.add_bos_token bool             = true\n",
            "llama_model_loader: - kv  23:               tokenizer.ggml.add_eos_token bool             = false\n",
            "llama_model_loader: - kv  24:                    tokenizer.chat_template str              = {{ bos_token }}{% for message in mess...\n",
            "llama_model_loader: - kv  25:               general.quantization_version u32              = 2\n",
            "llama_model_loader: - type  f32:   65 tensors\n",
            "llama_model_loader: - type  f16:   32 tensors\n",
            "llama_model_loader: - type q8_0:   64 tensors\n",
            "llama_model_loader: - type q2_K:  801 tensors\n",
            "llama_model_loader: - type q3_K:   32 tensors\n",
            "llama_model_loader: - type q6_K:    1 tensors\n",
            "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
            "llm_load_print_meta: format           = GGUF V3 (latest)\n",
            "llm_load_print_meta: arch             = llama\n",
            "llm_load_print_meta: vocab type       = SPM\n",
            "llm_load_print_meta: n_vocab          = 32000\n",
            "llm_load_print_meta: n_merges         = 0\n",
            "llm_load_print_meta: n_ctx_train      = 32768\n",
            "llm_load_print_meta: n_embd           = 4096\n",
            "llm_load_print_meta: n_head           = 32\n",
            "llm_load_print_meta: n_head_kv        = 8\n",
            "llm_load_print_meta: n_layer          = 32\n",
            "llm_load_print_meta: n_rot            = 128\n",
            "llm_load_print_meta: n_gqa            = 4\n",
            "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
            "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
            "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
            "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
            "llm_load_print_meta: n_ff             = 14336\n",
            "llm_load_print_meta: n_expert         = 8\n",
            "llm_load_print_meta: n_expert_used    = 2\n",
            "llm_load_print_meta: rope scaling     = linear\n",
            "llm_load_print_meta: freq_base_train  = 1000000.0\n",
            "llm_load_print_meta: freq_scale_train = 1\n",
            "llm_load_print_meta: n_yarn_orig_ctx  = 32768\n",
            "llm_load_print_meta: rope_finetuned   = unknown\n",
            "llm_load_print_meta: model type       = 7B\n",
            "llm_load_print_meta: model ftype      = mostly Q2_K\n",
            "llm_load_print_meta: model params     = 46.70 B\n",
            "llm_load_print_meta: model size       = 14.57 GiB (2.68 BPW) \n",
            "llm_load_print_meta: general.name     = mistralai_mixtral-8x7b-instruct-v0.1\n",
            "llm_load_print_meta: BOS token        = 1 '<s>'\n",
            "llm_load_print_meta: EOS token        = 2 '</s>'\n",
            "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
            "llm_load_print_meta: PAD token        = 0 '<unk>'\n",
            "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
            "llm_load_tensors: ggml ctx size =    0.39 MiB\n",
            "llm_load_tensors: using CUDA for GPU acceleration\n",
            "llm_load_tensors: mem required  = 14918.97 MiB\n",
            "llm_load_tensors: offloading 0 repeating layers to GPU\n",
            "llm_load_tensors: offloaded 0/33 layers to GPU\n",
            "llm_load_tensors: VRAM used: 0.00 MiB\n",
            "....................................................................................................\n",
            "llama_new_context_with_model: n_ctx      = 32768\n",
            "llama_new_context_with_model: freq_base  = 1000000.0\n",
            "llama_new_context_with_model: freq_scale = 1\n",
            "llama_new_context_with_model: KV self size  = 4096.00 MiB, K (f16): 2048.00 MiB, V (f16): 2048.00 MiB\n",
            "llama_build_graph: non-view tensors processed: 1124/1124\n",
            "llama_new_context_with_model: compute buffer total size = 2167.35 MiB\n",
            "llama_new_context_with_model: VRAM scratch buffer: 2164.04 MiB\n",
            "llama_new_context_with_model: total VRAM used: 2164.04 MiB (model: 0.00 MiB, context: 2164.04 MiB)\n",
            "AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
            "2023-12-14 07:11:04 INFO:\u001b[32mLOADER: llama.cpp\u001b[0m\n",
            "2023-12-14 07:11:04 INFO:\u001b[32mTRUNCATION LENGTH: 32768\u001b[0m\n",
            "2023-12-14 07:11:04 INFO:\u001b[32mINSTRUCTION TEMPLATE: Alpaca\u001b[0m\n",
            "2023-12-14 07:11:04 INFO:\u001b[32mLoaded the model in 94.67 seconds.\u001b[0m\n",
            "2023-12-14 07:11:04 INFO:\u001b[32mLoading the extension \"gallery\"...\u001b[0m\n",
            "Running on local URL:  http://127.0.0.1:7860\n",
            "Running on public URL: https://804e4d4c4daf551fb0.gradio.live\n",
            "\n",
            "This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)\n",
            "Output generated in 257.46 seconds (0.03 tokens/s, 8 tokens, context 26, seed 40486712)\n"
          ]
        }
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyP/v1e656d8lKVBzEUgqpeM",
      "include_colab_link": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}