interactive.ipynb

{
 "nbformat": 4,
 "nbformat_minor": 0,
 "metadata": {
  "colab": {
   "name": "V2_Submission_Custome_NER_From_XML.ipynb",
   "provenance": [],
   "collapsed_sections": []
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Binder for custom-ner-de\n",
    "\n",
    "This binder enables you to train, evaluate, and apply a custom NER model using spaCy. The only prerequisite is that you know how to annotate in and export documents from Transkribus."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Train a custom NER model using spaCy with German texts annotated in Transkribus\n",
    "\n",
    "Input URL to an annotated PAGE XML Zip file exported from Transkribus (for example a public link pointing to a file on SWITCHdrive):"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [],
   "source": [
    "ZIP_URL = \"https://drive.switch.ch/index.php/s/nIF5agoktDJP3li/download\"  # change this sample URL to your own URL"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Optional inputs:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [],
   "source": [
    "WORD_REMOVE = None  # list of words to be removed from the list of entities (false positives), e.g., WORD_REMOVE = [\"Händeklatschen\", \"Salpeter\"]\n",
    "PERSON_NAMES = None  # list of persons to be added to the model, e.g., PERSON_NAMES = [\"Max Mustermann\", \"Ada Lovelace\"]\n",
    "LOCATION_NAMES = None  # list of locations to be added to the model, e.g., LOCATION_NAMES = [\"Basel\", \"Mittelerde\"]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Train the model (this can take some time):"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from transkribus_custom_ner_de.client import Client\n",
    "my_client = Client()\n",
    "my_client.train_model(zip_url=ZIP_URL,\n",
    "                      word_remove=WORD_REMOVE,\n",
    "                      person_names=PERSON_NAMES,\n",
    "                      location_names=LOCATION_NAMES,\n",
    "                      epochs=100)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": true
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Save the model to the `/custom-ner-de/user_output/models/` directory:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "my_client.save_model()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": true
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "If you want to keep the model, you must download it from this directory to your local machine (Binder will reset after you close your browser).\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Evaluate a custom NER model\n",
    "\n",
    "Optional inputs:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [],
   "source": [
    "MODEL_PATH = None  # complete path to custom NER model directory (if no input is provided, the previously trained model is loaded)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Evaluate model:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "my_client.evaluate_model(model_path=MODEL_PATH)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": true
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Apply the custom NER model to new German texts transcribed in Transkribus\n",
    "\n",
    "Input URL to plain text file (for example a public link pointing to a file on SWITCHdrive) to which to custom NER model should be applied:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "outputs": [],
   "source": [
    "TEXT_URL = \"https://drive.switch.ch/index.php/s/4eBBIImulcOfMf7/download\"  # change this sample URL to your own URL"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Optional inputs:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "outputs": [],
   "source": [
    "MODEL_PATH = None  # complete path to custom NER model directory (if no input is provided, the previously trained model is loaded)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Apply the model to the text:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "my_client.apply_model(text_url=TEXT_URL,\n",
    "                      model_path=MODEL_PATH)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": true
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Display the result:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "my_client.result"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": true
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Save the result to the `/custom-ner-de/user_output/results/` directory:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "my_client.save_result2csv()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": true
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "If you want to keep the result, you must download it from this directory to your local machine (Binder will reset after you close your browser)."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  }
 ]
}