Text_Summarization.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Text_Summarization.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TIMsr2fPxRCr"
      },
      "source": [
        "# Automatic Text Summarization "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GIJzKna_xU6t"
      },
      "source": [
        "**Import Libraries**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Iw33Qd4NhWKY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "32319385-2b34-4fe4-93ea-3987f9ee3922"
      },
      "source": [
        "import bs4 as bs\n",
        "import urllib.request as url\n",
        "import re\n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "nltk.download('stopwords')\n",
        "from nltk.tokenize import sent_tokenize, word_tokenize\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.probability import FreqDist\n",
        "import heapq\n",
        "from string import punctuation"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LxRFfoYHyPBR"
      },
      "source": [
        "**Fetch Articles from Wikipedia**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QqJGxDS5heba"
      },
      "source": [
        "# web scrape Wikipedia article on Natural Language Processing\n",
        "scraped_data = url.urlopen('https://en.wikipedia.org/wiki/Natural_language_processing')\n",
        "article = scraped_data.read()\n",
        "parsed_article = bs.BeautifulSoup(article,'lxml')\n",
        "paragraphs = parsed_article.find_all('p')\n",
        "\n",
        "article_text = \"\"\n",
        "\n",
        "for p in paragraphs:\n",
        "    article_text += p.text"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yOazmqJmhvC7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 107
        },
        "outputId": "ec317d58-5001-4634-929a-f91bb65897d6"
      },
      "source": [
        "article_text"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Natural language processing (NLP) is a subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data.  The result is a computer capable of \"understanding\" the contents of documents, including the contextual nuances of the language within them. The technology can then accurately extract information and insights contained in the documents as well as categorize and organize the documents themselves. \\nChallenges in natural language processing frequently involve speech recognition, natural language understanding, and natural-language generation.\\nNatural language processing has its roots in the 1950s. Already in 1950, Alan Turing published an article titled \"Computing Machinery and Intelligence\" which proposed what is now called the Turing test as a criterion of intelligence, a task that involves the automated interpretation and generation of natural language, but at the time not articulated as a problem separate from artificial intelligence.\\nThe premise of symbolic NLP is well-summarized by John Searle\\'s Chinese room experiment: Given a collection of rules (e.g., a Chinese phrasebook, with questions and matching answers), the computer emulates natural language understanding (or other NLP tasks) by applying those rules to the data it is confronted with.\\nUp to the 1980s, most natural language processing systems were based on complex sets of hand-written rules.  Starting in the late 1980s, however, there was a revolution in natural language processing with the introduction of machine learning algorithms for language processing.  This was due to both the steady increase in computational power (see Moore\\'s law) and the gradual lessening of the dominance of Chomskyan theories of linguistics (e.g. transformational grammar), whose theoretical underpinnings discouraged the sort of corpus linguistics that underlies the machine-learning approach to language processing.[6]\\nIn the 2010s, representation learning and deep neural network-style machine learning methods became widespread in natural language processing, due in part to a flurry of results showing that such techniques[7][8] can achieve state-of-the-art results in many natural language tasks, for example in language modeling,[9] parsing,[10][11] and many others.\\nIn the early days, many language-processing systems were designed by symbolic methods, i.e., the hand-coding of a set of rules, coupled with a dictionary lookup:[12][13] such as by writing grammars or devising heuristic rules for stemming.\\nMore recent systems based on machine-learning algorithms have many advantages over hand-produced rules: \\nDespite the popularity of machine learning in NLP research, symbolic methods are still (2020) commonly used\\nSince the so-called \"statistical revolution\"[14][15] in the late 1980s and mid-1990s, much natural language processing research has relied heavily on machine learning. The machine-learning paradigm calls instead for using statistical inference to automatically learn such rules through the analysis of large corpora (the plural form of corpus, is a set of documents, possibly with human or computer annotations) of typical real-world examples.\\nMany different classes of machine-learning algorithms have been applied to natural-language-processing tasks. These algorithms take as input a large set of \"features\" that are generated from the input data. Increasingly, however, research has focused on statistical models, which make soft, probabilistic decisions based on attaching real-valued weights to each input feature. Such models have the advantage that they can express the relative certainty of many different possible answers rather than only one, producing more reliable results when such a model is included as a component of a larger system.\\nSome of the earliest-used machine learning algorithms, such as decision trees, produced systems of hard if-then rules similar to existing hand-written rules.  However, part-of-speech tagging introduced the use of hidden Markov models to natural language processing, and increasingly, research has focused on statistical models, which make soft, probabilistic decisions based on attaching real-valued weights to the features making up the input data. The cache language models upon which many speech recognition systems now rely are examples of such statistical models.  Such models are generally more robust when given unfamiliar input, especially input that contains errors (as is very common for real-world data), and produce more reliable results when integrated into a larger system comprising multiple subtasks.\\nSince the neural turn, statistical methods in NLP research have been largely replaced by neural networks. However, they continue to be relevant for contexts in which statistical interpretability and transparency is required.\\nA major drawback of statistical methods is that they require elaborate feature engineering. Since the early 2010s,[16] the field has thus largely abandoned statistical methods and shifted to neural networks for machine learning. Popular techniques include the use of word embeddings to capture semantic properties of words, and an increase in end-to-end learning of a higher-level task (e.g., question answering) instead of relying on a pipeline of separate intermediate tasks (e.g., part-of-speech tagging and dependency parsing). In some areas, this shift has entailed substantial changes in how NLP systems are designed, such that deep neural network-based approaches may be viewed as a new paradigm distinct from statistical natural language processing. For instance, the term neural machine translation (NMT) emphasizes the fact that deep learning-based approaches to machine translation directly learn sequence-to-sequence transformations, obviating the need for intermediate steps such as word alignment and language modeling that was used in statistical machine translation (SMT).\\nThe following is a list of some of the most commonly researched tasks in natural language processing. Some of these tasks have direct real-world applications, while others more commonly serve as subtasks that are used to aid in solving larger tasks.\\nThough natural language processing tasks are closely intertwined, they can be subdivided into categories for convenience. A coarse division is given below.\\nCognition refers to \"the mental action or process of acquiring knowledge and understanding through thought, experience, and the senses.\"[25] Cognitive science is the interdisciplinary, scientific study of the mind and its processes.[26] Cognitive linguistics is an interdisciplinary branch of linguistics, combining knowledge and research from both psychology and linguistics.[27] George Lakoff offers a methodology to build Natural language processing (NLP) algorithms through the perspective of Cognitive science, along with the findings of Cognitive linguistics:[28]\\nThe first defining aspect of this cognitive task of NLP is the application of the theory of Conceptual metaphor, explained by Lakoff as “the understanding of one idea, in terms of another” which provides an idea of the intent of the author.[29]\\nFor example, consider some of the meanings, in English, of the word “big”. When used as a Comparative, as in “That is a big tree,” a likely inference of the intent of the author is that the author is using the word “big” to imply a statement about the tree being ”physically large” in comparison to other trees or the authors experience.  When used as a Stative verb, as in ”Tomorrow is a big day”, a likely inference of the author’s intent it that ”big” is being used to imply ”importance”.  These examples are not presented to be complete, but merely as indicators of the implication of the idea of Conceptual metaphor.  The intent behind other usages, like in ”She is a big person” will remain somewhat ambiguous to a person and a cognitive NLP algorithm alike without additional information.\\nThis leads to the second defining aspect of this cognitive task of NLP, namely Probabilistic context-free grammar (PCFG) which enables cognitive NLP algorithms to assign relative measures of meaning  to a word, phrase, sentence or piece of text based on the information presented before and after the piece of text being analyzed. The mathematical equation for such algorithms is presented in US patent 9269353\\xa0:\\nWhere,\\n\\xa0 \\u2003 \\xa0RMM, is the Relative Measure of Meaning\\n\\xa0 \\u2003 \\xa0token, is any block of text, sentence, phrase or word\\n\\xa0 \\u2003 \\xa0N, is the number of tokens being analyzed\\n\\xa0 \\u2003 \\xa0PMM, is the Probable Measure of Meaning based on a corpora\\n\\xa0 \\u2003 \\xa0d, is the location of the token along the sequence of N-1 tokens\\n\\xa0 \\u2003 \\xa0PF, is the Probability Function specific to a language\\n'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YNebVP3YzLpT"
      },
      "source": [
        "**Text Preprocessing**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "72Xs4os6ihWN"
      },
      "source": [
        "# remove square brackets and extra spaces\n",
        "article_text = re.sub(r'\\[[0-9]*\\]', ' ', article_text)\n",
        "article_text = re.sub(r'\\s+', ' ', article_text)"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OcR_-PLdijFk"
      },
      "source": [
        "# remove special characters and digits\n",
        "formatted_article_text = re.sub('[^a-zA-Z]', ' ', article_text )\n",
        "formatted_article_text = re.sub(r'\\s+', ' ', formatted_article_text)"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cdfJfCnLzZHw"
      },
      "source": [
        "**Tokenize Sentences**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aLD1Mq1Jiyar"
      },
      "source": [
        "sentence_list = nltk.sent_tokenize(article_text)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VMLpE-qu0Doa"
      },
      "source": [
        "**Find Weighted Frequency of Occurrence**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2dgH3WKFjbdj"
      },
      "source": [
        "stopwords = nltk.corpus.stopwords.words('english')"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kl83RUF4j-On"
      },
      "source": [
        "word_frequencies = {}\n",
        "\n",
        "for word in nltk.word_tokenize(formatted_article_text):\n",
        "    if word not in stopwords and word not in punctuation:\n",
        "        if word not in word_frequencies.keys():\n",
        "            word_frequencies[word] = 1\n",
        "        else:\n",
        "            word_frequencies[word] += 1"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0X-IlQaDpCe2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e49cc3dd-e259-4ed8-a1c3-288fd6e06909"
      },
      "source": [
        "word_frequencies"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'A': 2,\n",
              " 'Alan': 1,\n",
              " 'Already': 1,\n",
              " 'Challenges': 1,\n",
              " 'Chinese': 2,\n",
              " 'Chomskyan': 1,\n",
              " 'Cognition': 1,\n",
              " 'Cognitive': 4,\n",
              " 'Comparative': 1,\n",
              " 'Computing': 1,\n",
              " 'Conceptual': 2,\n",
              " 'Despite': 1,\n",
              " 'English': 1,\n",
              " 'For': 2,\n",
              " 'Function': 1,\n",
              " 'George': 1,\n",
              " 'Given': 1,\n",
              " 'However': 2,\n",
              " 'In': 3,\n",
              " 'Increasingly': 1,\n",
              " 'Intelligence': 1,\n",
              " 'John': 1,\n",
              " 'Lakoff': 2,\n",
              " 'Machinery': 1,\n",
              " 'Many': 1,\n",
              " 'Markov': 1,\n",
              " 'Meaning': 2,\n",
              " 'Measure': 2,\n",
              " 'Moore': 1,\n",
              " 'More': 1,\n",
              " 'N': 2,\n",
              " 'NLP': 11,\n",
              " 'NMT': 1,\n",
              " 'Natural': 3,\n",
              " 'PCFG': 1,\n",
              " 'PF': 1,\n",
              " 'PMM': 1,\n",
              " 'Popular': 1,\n",
              " 'Probabilistic': 1,\n",
              " 'Probability': 1,\n",
              " 'Probable': 1,\n",
              " 'RMM': 1,\n",
              " 'Relative': 1,\n",
              " 'SMT': 1,\n",
              " 'Searle': 1,\n",
              " 'She': 1,\n",
              " 'Since': 3,\n",
              " 'Some': 2,\n",
              " 'Starting': 1,\n",
              " 'Stative': 1,\n",
              " 'Such': 2,\n",
              " 'That': 1,\n",
              " 'The': 9,\n",
              " 'These': 2,\n",
              " 'This': 2,\n",
              " 'Though': 1,\n",
              " 'Tomorrow': 1,\n",
              " 'Turing': 2,\n",
              " 'US': 1,\n",
              " 'Up': 1,\n",
              " 'When': 2,\n",
              " 'Where': 1,\n",
              " 'abandoned': 1,\n",
              " 'accurately': 1,\n",
              " 'achieve': 1,\n",
              " 'acquiring': 1,\n",
              " 'action': 1,\n",
              " 'additional': 1,\n",
              " 'advantage': 1,\n",
              " 'advantages': 1,\n",
              " 'aid': 1,\n",
              " 'algorithm': 1,\n",
              " 'algorithms': 8,\n",
              " 'alignment': 1,\n",
              " 'alike': 1,\n",
              " 'along': 2,\n",
              " 'ambiguous': 1,\n",
              " 'amounts': 1,\n",
              " 'analysis': 1,\n",
              " 'analyze': 1,\n",
              " 'analyzed': 2,\n",
              " 'annotations': 1,\n",
              " 'another': 1,\n",
              " 'answering': 1,\n",
              " 'answers': 2,\n",
              " 'application': 1,\n",
              " 'applications': 1,\n",
              " 'applied': 1,\n",
              " 'applying': 1,\n",
              " 'approach': 1,\n",
              " 'approaches': 2,\n",
              " 'areas': 1,\n",
              " 'art': 1,\n",
              " 'article': 1,\n",
              " 'articulated': 1,\n",
              " 'artificial': 2,\n",
              " 'aspect': 2,\n",
              " 'assign': 1,\n",
              " 'attaching': 2,\n",
              " 'author': 4,\n",
              " 'authors': 1,\n",
              " 'automated': 1,\n",
              " 'automatically': 1,\n",
              " 'based': 8,\n",
              " 'became': 1,\n",
              " 'behind': 1,\n",
              " 'big': 6,\n",
              " 'block': 1,\n",
              " 'branch': 1,\n",
              " 'build': 1,\n",
              " 'cache': 1,\n",
              " 'called': 2,\n",
              " 'calls': 1,\n",
              " 'capable': 1,\n",
              " 'capture': 1,\n",
              " 'categories': 1,\n",
              " 'categorize': 1,\n",
              " 'certainty': 1,\n",
              " 'changes': 1,\n",
              " 'classes': 1,\n",
              " 'closely': 1,\n",
              " 'coarse': 1,\n",
              " 'coding': 1,\n",
              " 'cognitive': 4,\n",
              " 'collection': 1,\n",
              " 'combining': 1,\n",
              " 'common': 1,\n",
              " 'commonly': 3,\n",
              " 'comparison': 1,\n",
              " 'complete': 1,\n",
              " 'complex': 1,\n",
              " 'component': 1,\n",
              " 'comprising': 1,\n",
              " 'computational': 1,\n",
              " 'computer': 4,\n",
              " 'computers': 2,\n",
              " 'concerned': 1,\n",
              " 'confronted': 1,\n",
              " 'consider': 1,\n",
              " 'contained': 1,\n",
              " 'contains': 1,\n",
              " 'contents': 1,\n",
              " 'context': 1,\n",
              " 'contexts': 1,\n",
              " 'contextual': 1,\n",
              " 'continue': 1,\n",
              " 'convenience': 1,\n",
              " 'corpora': 2,\n",
              " 'corpus': 2,\n",
              " 'coupled': 1,\n",
              " 'criterion': 1,\n",
              " 'data': 5,\n",
              " 'day': 1,\n",
              " 'days': 1,\n",
              " 'decision': 1,\n",
              " 'decisions': 2,\n",
              " 'deep': 3,\n",
              " 'defining': 2,\n",
              " 'dependency': 1,\n",
              " 'designed': 2,\n",
              " 'devising': 1,\n",
              " 'dictionary': 1,\n",
              " 'different': 2,\n",
              " 'direct': 1,\n",
              " 'directly': 1,\n",
              " 'discouraged': 1,\n",
              " 'distinct': 1,\n",
              " 'division': 1,\n",
              " 'documents': 4,\n",
              " 'dominance': 1,\n",
              " 'drawback': 1,\n",
              " 'due': 2,\n",
              " 'e': 5,\n",
              " 'earliest': 1,\n",
              " 'early': 2,\n",
              " 'elaborate': 1,\n",
              " 'embeddings': 1,\n",
              " 'emphasizes': 1,\n",
              " 'emulates': 1,\n",
              " 'enables': 1,\n",
              " 'end': 2,\n",
              " 'engineering': 1,\n",
              " 'entailed': 1,\n",
              " 'equation': 1,\n",
              " 'errors': 1,\n",
              " 'especially': 1,\n",
              " 'example': 2,\n",
              " 'examples': 3,\n",
              " 'existing': 1,\n",
              " 'experience': 2,\n",
              " 'experiment': 1,\n",
              " 'explained': 1,\n",
              " 'express': 1,\n",
              " 'extract': 1,\n",
              " 'fact': 1,\n",
              " 'feature': 2,\n",
              " 'features': 2,\n",
              " 'field': 1,\n",
              " 'findings': 1,\n",
              " 'first': 1,\n",
              " 'flurry': 1,\n",
              " 'focused': 2,\n",
              " 'following': 1,\n",
              " 'form': 1,\n",
              " 'free': 1,\n",
              " 'frequently': 1,\n",
              " 'g': 4,\n",
              " 'generally': 1,\n",
              " 'generated': 1,\n",
              " 'generation': 2,\n",
              " 'given': 2,\n",
              " 'gradual': 1,\n",
              " 'grammar': 2,\n",
              " 'grammars': 1,\n",
              " 'hand': 4,\n",
              " 'hard': 1,\n",
              " 'heavily': 1,\n",
              " 'heuristic': 1,\n",
              " 'hidden': 1,\n",
              " 'higher': 1,\n",
              " 'however': 2,\n",
              " 'human': 2,\n",
              " 'idea': 3,\n",
              " 'implication': 1,\n",
              " 'imply': 2,\n",
              " 'importance': 1,\n",
              " 'include': 1,\n",
              " 'included': 1,\n",
              " 'including': 1,\n",
              " 'increase': 2,\n",
              " 'increasingly': 1,\n",
              " 'indicators': 1,\n",
              " 'inference': 3,\n",
              " 'information': 3,\n",
              " 'input': 6,\n",
              " 'insights': 1,\n",
              " 'instance': 1,\n",
              " 'instead': 2,\n",
              " 'integrated': 1,\n",
              " 'intelligence': 3,\n",
              " 'intent': 4,\n",
              " 'interactions': 1,\n",
              " 'interdisciplinary': 2,\n",
              " 'intermediate': 2,\n",
              " 'interpretability': 1,\n",
              " 'interpretation': 1,\n",
              " 'intertwined': 1,\n",
              " 'introduced': 1,\n",
              " 'introduction': 1,\n",
              " 'involve': 1,\n",
              " 'involves': 1,\n",
              " 'knowledge': 2,\n",
              " 'language': 28,\n",
              " 'large': 4,\n",
              " 'largely': 2,\n",
              " 'larger': 3,\n",
              " 'late': 2,\n",
              " 'law': 1,\n",
              " 'leads': 1,\n",
              " 'learn': 2,\n",
              " 'learning': 13,\n",
              " 'lessening': 1,\n",
              " 'level': 1,\n",
              " 'like': 1,\n",
              " 'likely': 2,\n",
              " 'linguistics': 7,\n",
              " 'list': 1,\n",
              " 'location': 1,\n",
              " 'lookup': 1,\n",
              " 'machine': 13,\n",
              " 'major': 1,\n",
              " 'make': 2,\n",
              " 'making': 1,\n",
              " 'many': 6,\n",
              " 'matching': 1,\n",
              " 'mathematical': 1,\n",
              " 'may': 1,\n",
              " 'meaning': 1,\n",
              " 'meanings': 1,\n",
              " 'measures': 1,\n",
              " 'mental': 1,\n",
              " 'merely': 1,\n",
              " 'metaphor': 2,\n",
              " 'methodology': 1,\n",
              " 'methods': 6,\n",
              " 'mid': 1,\n",
              " 'mind': 1,\n",
              " 'model': 1,\n",
              " 'modeling': 2,\n",
              " 'models': 7,\n",
              " 'much': 1,\n",
              " 'multiple': 1,\n",
              " 'namely': 1,\n",
              " 'natural': 16,\n",
              " 'need': 1,\n",
              " 'network': 2,\n",
              " 'networks': 2,\n",
              " 'neural': 6,\n",
              " 'new': 1,\n",
              " 'nuances': 1,\n",
              " 'number': 1,\n",
              " 'obviating': 1,\n",
              " 'offers': 1,\n",
              " 'one': 2,\n",
              " 'organize': 1,\n",
              " 'others': 2,\n",
              " 'paradigm': 2,\n",
              " 'parsing': 2,\n",
              " 'part': 3,\n",
              " 'particular': 1,\n",
              " 'patent': 1,\n",
              " 'person': 2,\n",
              " 'perspective': 1,\n",
              " 'phrase': 2,\n",
              " 'phrasebook': 1,\n",
              " 'physically': 1,\n",
              " 'piece': 2,\n",
              " 'pipeline': 1,\n",
              " 'plural': 1,\n",
              " 'popularity': 1,\n",
              " 'possible': 1,\n",
              " 'possibly': 1,\n",
              " 'power': 1,\n",
              " 'premise': 1,\n",
              " 'presented': 3,\n",
              " 'probabilistic': 2,\n",
              " 'problem': 1,\n",
              " 'process': 2,\n",
              " 'processes': 1,\n",
              " 'processing': 16,\n",
              " 'produce': 1,\n",
              " 'produced': 2,\n",
              " 'producing': 1,\n",
              " 'program': 1,\n",
              " 'properties': 1,\n",
              " 'proposed': 1,\n",
              " 'provides': 1,\n",
              " 'psychology': 1,\n",
              " 'published': 1,\n",
              " 'question': 1,\n",
              " 'questions': 1,\n",
              " 'rather': 1,\n",
              " 'real': 5,\n",
              " 'recent': 1,\n",
              " 'recognition': 2,\n",
              " 'refers': 1,\n",
              " 'relative': 2,\n",
              " 'relevant': 1,\n",
              " 'reliable': 2,\n",
              " 'relied': 1,\n",
              " 'rely': 1,\n",
              " 'relying': 1,\n",
              " 'remain': 1,\n",
              " 'replaced': 1,\n",
              " 'representation': 1,\n",
              " 'require': 1,\n",
              " 'required': 1,\n",
              " 'research': 6,\n",
              " 'researched': 1,\n",
              " 'result': 1,\n",
              " 'results': 4,\n",
              " 'revolution': 2,\n",
              " 'robust': 1,\n",
              " 'room': 1,\n",
              " 'roots': 1,\n",
              " 'rules': 9,\n",
              " 'science': 3,\n",
              " 'scientific': 1,\n",
              " 'second': 1,\n",
              " 'see': 1,\n",
              " 'semantic': 1,\n",
              " 'senses': 1,\n",
              " 'sentence': 2,\n",
              " 'separate': 2,\n",
              " 'sequence': 3,\n",
              " 'serve': 1,\n",
              " 'set': 3,\n",
              " 'sets': 1,\n",
              " 'shift': 1,\n",
              " 'shifted': 1,\n",
              " 'showing': 1,\n",
              " 'similar': 1,\n",
              " 'soft': 2,\n",
              " 'solving': 1,\n",
              " 'somewhat': 1,\n",
              " 'sort': 1,\n",
              " 'specific': 1,\n",
              " 'speech': 4,\n",
              " 'state': 1,\n",
              " 'statement': 1,\n",
              " 'statistical': 11,\n",
              " 'steady': 1,\n",
              " 'stemming': 1,\n",
              " 'steps': 1,\n",
              " 'still': 1,\n",
              " 'study': 1,\n",
              " 'style': 1,\n",
              " 'subdivided': 1,\n",
              " 'subfield': 1,\n",
              " 'substantial': 1,\n",
              " 'subtasks': 2,\n",
              " 'summarized': 1,\n",
              " 'symbolic': 3,\n",
              " 'system': 2,\n",
              " 'systems': 6,\n",
              " 'tagging': 2,\n",
              " 'take': 1,\n",
              " 'task': 4,\n",
              " 'tasks': 8,\n",
              " 'techniques': 2,\n",
              " 'technology': 1,\n",
              " 'term': 1,\n",
              " 'terms': 1,\n",
              " 'test': 1,\n",
              " 'text': 3,\n",
              " 'theoretical': 1,\n",
              " 'theories': 1,\n",
              " 'theory': 1,\n",
              " 'thought': 1,\n",
              " 'thus': 1,\n",
              " 'time': 1,\n",
              " 'titled': 1,\n",
              " 'token': 2,\n",
              " 'tokens': 2,\n",
              " 'transformational': 1,\n",
              " 'transformations': 1,\n",
              " 'translation': 3,\n",
              " 'transparency': 1,\n",
              " 'tree': 2,\n",
              " 'trees': 2,\n",
              " 'turn': 1,\n",
              " 'typical': 1,\n",
              " 'underlies': 1,\n",
              " 'underpinnings': 1,\n",
              " 'understanding': 5,\n",
              " 'unfamiliar': 1,\n",
              " 'upon': 1,\n",
              " 'usages': 1,\n",
              " 'use': 2,\n",
              " 'used': 7,\n",
              " 'using': 2,\n",
              " 'valued': 2,\n",
              " 'verb': 1,\n",
              " 'viewed': 1,\n",
              " 'weights': 2,\n",
              " 'well': 2,\n",
              " 'whose': 1,\n",
              " 'widespread': 1,\n",
              " 'within': 1,\n",
              " 'without': 1,\n",
              " 'word': 6,\n",
              " 'words': 1,\n",
              " 'world': 3,\n",
              " 'writing': 1,\n",
              " 'written': 2}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Efgx86zJpF6b"
      },
      "source": [
        "maximum_frequncy = max(word_frequencies.values())\n",
        "\n",
        "for word in word_frequencies.keys():\n",
        "    word_frequencies[word] = (word_frequencies[word]/maximum_frequncy)"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lTEwJZ-mpRpb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "24848bce-67b7-4b34-a0af-6be923d35861"
      },
      "source": [
        "word_frequencies"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'A': 0.07142857142857142,\n",
              " 'Alan': 0.03571428571428571,\n",
              " 'Already': 0.03571428571428571,\n",
              " 'Challenges': 0.03571428571428571,\n",
              " 'Chinese': 0.07142857142857142,\n",
              " 'Chomskyan': 0.03571428571428571,\n",
              " 'Cognition': 0.03571428571428571,\n",
              " 'Cognitive': 0.14285714285714285,\n",
              " 'Comparative': 0.03571428571428571,\n",
              " 'Computing': 0.03571428571428571,\n",
              " 'Conceptual': 0.07142857142857142,\n",
              " 'Despite': 0.03571428571428571,\n",
              " 'English': 0.03571428571428571,\n",
              " 'For': 0.07142857142857142,\n",
              " 'Function': 0.03571428571428571,\n",
              " 'George': 0.03571428571428571,\n",
              " 'Given': 0.03571428571428571,\n",
              " 'However': 0.07142857142857142,\n",
              " 'In': 0.10714285714285714,\n",
              " 'Increasingly': 0.03571428571428571,\n",
              " 'Intelligence': 0.03571428571428571,\n",
              " 'John': 0.03571428571428571,\n",
              " 'Lakoff': 0.07142857142857142,\n",
              " 'Machinery': 0.03571428571428571,\n",
              " 'Many': 0.03571428571428571,\n",
              " 'Markov': 0.03571428571428571,\n",
              " 'Meaning': 0.07142857142857142,\n",
              " 'Measure': 0.07142857142857142,\n",
              " 'Moore': 0.03571428571428571,\n",
              " 'More': 0.03571428571428571,\n",
              " 'N': 0.07142857142857142,\n",
              " 'NLP': 0.39285714285714285,\n",
              " 'NMT': 0.03571428571428571,\n",
              " 'Natural': 0.10714285714285714,\n",
              " 'PCFG': 0.03571428571428571,\n",
              " 'PF': 0.03571428571428571,\n",
              " 'PMM': 0.03571428571428571,\n",
              " 'Popular': 0.03571428571428571,\n",
              " 'Probabilistic': 0.03571428571428571,\n",
              " 'Probability': 0.03571428571428571,\n",
              " 'Probable': 0.03571428571428571,\n",
              " 'RMM': 0.03571428571428571,\n",
              " 'Relative': 0.03571428571428571,\n",
              " 'SMT': 0.03571428571428571,\n",
              " 'Searle': 0.03571428571428571,\n",
              " 'She': 0.03571428571428571,\n",
              " 'Since': 0.10714285714285714,\n",
              " 'Some': 0.07142857142857142,\n",
              " 'Starting': 0.03571428571428571,\n",
              " 'Stative': 0.03571428571428571,\n",
              " 'Such': 0.07142857142857142,\n",
              " 'That': 0.03571428571428571,\n",
              " 'The': 0.32142857142857145,\n",
              " 'These': 0.07142857142857142,\n",
              " 'This': 0.07142857142857142,\n",
              " 'Though': 0.03571428571428571,\n",
              " 'Tomorrow': 0.03571428571428571,\n",
              " 'Turing': 0.07142857142857142,\n",
              " 'US': 0.03571428571428571,\n",
              " 'Up': 0.03571428571428571,\n",
              " 'When': 0.07142857142857142,\n",
              " 'Where': 0.03571428571428571,\n",
              " 'abandoned': 0.03571428571428571,\n",
              " 'accurately': 0.03571428571428571,\n",
              " 'achieve': 0.03571428571428571,\n",
              " 'acquiring': 0.03571428571428571,\n",
              " 'action': 0.03571428571428571,\n",
              " 'additional': 0.03571428571428571,\n",
              " 'advantage': 0.03571428571428571,\n",
              " 'advantages': 0.03571428571428571,\n",
              " 'aid': 0.03571428571428571,\n",
              " 'algorithm': 0.03571428571428571,\n",
              " 'algorithms': 0.2857142857142857,\n",
              " 'alignment': 0.03571428571428571,\n",
              " 'alike': 0.03571428571428571,\n",
              " 'along': 0.07142857142857142,\n",
              " 'ambiguous': 0.03571428571428571,\n",
              " 'amounts': 0.03571428571428571,\n",
              " 'analysis': 0.03571428571428571,\n",
              " 'analyze': 0.03571428571428571,\n",
              " 'analyzed': 0.07142857142857142,\n",
              " 'annotations': 0.03571428571428571,\n",
              " 'another': 0.03571428571428571,\n",
              " 'answering': 0.03571428571428571,\n",
              " 'answers': 0.07142857142857142,\n",
              " 'application': 0.03571428571428571,\n",
              " 'applications': 0.03571428571428571,\n",
              " 'applied': 0.03571428571428571,\n",
              " 'applying': 0.03571428571428571,\n",
              " 'approach': 0.03571428571428571,\n",
              " 'approaches': 0.07142857142857142,\n",
              " 'areas': 0.03571428571428571,\n",
              " 'art': 0.03571428571428571,\n",
              " 'article': 0.03571428571428571,\n",
              " 'articulated': 0.03571428571428571,\n",
              " 'artificial': 0.07142857142857142,\n",
              " 'aspect': 0.07142857142857142,\n",
              " 'assign': 0.03571428571428571,\n",
              " 'attaching': 0.07142857142857142,\n",
              " 'author': 0.14285714285714285,\n",
              " 'authors': 0.03571428571428571,\n",
              " 'automated': 0.03571428571428571,\n",
              " 'automatically': 0.03571428571428571,\n",
              " 'based': 0.2857142857142857,\n",
              " 'became': 0.03571428571428571,\n",
              " 'behind': 0.03571428571428571,\n",
              " 'big': 0.21428571428571427,\n",
              " 'block': 0.03571428571428571,\n",
              " 'branch': 0.03571428571428571,\n",
              " 'build': 0.03571428571428571,\n",
              " 'cache': 0.03571428571428571,\n",
              " 'called': 0.07142857142857142,\n",
              " 'calls': 0.03571428571428571,\n",
              " 'capable': 0.03571428571428571,\n",
              " 'capture': 0.03571428571428571,\n",
              " 'categories': 0.03571428571428571,\n",
              " 'categorize': 0.03571428571428571,\n",
              " 'certainty': 0.03571428571428571,\n",
              " 'changes': 0.03571428571428571,\n",
              " 'classes': 0.03571428571428571,\n",
              " 'closely': 0.03571428571428571,\n",
              " 'coarse': 0.03571428571428571,\n",
              " 'coding': 0.03571428571428571,\n",
              " 'cognitive': 0.14285714285714285,\n",
              " 'collection': 0.03571428571428571,\n",
              " 'combining': 0.03571428571428571,\n",
              " 'common': 0.03571428571428571,\n",
              " 'commonly': 0.10714285714285714,\n",
              " 'comparison': 0.03571428571428571,\n",
              " 'complete': 0.03571428571428571,\n",
              " 'complex': 0.03571428571428571,\n",
              " 'component': 0.03571428571428571,\n",
              " 'comprising': 0.03571428571428571,\n",
              " 'computational': 0.03571428571428571,\n",
              " 'computer': 0.14285714285714285,\n",
              " 'computers': 0.07142857142857142,\n",
              " 'concerned': 0.03571428571428571,\n",
              " 'confronted': 0.03571428571428571,\n",
              " 'consider': 0.03571428571428571,\n",
              " 'contained': 0.03571428571428571,\n",
              " 'contains': 0.03571428571428571,\n",
              " 'contents': 0.03571428571428571,\n",
              " 'context': 0.03571428571428571,\n",
              " 'contexts': 0.03571428571428571,\n",
              " 'contextual': 0.03571428571428571,\n",
              " 'continue': 0.03571428571428571,\n",
              " 'convenience': 0.03571428571428571,\n",
              " 'corpora': 0.07142857142857142,\n",
              " 'corpus': 0.07142857142857142,\n",
              " 'coupled': 0.03571428571428571,\n",
              " 'criterion': 0.03571428571428571,\n",
              " 'data': 0.17857142857142858,\n",
              " 'day': 0.03571428571428571,\n",
              " 'days': 0.03571428571428571,\n",
              " 'decision': 0.03571428571428571,\n",
              " 'decisions': 0.07142857142857142,\n",
              " 'deep': 0.10714285714285714,\n",
              " 'defining': 0.07142857142857142,\n",
              " 'dependency': 0.03571428571428571,\n",
              " 'designed': 0.07142857142857142,\n",
              " 'devising': 0.03571428571428571,\n",
              " 'dictionary': 0.03571428571428571,\n",
              " 'different': 0.07142857142857142,\n",
              " 'direct': 0.03571428571428571,\n",
              " 'directly': 0.03571428571428571,\n",
              " 'discouraged': 0.03571428571428571,\n",
              " 'distinct': 0.03571428571428571,\n",
              " 'division': 0.03571428571428571,\n",
              " 'documents': 0.14285714285714285,\n",
              " 'dominance': 0.03571428571428571,\n",
              " 'drawback': 0.03571428571428571,\n",
              " 'due': 0.07142857142857142,\n",
              " 'e': 0.17857142857142858,\n",
              " 'earliest': 0.03571428571428571,\n",
              " 'early': 0.07142857142857142,\n",
              " 'elaborate': 0.03571428571428571,\n",
              " 'embeddings': 0.03571428571428571,\n",
              " 'emphasizes': 0.03571428571428571,\n",
              " 'emulates': 0.03571428571428571,\n",
              " 'enables': 0.03571428571428571,\n",
              " 'end': 0.07142857142857142,\n",
              " 'engineering': 0.03571428571428571,\n",
              " 'entailed': 0.03571428571428571,\n",
              " 'equation': 0.03571428571428571,\n",
              " 'errors': 0.03571428571428571,\n",
              " 'especially': 0.03571428571428571,\n",
              " 'example': 0.07142857142857142,\n",
              " 'examples': 0.10714285714285714,\n",
              " 'existing': 0.03571428571428571,\n",
              " 'experience': 0.07142857142857142,\n",
              " 'experiment': 0.03571428571428571,\n",
              " 'explained': 0.03571428571428571,\n",
              " 'express': 0.03571428571428571,\n",
              " 'extract': 0.03571428571428571,\n",
              " 'fact': 0.03571428571428571,\n",
              " 'feature': 0.07142857142857142,\n",
              " 'features': 0.07142857142857142,\n",
              " 'field': 0.03571428571428571,\n",
              " 'findings': 0.03571428571428571,\n",
              " 'first': 0.03571428571428571,\n",
              " 'flurry': 0.03571428571428571,\n",
              " 'focused': 0.07142857142857142,\n",
              " 'following': 0.03571428571428571,\n",
              " 'form': 0.03571428571428571,\n",
              " 'free': 0.03571428571428571,\n",
              " 'frequently': 0.03571428571428571,\n",
              " 'g': 0.14285714285714285,\n",
              " 'generally': 0.03571428571428571,\n",
              " 'generated': 0.03571428571428571,\n",
              " 'generation': 0.07142857142857142,\n",
              " 'given': 0.07142857142857142,\n",
              " 'gradual': 0.03571428571428571,\n",
              " 'grammar': 0.07142857142857142,\n",
              " 'grammars': 0.03571428571428571,\n",
              " 'hand': 0.14285714285714285,\n",
              " 'hard': 0.03571428571428571,\n",
              " 'heavily': 0.03571428571428571,\n",
              " 'heuristic': 0.03571428571428571,\n",
              " 'hidden': 0.03571428571428571,\n",
              " 'higher': 0.03571428571428571,\n",
              " 'however': 0.07142857142857142,\n",
              " 'human': 0.07142857142857142,\n",
              " 'idea': 0.10714285714285714,\n",
              " 'implication': 0.03571428571428571,\n",
              " 'imply': 0.07142857142857142,\n",
              " 'importance': 0.03571428571428571,\n",
              " 'include': 0.03571428571428571,\n",
              " 'included': 0.03571428571428571,\n",
              " 'including': 0.03571428571428571,\n",
              " 'increase': 0.07142857142857142,\n",
              " 'increasingly': 0.03571428571428571,\n",
              " 'indicators': 0.03571428571428571,\n",
              " 'inference': 0.10714285714285714,\n",
              " 'information': 0.10714285714285714,\n",
              " 'input': 0.21428571428571427,\n",
              " 'insights': 0.03571428571428571,\n",
              " 'instance': 0.03571428571428571,\n",
              " 'instead': 0.07142857142857142,\n",
              " 'integrated': 0.03571428571428571,\n",
              " 'intelligence': 0.10714285714285714,\n",
              " 'intent': 0.14285714285714285,\n",
              " 'interactions': 0.03571428571428571,\n",
              " 'interdisciplinary': 0.07142857142857142,\n",
              " 'intermediate': 0.07142857142857142,\n",
              " 'interpretability': 0.03571428571428571,\n",
              " 'interpretation': 0.03571428571428571,\n",
              " 'intertwined': 0.03571428571428571,\n",
              " 'introduced': 0.03571428571428571,\n",
              " 'introduction': 0.03571428571428571,\n",
              " 'involve': 0.03571428571428571,\n",
              " 'involves': 0.03571428571428571,\n",
              " 'knowledge': 0.07142857142857142,\n",
              " 'language': 1.0,\n",
              " 'large': 0.14285714285714285,\n",
              " 'largely': 0.07142857142857142,\n",
              " 'larger': 0.10714285714285714,\n",
              " 'late': 0.07142857142857142,\n",
              " 'law': 0.03571428571428571,\n",
              " 'leads': 0.03571428571428571,\n",
              " 'learn': 0.07142857142857142,\n",
              " 'learning': 0.4642857142857143,\n",
              " 'lessening': 0.03571428571428571,\n",
              " 'level': 0.03571428571428571,\n",
              " 'like': 0.03571428571428571,\n",
              " 'likely': 0.07142857142857142,\n",
              " 'linguistics': 0.25,\n",
              " 'list': 0.03571428571428571,\n",
              " 'location': 0.03571428571428571,\n",
              " 'lookup': 0.03571428571428571,\n",
              " 'machine': 0.4642857142857143,\n",
              " 'major': 0.03571428571428571,\n",
              " 'make': 0.07142857142857142,\n",
              " 'making': 0.03571428571428571,\n",
              " 'many': 0.21428571428571427,\n",
              " 'matching': 0.03571428571428571,\n",
              " 'mathematical': 0.03571428571428571,\n",
              " 'may': 0.03571428571428571,\n",
              " 'meaning': 0.03571428571428571,\n",
              " 'meanings': 0.03571428571428571,\n",
              " 'measures': 0.03571428571428571,\n",
              " 'mental': 0.03571428571428571,\n",
              " 'merely': 0.03571428571428571,\n",
              " 'metaphor': 0.07142857142857142,\n",
              " 'methodology': 0.03571428571428571,\n",
              " 'methods': 0.21428571428571427,\n",
              " 'mid': 0.03571428571428571,\n",
              " 'mind': 0.03571428571428571,\n",
              " 'model': 0.03571428571428571,\n",
              " 'modeling': 0.07142857142857142,\n",
              " 'models': 0.25,\n",
              " 'much': 0.03571428571428571,\n",
              " 'multiple': 0.03571428571428571,\n",
              " 'namely': 0.03571428571428571,\n",
              " 'natural': 0.5714285714285714,\n",
              " 'need': 0.03571428571428571,\n",
              " 'network': 0.07142857142857142,\n",
              " 'networks': 0.07142857142857142,\n",
              " 'neural': 0.21428571428571427,\n",
              " 'new': 0.03571428571428571,\n",
              " 'nuances': 0.03571428571428571,\n",
              " 'number': 0.03571428571428571,\n",
              " 'obviating': 0.03571428571428571,\n",
              " 'offers': 0.03571428571428571,\n",
              " 'one': 0.07142857142857142,\n",
              " 'organize': 0.03571428571428571,\n",
              " 'others': 0.07142857142857142,\n",
              " 'paradigm': 0.07142857142857142,\n",
              " 'parsing': 0.07142857142857142,\n",
              " 'part': 0.10714285714285714,\n",
              " 'particular': 0.03571428571428571,\n",
              " 'patent': 0.03571428571428571,\n",
              " 'person': 0.07142857142857142,\n",
              " 'perspective': 0.03571428571428571,\n",
              " 'phrase': 0.07142857142857142,\n",
              " 'phrasebook': 0.03571428571428571,\n",
              " 'physically': 0.03571428571428571,\n",
              " 'piece': 0.07142857142857142,\n",
              " 'pipeline': 0.03571428571428571,\n",
              " 'plural': 0.03571428571428571,\n",
              " 'popularity': 0.03571428571428571,\n",
              " 'possible': 0.03571428571428571,\n",
              " 'possibly': 0.03571428571428571,\n",
              " 'power': 0.03571428571428571,\n",
              " 'premise': 0.03571428571428571,\n",
              " 'presented': 0.10714285714285714,\n",
              " 'probabilistic': 0.07142857142857142,\n",
              " 'problem': 0.03571428571428571,\n",
              " 'process': 0.07142857142857142,\n",
              " 'processes': 0.03571428571428571,\n",
              " 'processing': 0.5714285714285714,\n",
              " 'produce': 0.03571428571428571,\n",
              " 'produced': 0.07142857142857142,\n",
              " 'producing': 0.03571428571428571,\n",
              " 'program': 0.03571428571428571,\n",
              " 'properties': 0.03571428571428571,\n",
              " 'proposed': 0.03571428571428571,\n",
              " 'provides': 0.03571428571428571,\n",
              " 'psychology': 0.03571428571428571,\n",
              " 'published': 0.03571428571428571,\n",
              " 'question': 0.03571428571428571,\n",
              " 'questions': 0.03571428571428571,\n",
              " 'rather': 0.03571428571428571,\n",
              " 'real': 0.17857142857142858,\n",
              " 'recent': 0.03571428571428571,\n",
              " 'recognition': 0.07142857142857142,\n",
              " 'refers': 0.03571428571428571,\n",
              " 'relative': 0.07142857142857142,\n",
              " 'relevant': 0.03571428571428571,\n",
              " 'reliable': 0.07142857142857142,\n",
              " 'relied': 0.03571428571428571,\n",
              " 'rely': 0.03571428571428571,\n",
              " 'relying': 0.03571428571428571,\n",
              " 'remain': 0.03571428571428571,\n",
              " 'replaced': 0.03571428571428571,\n",
              " 'representation': 0.03571428571428571,\n",
              " 'require': 0.03571428571428571,\n",
              " 'required': 0.03571428571428571,\n",
              " 'research': 0.21428571428571427,\n",
              " 'researched': 0.03571428571428571,\n",
              " 'result': 0.03571428571428571,\n",
              " 'results': 0.14285714285714285,\n",
              " 'revolution': 0.07142857142857142,\n",
              " 'robust': 0.03571428571428571,\n",
              " 'room': 0.03571428571428571,\n",
              " 'roots': 0.03571428571428571,\n",
              " 'rules': 0.32142857142857145,\n",
              " 'science': 0.10714285714285714,\n",
              " 'scientific': 0.03571428571428571,\n",
              " 'second': 0.03571428571428571,\n",
              " 'see': 0.03571428571428571,\n",
              " 'semantic': 0.03571428571428571,\n",
              " 'senses': 0.03571428571428571,\n",
              " 'sentence': 0.07142857142857142,\n",
              " 'separate': 0.07142857142857142,\n",
              " 'sequence': 0.10714285714285714,\n",
              " 'serve': 0.03571428571428571,\n",
              " 'set': 0.10714285714285714,\n",
              " 'sets': 0.03571428571428571,\n",
              " 'shift': 0.03571428571428571,\n",
              " 'shifted': 0.03571428571428571,\n",
              " 'showing': 0.03571428571428571,\n",
              " 'similar': 0.03571428571428571,\n",
              " 'soft': 0.07142857142857142,\n",
              " 'solving': 0.03571428571428571,\n",
              " 'somewhat': 0.03571428571428571,\n",
              " 'sort': 0.03571428571428571,\n",
              " 'specific': 0.03571428571428571,\n",
              " 'speech': 0.14285714285714285,\n",
              " 'state': 0.03571428571428571,\n",
              " 'statement': 0.03571428571428571,\n",
              " 'statistical': 0.39285714285714285,\n",
              " 'steady': 0.03571428571428571,\n",
              " 'stemming': 0.03571428571428571,\n",
              " 'steps': 0.03571428571428571,\n",
              " 'still': 0.03571428571428571,\n",
              " 'study': 0.03571428571428571,\n",
              " 'style': 0.03571428571428571,\n",
              " 'subdivided': 0.03571428571428571,\n",
              " 'subfield': 0.03571428571428571,\n",
              " 'substantial': 0.03571428571428571,\n",
              " 'subtasks': 0.07142857142857142,\n",
              " 'summarized': 0.03571428571428571,\n",
              " 'symbolic': 0.10714285714285714,\n",
              " 'system': 0.07142857142857142,\n",
              " 'systems': 0.21428571428571427,\n",
              " 'tagging': 0.07142857142857142,\n",
              " 'take': 0.03571428571428571,\n",
              " 'task': 0.14285714285714285,\n",
              " 'tasks': 0.2857142857142857,\n",
              " 'techniques': 0.07142857142857142,\n",
              " 'technology': 0.03571428571428571,\n",
              " 'term': 0.03571428571428571,\n",
              " 'terms': 0.03571428571428571,\n",
              " 'test': 0.03571428571428571,\n",
              " 'text': 0.10714285714285714,\n",
              " 'theoretical': 0.03571428571428571,\n",
              " 'theories': 0.03571428571428571,\n",
              " 'theory': 0.03571428571428571,\n",
              " 'thought': 0.03571428571428571,\n",
              " 'thus': 0.03571428571428571,\n",
              " 'time': 0.03571428571428571,\n",
              " 'titled': 0.03571428571428571,\n",
              " 'token': 0.07142857142857142,\n",
              " 'tokens': 0.07142857142857142,\n",
              " 'transformational': 0.03571428571428571,\n",
              " 'transformations': 0.03571428571428571,\n",
              " 'translation': 0.10714285714285714,\n",
              " 'transparency': 0.03571428571428571,\n",
              " 'tree': 0.07142857142857142,\n",
              " 'trees': 0.07142857142857142,\n",
              " 'turn': 0.03571428571428571,\n",
              " 'typical': 0.03571428571428571,\n",
              " 'underlies': 0.03571428571428571,\n",
              " 'underpinnings': 0.03571428571428571,\n",
              " 'understanding': 0.17857142857142858,\n",
              " 'unfamiliar': 0.03571428571428571,\n",
              " 'upon': 0.03571428571428571,\n",
              " 'usages': 0.03571428571428571,\n",
              " 'use': 0.07142857142857142,\n",
              " 'used': 0.25,\n",
              " 'using': 0.07142857142857142,\n",
              " 'valued': 0.07142857142857142,\n",
              " 'verb': 0.03571428571428571,\n",
              " 'viewed': 0.03571428571428571,\n",
              " 'weights': 0.07142857142857142,\n",
              " 'well': 0.07142857142857142,\n",
              " 'whose': 0.03571428571428571,\n",
              " 'widespread': 0.03571428571428571,\n",
              " 'within': 0.03571428571428571,\n",
              " 'without': 0.03571428571428571,\n",
              " 'word': 0.21428571428571427,\n",
              " 'words': 0.03571428571428571,\n",
              " 'world': 0.10714285714285714,\n",
              " 'writing': 0.03571428571428571,\n",
              " 'written': 0.07142857142857142}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ip5afGww06no"
      },
      "source": [
        "**Frequency Distribution**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GxvqKzwyqUwr",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 342
        },
        "outputId": "88418228-6bc7-4824-9462-4bfa98679122"
      },
      "source": [
        "frequency_dist = nltk.FreqDist(word_frequencies)\n",
        "frequency_dist.plot(30)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAFFCAYAAADhF+qFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO2deZwUxfXAv28PWK4FBEWCAoocQUVlUfGIt4nG23jEK95XojHhl0STmHjEJCZqjPHWxCseUWM8QLxiREUBOQVFUAQvRFE5ZbmWfb8/qobt7e2Z6dmd2dndft/Ppz+73fO6qnqmu1/Vq1fviapiGIZhJJeSYjfAMAzDKC6mCAzDMBKOKQLDMIyEY4rAMAwj4ZgiMAzDSDhlxW5ArvTs2VP79+/fqHNXr15Nhw4d8iprZVqZVqaV2dLKjGLq1KlfquqmkR+qaqvaqqqqtLFMmTIl77JWppVpZVqZLa3MKIApmua9aqYhwzCMhGOKwDAMI+GYIjAMw0g4pggMwzASjikCwzCMhFMwRSAid4nIYhF5K83nIiJ/E5F5IjJTRIYXqi2GYRhGego5IrgHOCjD5wcDA/12DnBrAdtiGIZhpKFgikBVXwGWZBA5ArjPu7hOBLqJSO9CteePz7zDJS9+xVsLlxeqCsMwjFaJaAHzEYhIf2CMqm4X8dkY4GpVHe/3XwQuVtUpEbLn4EYN9O7du2r06NE5t+Xq15Yy+dO1/Gy3buy2RUVW+erqajp27Jg3OSvTyrQyrczmKjOKESNGTFXVEZEfpltplo8N6A+8leazMcCegf0XgRHZymzsyuJfPz5T+108Rv/x6vxY8m1tpaGVaWVamckpMwpa6MrihcCWgf0t/LGC0Luri8/x+Yo1harCMAyjVVJMRfAU8APvPTQSWK6qiwpVWa9KZw76zBSBYRhGPQoWfVREHgL2AXqKyCfAZUA5gKreBowFvgvMA6qB0wvVFoDNU4pguSkCwzCMIAVTBKp6QpbPFfhRoeoPs3nX9oCNCAzDMMIkZmVxr8CIQAvoKWUYhtHaSIwi6FJRTocyYW1NLctXry92cwzDMFoMiVEEAJt0cJdr5iHDMIw6EqYISgGbMDYMwwiSKEXQIzUiMEVgGIaxkUQpgo0jAjMNGYZhbCRhisBdrq0uNgzDqCNRiqCHzREYhmE0IFGKoM40tLbILTEMw2g5JEoR9DDTkGEYRgMSpQgqK0ooLRGWrFrHmvUbit0cwzCMFkGiFEGpCJt1cTGHFpt5yDAMA0iYIgDYvKuFozYMwwiSPEVgeQkMwzDqkThFkIpC+rm5kBqGYQAJVAQp09AiUwSGYRhAEhVBakRgpiHDMAwggYrAchcbhmHUJ3GKoHdXy11sGIYRJHGKIDVHsHjlGmprLWWlYRhG4hRBRXkpXTuUs36DsqR6XbGbYxiGUXQSpwggsJbAzEOGYRjJVAS9bJ7AMAxjI4lUBL3Nc8gwDGMjiVQEqRGBrSUwDMNIqCKwOQLDMIw6kqkIurpQ1GYaMgzDSKgi6GUjAsMwjI0kUhH07toBsBGBYRgGJFQRdO9YTruyElauqaF6XU2xm2MYhlFUEqkIRIRelX6ewMxDhmEknEQqArBMZYZhGCkSqwhswtgwDMORWEVgIwLDMAxHchVBV8tdbBiGAaYIbERgGEbiSa4i2GgaWlvklhiGYRSXgioCETlIROaKyDwRuSTi874i8pKITBeRmSLy3UK2J0hqsthMQ4ZhJJ2CKQIRKQVuBg4GhgIniMjQkNilwCOquhPwfeCWQrUnTEoRLF65hpoNtc1VrWEYRoujkCOCXYB5qjpfVdcB/wKOCMkoUOn/7wp8WsD21KNdWQk9O7ejVuHLry1lpWEYyUVUC5PAXUSOAQ5S1bP8/inArqp6QUCmN/A80B3oBBygqlMjyjoHOAegd+/eVaNHj25Um6qrq+nYsePG/Z+98CULltVw9f6bMHCTdhll45aZD1kr08q0Mq3MpsqGGTFixFRVHRH5oaoWZAOOAf4e2D8FuCkkMwr4P///bsBsoCRTuVVVVdpYpkyZUm//jLvf0H4Xj9FnZi3KKhu3zHzIWplWppVpZTZVNgwwRdO8VwtpGloIbBnY38IfC3Im8AiAqk4AKoCeBWxTPSxTmWEYRmHnCCYDA0VkKxFph5sMfiok8xGwP4CIfBOnCL4oYJvqkXIhXWSeQ4ZhJJiCKQJVrQEuAJ4D3sF5B70tIleKyOFe7P+As0XkTeAh4DQ/hGkWUorARgSGYSSZskIWrqpjgbGhY78N/D8b2KOQbcjExtXFNiIwDCPBJHZlMQTiDdmIwDCMBJNoRdArEIG0GS1ShmEYLYpEK4LKijI6lJdSvW4DK9daykrDMJJJohWBiNg8gWEYiSfRigACUUhNERiGkVBMEVheAsMwEk7iFYGFozYMI+kkXhFsXtkesBGBYRjJxRSBTRYbhpFwTBF07QDYiMAwjORiisDiDRmGkXASrwh6dm5HibgsZetqLGWlYRjJI/GKoKy0hE27uAnjxSttVGAYRvJIvCIAMw8ZhpFsTBFQt5bAEtQYhpFETBEAvc2F1DCMBGOKAMtdbBhGsjFFQCDw3Iq1RW6JYRhG82OKgMBksZmGDMNIIKYIqDMNLVqxusgtMQzDaH5MERB0H11rKSsNw0gcpgiATu3L6FJRxrqaWpZWry92cwzDMJoVUwQey1RmGEZSMUXg2dxcSA3DSCimCDy9Ki1lpWEYycQUgSe1utjCTBiGkTRMEXgsd7FhGEnFFIFnczMNGYaRUEwReGyy2DCMpGKKwGOTxYZhJBVTBJ4endpRXiosq17PmvUbit0cwzCMZsMUgaekRNisiy0qMwwjeeSsCESku4gMK0Rjik1qnsDMQ4ZhJIlYikBExolIpYhsAkwD7hSRvxS2ac2P5S42DCOJxB0RdFXVFcDRwH2quitwQOGaVRx6WbwhwzASSFxFUCYivYHjgDEFbE9R2bxre8BMQ4ZhJIu4iuAK4DlgnqpOFpGtgfcK16zisHnXDoCNCAzDSBZxFcEiVR2mqj8EUNX5QNY5AhE5SETmisg8EbkkjcxxIjJbRN4WkQfjNz3/2OpiwzCSSFxFcGPMYxsRkVLgZuBgYChwgogMDckMBH4J7KGq2wI/idmegmC5iw3DSCJlmT4Ukd2A3YFNRWRU4KNKoDRL2bvgTEnzfVn/Ao4AZgdkzgZuVtWlAKq6OLfm55fNKt0cweKVa6m1lJWGYSQEyZSjV0T2BvYBzgNuC3y0EhitqmnnCUTkGOAgVT3L758C7KqqFwRkngDeBfbAKZbLVfXZiLLOAc4B6N27d9Xo0aPjXl89qqur6dixY0aZ0578nJXrlBv378w3NumclzJzlbUyrUwr08psqmyYESNGTFXVEZEfqmrWDegXRy50zjHA3wP7pwA3hWTGAI8D5cBWwMdAt0zlVlVVaWOZMmVKVpnvXP+y9rt4jD78woS8lZmrrJVpZVqZVmZTZcMAUzTNezWjaShAexG5A+hPwJykqvtlOGchsGVgfwt/LMgnwCRVXQ8sEJF3gYHA5Jjtyju9u1Yw57OVfFVt8YYMw0gGcRXBozjT0N+BuG/IycBAEdkKpwC+D5wYknkCOAG4W0R6AoOA+THLLwipMBNLVtcWsxmGYRjNRlxFUKOqt+ZSsKrWiMgFuPUHpcBdqvq2iFyJG6I85T/7tojMximYn6vqV7nUk29Sq4u/Wm0jAsMwkkFcRTBaRH6Is+evTR1U1SWZTlLVscDY0LHfBv5XYJTfWgQpF9IlpggMw0gIcRXBqf7vzwPHFNg6v80pPr28aWj8R2vY6crns8p3Kq3l0W1W09uvSjYMw2htxFIEqrpVoRvSUti+T1e6dSxnWfV6llavzyq/FLhr/AJ+fcjQrLKGYRgtkViKQER+EHVcVe/Lb3OKT8/O7Zn0q/15/Y1p7LDDDhll5362khPunMjDkz/mpwcOomO7uAMswzCMlkPcN9fOgf8rgP1xeQnanCIAaF9WSmX7Ejbp1C6j3G4DejBok3LeXbKeJ6Z/yom79m2mFhqGYeSPuKahC4P7ItIN+FdBWtTK+O7Ajrw7aTn3TfiAE3bZEhEpdpMMwzByorE5i1fhVgInnpFbVNCzc3vmfLaSSQsyOlEZhmG0SOKmqhwtIk/57WlgLs6VNPGUl8hGk9C9r39Q3MYYhmE0grhzBNcG/q8BPlTVTwrQnlbJSbv25ZaX5vH87M/5dNlqvtHNXEkNw2g9xBoRqOrLwBygC9AdWFfIRrU2elVWcNB2m7OhVnlg0ofFbo5hGEZOxDUNHQe8ARyLy1s8yYeZNjyn7d4fgIfe+Jg1621VsmEYrYe4k8W/BnZW1VNV9Qe4pDO/KVyzWh9V/bqz7TcqWbJqHWNmLip2cwzDMGITVxGUaP3sYV/lcG4iEBFO9aOCe1//IJVvwTAMo8UT92X+rIg8JyKnichpwNOEgskZcPgO36B7x3JmLVzO9I+XFbs5hmEYscioCERkGxHZQ1V/DtwODPPbBOCOZmhfq6KivJTjdzZXUsMwWhfZRgR/BVYAqOp/VHWUqo7CrSH4a6Eb1xo5eWRfSgTGzlrE4pVrit0cwzCMrGRTBL1UdVb4oD/WvyAtauVs0b0jBw7txfoNyoOTPip2cwzDMLKSTRF0y/CZrZpKw6m79QfggUkfsa7GUl4ahtGyyaYIpojI2eGDInIWMLUwTWr97DagB4N6deaLlWt59u3Pit0cwzCMjGRTBD8BTheRcSJynd9eBs4ELip881onIsIP/KjAJo0Nw2jpZFQEqvq5qu4OXAF84LcrVHU3VbWubgaO2qkPXSrKmPrhUt5auLzYzTEMw0hL3FhDL6nqjX77X6Eb1Rbo1L6MY6u2BGxUYBhGy8ZWBxeQH+zWDxF48s1PWbLK4vQZhtEyMUVQQPr37MQ+gzZlXU0t/5psrqSGYbRMTBEUmFT8ofsnfMiGWos/ZBhGyyNuYhqjkew1cFO26tmJBV+u4tn3y2m3efYYRJ99XdMMLTMMw3CYIigwJSXCKSP7ceWY2dw1YyV3zXgt6zkCdOn9BXsN2rTwDTQMI/GYImgGjt95S95YsIR5n35Fp04dM8quWreBeYu/5lePz+L5n+5Fx3b2ExmGUVjsLdMMdGpfxm2nVDF16lSqqqoyytZsqOXAa15gwdLV/OX5d7n00KHN1ErDMJKKTRa3MMpKSzh/RFdKBO56bQEzP7G8BoZhFBZTBC2QAd3LOWOPrahVuOSxWazfYIHrDMMoHKYIWiijvj2ILbp3YPaiFfxj/IJiN8cwjDaMKYIWSsd2Zfz+qO0BuP6Fd/nwq1VFbpFhGG0VUwQtmL0HbcqRO36DtTW1/OrxWajagjTDMPKPKYIWzm8OHUr3juW8Nu8rHpu2sNjNMQyjDWKKoIXTo3N7Lj3EuZBe9fRsvvx6bZFbZBhGW8MUQSvg6OF9+NbAniyrXs/vxswudnMMw2hjmCJoBYgIvz9yeyrKS3hyxqe8NHdxsZtkGEYboqCKQEQOEpG5IjJPRC7JIPc9EVERGVHI9rRm+vboyKgDBwFw6eNvsWqtBaYzDCM/FEwRiEgpcDNwMDAUOEFEGsRLEJEuuPzHkwrVlrbCGXtsxXZ9Klm4bDXXPf9usZtjGEYboZAjgl2Aeao6X1XXAf8CjoiQ+x3wJ2BNAdvSJigrLeHqo4dRWiLc8/oC3vzYwk8YhtF0CqkI+gAfB/Y/8cc2IiLDgS1V9ekCtqNNsV2frpy5pws/cfFjM6mxZDeGYTQRKdQiJRE5BjhIVc/y+6cAu6rqBX6/BPgfcJqqfiAi44CfqeqUiLLOAc4B6N27d9Xo0aMb1abq6mo6dswcBjpX2WKUubZG+cnzX7J41Qb6V5bQpSJ7ENnOZbWcv/MmdGqXXfe35Gu3Mq1MKzOebJgRI0ZMVdXoeVhVLcgG7AY8F9j/JfDLwH5X4EvgA7+tAT4FRmQqt6qqShvLlClT8i5brDJfeXex9r9kjPa7OP528b/fbPZ2WplWppXZ/GVGAUzRNO/VQuYjmAwMFJGtgIXA94ETAwpoOdAztZ9pRGA05FsDN+W5n+zFa1NnMWjQoIyyy1ev58cPTuNfkz/myJ36MHLrHs3USsMwWgMFUwSqWiMiFwDPAaXAXar6tohcidNMTxWq7qQwqFcXVvZqT9U2PbPKjvtmJx6ZvYpf/WcWYy/6FhXlpc3QQsMwWgMFzVCmqmOBsaFjv00ju08h25J0jh7SmWlfCvMWf81N/5vHz74zuNhNMgyjhWArixNCeanwx6NdWOvbXn6fOZ+tKHKLDMNoKZgiSBA799+Ek3btS02tcsljs9hgrqeGYWCKIHFcfPAQelW2Z8bHy/jnhA+K3RzDMFoApggSRmVFOVcesR0A1zw3l0+XrS5yiwzDKDamCBLId7bdnIO23ZxV6zbwmyfessxnhpFwTBEklCuO2JYuFWW8OGcxT89aVOzmGIZRREwRJJRelRVccvAQAC5/ajbLq9cXuUWGYRQLUwQJ5oSd+7JL/0348uu1/GHsO8VujmEYRcIUQYIpKRH+cPT2tCst4eEpH/P6+18Wu0mGYRQBUwQJZ5vNOnPBftsA8Kv/zGLN+g1FbpFhGM2NKQKD8/YewKBenfngq2r+9uJ7xW6OYRjNTEFjDRmtg3ZlJfzx6GEcc9vr3PHKfNrvXMnqyuxmoi+Wr6eqGdpnGEZhMUVgAFDVrzunjOzHfRM+5PpJy2FSvBTSs1bN5pffHUJ5qQ0uDaO1YorA2MgvDhrCyjU1zFv4BZWVXTLKqsIb87/irtcW8NbC5dx00k5s1qWimVpqGEY+MUVgbKRz+zKuP35Hpk6dSlVVdqPPg89P4IYpq3jjgyUc+rfx3HrycKr6bdIMLTUMI5/YeN5oNIN7tGP0hXuyy1absHjlWo6/fSL3vv6BhawwjFaGKQKjSWzWpYIHztqVs/bcippa5bKn3mbUI2+yep25oRpGa8EUgdFkyktLuPTQodx4wk50bFfK49MXctQtr/HhV6uK3TTDMGJgisDIG4ft8A2e+NEebNWzE3M+W8lhN47npTmLi90swzCyYIrAyCuDenXhyQv24MChvVixpoYz7p3M9S+8y7oNytqaDVm39THl1tXUFvtSDaPNYF5DRt6prCjn9pOruPXl97n2+bnc8OJ73ADwn2fjFRBTbtc+7bnjm+vp2rG80W01DMNGBEaBKCkRfrTvNtx7+i5s0b0DZSXQrrQk6xZXTgQmLVzL4TeP551FK4p9uYbRqrERgVFQ9hq0KeMv3i/22oS4ch99Vc2pd77Kgq+qOeqW17j66GEcuVOffDTZMBKHjQiMVknfHh35/X49+N7wLVizvpafPDyDy596m/UbbO7AMHLFFIHRamlfKlx77DB+d+R2lJcK97z+ASfeOZHFK9YUu2mG0aowRWC0akSEU0b24+Fzd2Pzygomf7CUQ24cz+QPlhS7aYbRajBFYLQJhvftzugL92Tk1pvwxcq1nHDHRO5+bYGFuzCMGJgiMNoMm3Zpz/1n7srZ33LhLq4YPZufPDyDNbbmwDAyYl5DRpuirLSEXx8ylB237M7P//0mT874lPFzS+gzeXys81etWkWnCdll29eu4br+1fTt0bGpTTaMomOKwGiTHDKsN4N6debc+6cy/4tVfPXJ8vgnL40ne+iNr3LD93di3yGbNbKVhtEyMEVgtFkG9urCsxftxRPj3mDw4CGxzpkzZw5DhmSWrVXlD09MZfKnaznj3slctP9AfrzfQEpKJB/NNoxmxxSB0aZpV1bCgO7l7LBlt1jyNYvjyf5i9268sbIb1z4/l7/+9z1mfrKc64/b0cJdGK0Smyw2jEZQIi6Exj2n70K3juX8b85iDrvJwl0YrRNTBIbRBPYetCmjL9iT7fpU8tESF+7iiekLi90sw8gJUwSG0US23KQj/z5vd46tqh/uwkJlG60FUwSGkQcqykv58zHD+P1RFu7CaH3YZLFh5AkR4aRd+zG0dyXn3z+NKR+6cBe7bl7K2EWzs57/+ecrYsnlIttayqxduYrtdthA+7LSWOUa+cUUgWHkmZ36dmfMj/fkggenMXH+EsasBN5bEO/kuHK5yLaSMqcvmcitJw+nd9cO8cs28kJBFYGIHATcAJQCf1fVq0OfjwLOAmqAL4AzVPXDQrbJMJqDnp1duIsxMxcxfc77bLnFFlnP+fiTT2LJ5SLbGsrcUKvcOe5dZny8jMNuHM+NJwxntwE9YpVv5IeCKQIRKQVuBg4EPgEmi8hTqhocJ04HRqhqtYicD/wZOL5QbTKM5qSstIQjd+rDlrWfUVW1dVb5qVOXxpLLRba1lDmwfAn/mL2B1+Z9xcn/mMQlBw3hrG9thYgt0msOCjlZvAswT1Xnq+o64F/AEUEBVX1JVav97kQgXjfDMIw2Rdf2Jdx7+i6ct/cANtQqvx/7Dhc8NJ1Va2uK3bREIIUK0ysixwAHqepZfv8UYFdVvSCN/E3AZ6p6VcRn5wDnAPTu3btq9OjRjWpTdXU1HTvGCxIWV9bKtDKtzPyWOfGTNdw4eTlrapQtKsv4xe7d6NOlLFK2mO1syWVGMWLEiKmqOiLyQ1UtyAYcg5sXSO2fAtyURvZk3IigfbZyq6qqtLFMmTIl77JWppVpZea/zPc+X6n7XzdO+108Rrf97bP67FuLWmQ7W2qZUQBTNM17tZCmoYXAloH9LfyxeojIAcCvgcNVdW0B22MYRithm80688SP9uC722/O12trOPefU/nzs3PYUGuJhgpBIb2GJgMDRWQrnAL4PnBiUEBEdgJux5mQFhewLYZhtDI6ty/j5hOHc+er87n6mTncMu59Zn6ynJGb1vBF+0VZz3//kzWx5HKRXbl0PVWxSmxdFEwRqGqNiFwAPIdzH71LVd8WkStxQ5SngGuAzsCj3jvgI1U9vFBtMgyjdSEinLPXALbr05ULH5zO+HlfMn4eMGFavALiyuUgO3fNbC45eAhlpW0nMENB1xGo6lhgbOjYbwP/H1DI+g3DaBvsPqAnY368J397cR7zF35Ot27ZQ4UvW7Ysllxc2Q21yktzFvP38QuYtXA5N504nE27tI9VfkvHVhYbhtEq6N21A388enumTl1HVVV2A83UqVNjyeUie/9zE7hhyiomLVjCYTeO55aThzO8b/dYdbRk2s7YxjAMo8B8s2c7nr5wT3bu353PVqzh+Nsn8M+JH6a8H1stpggMwzByYLPKCh48eySn79Gf9RuU3zzxFj97dCZr1m8odtMajSkCwzCMHCkvLeGyw7blhu/vSEV5CY9N+4Tv3fo6Hy+pzn5yC8QUgWEYRiM5Ysc+PP7DPejXoyNvf7qCQ28cz8vvflHsZuWMTRYbhmE0gW/2ruSpC/Zk1MMzeHHOYk67+w1GHTCIbSs28NXX2dfILl9bG0sOoHp9YbLemSIwDMNoIl07lHPnD0Zw00vzuP6/73LdC++6D0b/N14BT8WT22PLCr41spGNzIApAsMwjDxQUiL8eP+BDNuiK78bM5svlldTVl6e9bya9etjyQF0KCtMWG5TBIZhGHlkn8Gbsc/gzWKvTch1vUMhsMliwzCMhGOKwDAMI+GYIjAMw0g4pggMwzASjikCwzCMhGOKwDAMI+GYIjAMw0g4pggMwzASjrS2ONoi8gXwYSNP7wl8mWdZK9PKtDKtzJZWZhT9VHXTyE9UNTEbLldyXmWtTCvTyrQyW1qZuW5mGjIMw0g4pggMwzASTtIUwR0FkLUyrUwr08psaWXmRKubLDYMwzDyS9JGBIZhGEYIUwSGYRgJxxSBYRhGwkmEIhCRDiIyuNjtaKmIyCAReVFE3vL7w0Tk0hjndReRYYVvoRFGREpEpLIZ67sozrHWgjhOFpHf+v2+IrJLsdtVLNr8ZLGIHAZcC7RT1a1EZEfgSlU9vBnbsCcwUFXvFpFNgc6quiAkMzzi1OXAh6pa42UqgPOAbYBZwD9Sn4XKuhFI+8Oq6o9D8i8DPwduV9Wd/LG3VHW7iLLHAYfj0pxOBRYDr6nqqJDcscCzqrrSK5XhwFWqOi3LNQfbOS24LyJ/Bq4CVgPPAsOAn6rq/RHtzCrbiPqzXlMO175JlrqXRFzTg7jffwMwGagEblDVa0Jy1wF3qerbmeoIyO8O9CeQulZV7wvJTFPV4aFj0wP3S073nD/nbxGiy3GLpp4MyFUAZwLbAhWBMs+IKDPqe12pqutDcrcCtcB+qvpNEekOPK+qO2cpK3hNSwKypcB/VXXfTOcE5AcBtwK9VHU736E6XFWvCsnFvuebQhJyFl8O7AKMA1DVGSKyVZSgiKyk4c28HJgC/B/wZsTnAOKK1gY9NBG5DBgBDAbuBsqB+4E9QqK34F4YM3152wFvA11F5HxVfR64F1gPvAocDAwFonplU6KuLwMdVfUNkXqJsRsoGE9XVV0hImcB96nqZSIyM0LuN6r6qFeCBwDX4G78XQMy1/m/Fbjv6E3ctQ/z17BbqMxvq+ovROQo4APgaOAV3PcZJo5srvXHuaa4clNx91JUNnIFto44PtR/9ycBzwCX+HKuCcm9A9whImW4e+4hVV0eUR4i8k9gADADp2BS9d/nPz8BOBHYSkSeCpzaBQgqq9Q9twfuvnzY7x8LzI6qG/e9DwEe9fvfAxYAO4jIvqr6E3/8n8Ac4DvAlcBJ/hqjmAZsCSzFfbfdgM9E5HPgbFVNJf3dVVWHi8h0AFVdKiLtQmXF/o1UdYOI1IpI13TfdYg78Z0vf/5Mr+ivCsnlcs83nkIsV25JGzDR/50eODYzjezvgHNxN3klcA7wJ+B4YFwj65+Bu5Ey1g/8B9g2sD8U+DfuZpvhj80KfF4GTMvTd/QM7mUwze8fAzyTRnYW0Bt4Htg5w/VM93//CJwY/g0irn37wP52wL8j5N7yf/8OHOT/fzNNmbnIxq0/1jXlcu05/k5v4zoSjwJ7Z7om/9lg4GpcbK4HgX0jZN7BWwbSlNEP2AeYAOwd2IYDZRHyE4PHfXsnpil7IlAauqcnAKXA7Ijvc2aMMu8EvhPY/zbuZTsSmBQ4PsnXk7rnN23qbwQ8CXwE/AP4WxxgGeoAACAASURBVGpLIzs5fF/gn/Pwbx73Pm7KloQRwdsiciJQKiIDgR8Dr6eRPVxVdwjs3yEiM1T1YhH5VVhYRDaj/lD1o4gy16mqioj6czqlqXuQBobyqjpbRIao6vxAT3194POaUA++Ad4MdTFOqQTbuV9I9Ee4hSpDRGQhrld2cppirwSew5mDJovI1sB7EXILReR24EDgTyLSnvRzUoNVdVagfW+JyDcj5MaIyBzcMPl8f31r0pSZi2zc+uNeUy7XjjdLDKT+b/RKhOjtuF7hm8ArItIPWJGmzFJcb3sILkjZm8AoETlXVb8fEH0L2BxYFFWOqn6IUyTh0VE6uuM6UanRQmd/LJ1sZ9yoG6ATsIm63vXagFzqvl8mItsBnwGbpSlzpKqeHWj/8yJyraqe63+HFH8DHgc2E5Hf4zo/aefFYv5G//FbHL4UkQF4C4OIHEP0bzA6h/u48eRbs7S0DegI/B5nU53i/69IIzsBOA730Jb4/1MjihkBucNxL79VuJdmLV5zR5T5M9wDPB8429dxYYTcwzjzQarHdQvwCNCeut7DBtyDvwJYiTPfpP5fEVHm8zjb6ju+zLuAP2X4rjoBXfL4vR+NmxsBN4r4dhrZh3A9nn38difOnBEluwm+F+nr2DxDG4KyndLJxq0/7jXleO1n4UZZS4GXcA/8/3L4nqN65dcD8/x9t0vos7n+72jgKV/nUpxyfyq1RZS5MnDvrUndixFyp+MUxz04U+YC4NQ0bT/Tf363l5/vv49OwDWh76g7sJeXWQycm6bM53Gdn35++wXwAoHef0B2CK4TdAHwzQzfcezfCOiA61hk+922Bv4LVAMLgfFA/wi59hH3ca98PKPBrc1PFueC793egOv9KG7o+lPcD1WlquO93JvAfrjJoZ1EZF/gZFU9M025B+KGqAI8p6ovRMh0AH4I7OkPvYZTBmtwNvyvG3E9U1W1SkRmquowf2yyBibE/LH2OPtsf+pPGF4ZUWbcSa4zVfUfoWNXq+olEWVWAOfjHnRwNtBbVXVNSK4UOCSinX+JKPNHwAOquszvdwdOUNVbcq2/kRO73XG26mA7p0XIzQJ2xnU4dhSRIcAfVPXogMyo8Hmh+utdv4icDjyiqqsi6uuqqstFZO8sZb6c7jNxQ9EjcL3vSwLHS3AmmPnUzYdMUtXPMpTVGzeHB67D82mEzFba0LmiwTF/vCdwGfWfoytwo46+qjovXVsytDHrb+TlcnZM8RaCElVdmebzqEn6BseaSptXBCIymvQTwLeHXzYxy5yiqiO8QthJVWtF5E2tb1ZqVkTkI1XtGzo2UVVHishzuKHwpzjb94CQ3LO472QqdROGqOp1hJCYHkYiMhb3In7A79+MG4mlU5YdcA/q3AzXOBanGGfhRmGpdl4RITtDVXcMHdvo5ZJL/SKygAyThqq6dUj+d8BpwPvU3XuqDU1yGxWziMzATWCuFZG3VXXbgMxl/t/BuBdSatL2MOANVW1gxhORPrgecVARNTA3iXOcWBRQeh1wSv6DiGsNn9vg+8z0HQdkhqjqHEnjtRVWmGlehlNVtSpbG/NBnN8o1SZcB3FcpmfDH49S7suBqeocWjYH+uAmhU+k7t6rBG5T1SF5u0CS4TU0HzcR9JDfPx43zB2EMwGckhL09rezadjjDLupLRORzrie4wMishhnJmqAiByNm3DeDPdjRnoYicgeOA+n8MMb5T0SWVXEsatEpCvO4+lG3E300wi5LVT1oJj1xPUw+h7wlIjUAgcByzIogcNxni/tcN4p6XpSW6RGNjEoFRFR39Pxo4mwV0is+lU10sssA8cBA1R1XQzZT0SkG/AE8IKILCWUeCml6ETkFWB4qvcoIpcDT0dcz9XA93HeOkFPoKh5h0eB3QP7G/yx8Kgx2PstwXlZRXWiXhSR7wH/SX33EYzCOWJcR/1Omvj9/XydQ3Auo11D9VcSsNWH2jkIZ47tT/3nqIESzoGsv5FnvR9tBY/VRsiB+/5G4Ex0AIfiPAbPE5FHgc9xnYktgOCIbyXQYL6yyeTb1tTSNrx9PeoYIbs+bhL5T7gH+XupLeL8TriHoQw4FTcB3SNN/fPIYH8MyM3BuYRuBvRIbTlc50dN+I7uIOA1k0U2o4cRzp6Z2voB04GbUsfSlDkV6Ep9D4pZEXJ/Io2tPUL2Gtwcy/5+ewS4ron1C24S/Td+vy8hG7w//hiwWSN+h71x80/t0nw+F2gf2G+Pt/lnkstSZ5SnSgOvFJwdP7XdCfw66hpxL6pa3ATvStLMX3nZDrhOyuO4SdafEpi/w5mf7ga+CtX/N2D3NGW+iTPz7QJUpbbGPhu5/EY4b6ETcS/0gbjO121pynkFt54otd8ZeNl/J0GPqQbvn0JsSRgRdBaRvuo9ekSkL+5LBwj32Dqq6sWZCvM9yzHqFo7U4ibEMvG5qqbzeQ6yXFWfyVJ3OluxUHdNQfl7gYu0vp38OvUjHHEriWtxCu10EZkPrKVu1BLV+87mYRT0vU79PcRvSrR/fFRPKqo3ORF43Nui15NmdOW5GOcKfL7ffwE3IRxF3PpvwS9Cwrkar8S99HcOyf0RmO6/343eLxoa4fh76W31w3zNYJf33Ae8ISKP4679CNwka5j5OBfLtRGfhflCRA5X1ad8m44gIhWiqp4eoyxUtUscOc+9uMnn1MKyE3HXeJwv60ngSRHZTVUnxCyzRlVvzaENsZCGi0L74O79IBfiFORanLvuc7j7JIrNqP/7rMeZ5FZLfY+pMeK8HvuTZf6uKSRBEfwfMF5E3sc9PFsBP/STNOGX+BgR+a6qjk1XmOa+cGSKiDyMG1YGXwphN7OXROQaXM8oKBe0l2Z6yG6IODYspQR8WUtFJGi/7QPs2PC09KjqfOCAdJNcmrsZBeK7+P4FN5E/S313KUM7a3GT2nFeCnHrj7MICdx99SdCcxkRbdwgInODHZVMqOrvReQZ4Fs4RXW6qk6PEK0GZojIi9S/lxqs7sWtVH5ARG7y+58QMJemkDpHipG+7gm4Fa7zI2QPp27ifZyqjklzSdup6tDA/ksiErX4bLq4yf+sK4tx7pY/xI0ygtfeYEI/LhJ/UeghqvprnDJInXssdQvmgjwATBKR1Arqw4AH/XMV/A6epG7+Lo5ibxRtXhGo6lj/cKcmV+Zq3QTxX0PiFwG/8ho5U4/za2CWiLxAYG4gzYNWiXswvx1sFg39jVNeFiNCchttmxoxKZqFEhHprqpLYaP3S/A3X6DOTzwr6UYjqV60NvRciQqz8Ls0L65gT+oh0vekPsYtFEurBETkEVU9TpynRwO5NKOcuPWv97341LzDpkS/6KtVNSp8QhTdcYroDerfS+k8TTb4OjVN3VDnBpoRfy3nq3Mo6OzrTeed9iBwM3CU3/8+7ruqt6raz0/sjHvRAVwkInuo6i8jypwmIiNVdaI/d1eiV8XnsrL4VP/354Fj6UaicTkK2Am3ahlV/VREojplv6ThS7/BMXEPzT04M2tKmZynqqlrPykgnsv8XaNp815DAOIWoYQXVd2X/oys5Z0adVxVs5mJmoT4AFlpUFWt9/ISkR/gJpYexSm1Y4Dfq+o//eefUH8iKlzgxs+kznMlnWw9JSXeZdUPqa/C2ex/q6rhcAyEzisFOqlqg4VSInIP7oF+hvq9vWA7e6vqInGLraLamVHxZan/JJyzwXBcr/8Y4FJVDT/of/Hte4r0o7uUbKQbZ5SZSFyQt7Nx5ijBvaDuUNUbM11TJsR7lsWQmxlWohLhKScu3MiOfkSW+j6nB88NKOlyXC/7I7/fD5gTGiVs9EQK3FPlwKtx2p0PROQNVd1FvPeS77VP0DqX7IOB7+JMWg8HTq3EhQVpEMxORGap6vYx6r4DuFEDCx4LQZsfEfgX2D44RTAWNyE7Hh9Lxcvk5M4W54UvIr9Q1T9LmmBcqdGDiJysqven63GHetpRnkmdcAtzehDqxarqfSIyhbpRxdGqGhx2luLmFjIvUaZRo5GUt8ohuJfV0yISjqMCgEQEUxORBsHUcDbZBTjvnkgPIFVNrc78YXi+R0T+hJs7aFT9qvqAOBfB/XHf2ZFp5n9S5rfgi6re6C5QZrZ5gSBn4sxTqwLXMwE3KRm8noG4eYpw5yeqVzxdXAyhR6k/IgmPWJ8RkUuAf/lrOR4Y60eZYdNLN+pWFneNqPPQzJfZgKwri0VkP1X9n9T3LtpIxPXEwvfex4hbKd5NRM4GzsBNmKf4FDeSORxnwkmxkmgvPXCjoZ1VdXKWJuwJnCbOhTnb/F2jafMjAt/72AHXK9lBRHoB96vqgQGZO1T1HBF5KaII1ZDrmdT5lYcFtw7IHKaqo7ONHsQt+b89XY873QvYD00vwr0cUh4xi/1nleqCk0UuhEo9tNKIhSkicjfR135GSG4MbiHegbge9Gqcz3uDtRbiff59j3s4PphaU272qGuL6tXGqT/d95iiifbnYKDDdrhe8qoIc+TGhU1a5/NfgfOA2z4kNx63qOp6nO35dNx8ToMRpf89Iy6pwe/ZYPFWSH5rL/d9XHyjcbiX1l7AJar6cPrTMyMuwOFjwPY4k0pnnOfW7QGZK9QFQIx1PTnWPwvn8pptUWiZRkQDTlPmHFwU4Q9xCjjyBd/YkW2utPkRAbBa3YKvGnHx2xfjVnxuRFXP8X9jhZClvh2/Ahdhsd7LQlVH+78ZRw+pmzluj9u/lEbh7Ij34vzKl4bEHsT1ulIePBtPp769NOtIIILgxF8FzjzRYDUobph8EHCtqi4Tt4L05xFyAOV+uH8kcJOqrhcfmymIxPARF5HzcSu0B0j9qKhdcKtM49Yf/DzoCdWX+pEtP8I5IATb2Qv4A/ANVT1YRIYCu2lopbVve5fAeRtX7KZp5924CcbH/f6ROJfFMB1U9UUREf/CuNyPZBooAo3vDRTXCeBQXCiTpbi4SBdrhpXFmQiNklPtvNn/rRezS1Uv839jXU+OTMOtg0l3/6Z4L+q+TTMS+06cilX1Q4kIYx/n3FxIgiKYIm4xyJ24B/pr3HC6ARJzglNVvwqd+td0D1qcl5eXy7qYTZxX0dHU+f1HTuyp6qH+b7aHd/8sn0eV/Vio3Q/hTG1huWrgPyKymTiXXXATflHEDab2KHAbzg10Q8Tn4JTgMzjTSDCcxcoMPffbcCanmYH6N3qEpb5HEbkTeFy9V5m3DR8ZUd49uJd2ynvkXZztOOqlvRF1w/Mn/OiwQSgOVf2LuJXdqQnGdF5Da8W52L4nIhfgRmaRLw/JLdZ/1rwFuGv8Fs5MMgBnenpFVaO82rKRUpKRK6pDbcspDEeO7AqcJCKp3nuqzPDoMmsHMXDuhwASClwZRuJ7LDWJNm8aCiIi/YFKVY2Knx97gjM0l5BaZXl+GrPHm7gXTTh8w9SQ3Ou4PANhuccCMrU4O2ENET39sDlBRF5U1f2zHWsK4jK/Pa2q24SOH45bOfoN3CisL24icNuGpUSW22CYLTmEFRAX2fETdeEA9sHlGLhPA+60AdmgWU5xv2mpqv4mJNdggi/NsVRIgmDilgYhL/zxqBW7e6tqZLRPcZOvvaj/Mv4oJLMzzqumG27eqBL4s6pOiijvUZyCPpGAR46qXhSSi8xboNEJZ0pxL+59cXMvq7UJIRHErag+ROtWVHfB3XN7BWRyDsORQ/2NNs+ku2cjno9+uO89HLZiBt5jKXAvRZo4m0KbHxGIyF5RxzQ6zG/cCc5gDJ4aXG/yuDRNiLvAJetiNlWNlVrU9/I6Aj3FLSILxinpE6eMNOUK7jsKjkQ+I2ICFvcCGkkoMF+Gsg8h1CvFvZiCNvpcfMQfA0aIyDa4EdSTuNHCdyNkg9dTgXMoiJoE/tSPFFNJQU4i2iy2SkR6UOdmOpLACCPEYYH/a3AjoyOiBEXkQpzt/3Pc75Ay9YVfCopzueyH60GCGxFHvTy2UdVjReQIVb1X3MT5qxFyI3AeMBl7juLWLnTCjbpfxc1pLM50Tgx6UX/x5zp/bCOaYxiOXIhrj0/TQUz3jo37fMQNY98k2rwioL5dugK39DwVHCpM3DjyZ2poIY2Esp414uWVdTFbDpwL/ATX25hKnSJYgQv30Cj8DTlbI4JoRbBeVb8Sl1u3RFVfEpHwug0AROQ2nOLaF2f2OYb6Q/9wpqg4PuK16nI2HI1zv7tR/EKwiOuqF1xPRK7FrSUIcwLuRZyy0b/ij4UZheuRDhCR13Cxro5NU3cuNu2LcCGOw6bJMA/gvqOMC9o8cWP9Z8xbEGAmLqzDdjjlt0xEJqjq6iznZSK4ohqcOe6eNLJZlUYBCXcQPyB9BzHu8/GIZPZYyg/aDHEsWtKGmyh+LM1ncePNN8gMhvMyCe4vwC31XxCxzY84PxWjZTUZcgzkeK0N8h7k4fu7F5+ZLIvcf3F26RtxC49uwCWziZKdGfrbGecnHpZrkEci6pg/Pgn3kn4L2MofeyvmNXYH5mX4vAuBODERn7fHdbK2xb0Qy0kT+wcX/PBF6jKqDcOtTYiSfYmI/AMRcuNz+D1Tsf73pi7W/3lp6s6atyD0HV2I84pZm4f7bjhOEV6Ei/ibTu7XuLmmy/02A/hVPu79fG5pno/X08geiDNTXwscWIj2JGqOADaaN97W0KIV/1lGu7LURUP8M/V7pZXAzzWm/Tui3hKcV0k6r5ZGkWbyu0Gy9RzLjOv2dh3uOyrBmVC6AjtoRARSEZmkqruKyEScIv4K9xuF5x1ix2YX56lzHm7hz0N+xHacqv4pQja4CrkU14O/UlVvCsltj+udpkZ7X+KSrrzVhHZmDesdmAjdFmcDf5o0C+q8/P44JRgOMdEoX3pfZqyFb35y+lu4UcEHOPPQq6r6v8bWnSsiUkVdPoJXNHpCvRD19qAuF4LinCiu1IgRnDfxrME9P6nn44EoWS9fSf15oUa7LEfR5k1DUn9BVwkutk66F2E2u/JgnHtcN+rbdlfiPH6i6s+aIEWde+tN1C1Eyhdxk63nQiy3N1x+3FoCgfkkOsk9OLNYN5yCTU2ibwwQJ3Wx2TuIi5UUnPPoGFWguoVzPw7sL8DF/4kiuMCpBhcoMMof/HZglKq+5Nu1D+4+2b2x7SReWO+U98xHfgsuqIvqyZ2OC6lSTp1pKCqsSWxX1/ALPwMVuNXqU9N8h83BDJwJqwxAYsZyygP/wpkLv+f3T8J5ix0QFtT6SYPSupiLyLm4xDprcL9l2AU8PxR7iFToDRd7JLWdBOyRQTYVWvkXeLMK0cnJd8uh/qgwv1FlXou7gdImEm/EtRckiXqWOs/H2aarcfbi1LYAt5Av6pwOOLt6unDEp+JMEyv939T2FG61dLCsR/zfWaH6Z+JNT024tqjwzG+maef/MrUzcE7GsN4h2WNjHmsQmjrDNT2Ds2O/6ffLCITgxpuZqJ+qMi+mywLdfxfiRmpv+998VlN/9xzqbmB6JBTOPOJ7zPh94lLi9ix424v9w7WkjZh2ZVyv50e4sMR3pbY0Zc4KvtxxpocG+Y2pmyNYl68HDbf4K5UvuRvOdt3gZZbn77Arztf8IeryxvYjTS4Cf84jOP/zff12Z+qFHpLLGpsd6O3/9ovamnhtjwO/8dfXH5fs/PHGtDMgG5W7NrKdRM9NRR27G+fhE6f+VG6OYC6GBp2X1rLh8n/EzuOR57r/ggvGF8x5fm0Ty3wWN2osaNuTYBqKikKZSlV5lda3yZ2Osyv/XlUXeLvyPyOKzSUa4rPAw37mH5xHz7NhIc0tjntcclndmxfUheZeTrQ3TToyhiMWH48J6C8RC4c0YCNXH2tI87wE33MGbpieWtvxKnUrXoNs4W26K3FKbTguzMLzEbILcS/ul3BzDytwI4uN8ealLqhZHxEJRjWtJDo73EhcGOoFZI9Pk4ura2vgY4rX/rNx3nop9+IS3Pd7LhHrfGLyS+B1EZlE9pDijabNKwLc0HcDztYPTmN3xLnJ3UPA1q+qs0XkYtziJzS9XTmu7zXkkCDFzx8MpP4Kz6j1DhkRH2vIlzPOH9sEdyNFhfktNtnCEad8p2MvrZccYvjkwACc11kqO93+ODfk8Av2DFW9QUS+gwsGeAqu8xClCJ4EluHmraLWJEDuQc1yCVsc5ep6TA7ntzTmA+NEJOOEeiEoUGfudpyZMY4rcKNJgiI4QOt7a8ySunCy9RZwiMhhOFt9tty5cX2v0ZgJUsQF1roIl6N0Bq5XN4Ho9Q7ZCMcaCs5E5n+iqZFI/XDEr4tIvXDEKTl1QflKcaay6+OUrbnF8InLA7hwIanMbulIfd/fxXmdvS2h2eAAWePNq+qbwJu+w1EG9FXVuRnkY4+GVHWa9wga7Ns9V1XXZzmtJRM1od5siMgwGobhaLS3FlCuqhnDZ+SFYtjSmnPD+RTvEtjfmbqJsekh2ajctVFzBCnf672o870+N039A4F/47IOzU9tEXKzcD34GX5/CC4BeNG/wwL+NpF2fNLY83GhAppSX5Mmyonpn48z9TyPm+jriPP6mZpGNpd80Yfh8hEv8Ps7ksGXP2aZxwJd/P+X4ibrhxf73miNG26ucArOC+huv0XOHeZQ5h+Ac3BrmjbmA89325MwIjgLuEtcBibB2WDP8n68fwzJRuWujer5/RPn4dOfOtevdKsX76YuJPC++JDAEXJrVHWNiCAi7dXlRxic9eoyIM0Qa6gpaO52/Ne8m+3D1A/+FZXwJSqGz5qwXI5cJiJ/J7t//pm4l/R8Va32Nvh0K4hziTd/OW5l/Dhf74zwivZGEHQx3h83Im6qi3HREBe88Rc0DKLXmJF1rozUiPVJTSQ11xbM8Jb3UX2bVwTqEj9sLyJd/X5wIumRkHjc3LW55BGNGxL4E+9L/wTwgogsxS3ayhkpUKyhFkAqaFswcbcSbT6LHcMnB+L65z+K6x3OgI3RatOFhTg4h/qjOiphR4hcCcbXulMzJBBqJTyA6ygcinP8OBX4opnqniAiQ7V+8qcmoY3LAZ4zbX5lsbh4Qanee9Bud2WEbEfcEvVUfuHncJ5Fa0Jy9VZ+Zqn/dVyv79+4SZ+FwNWqmra37222XXGrgtelk8tw/kXUxRpaSP1YQ3dqaMWsEQ8RmZvpdwvIHYBTGiNxSuFuzWDTz6H+f+BGI5fg7ukf42zI5zWhzNgJhFoD4qN9SiBCp/hosM1Q9964iffPyFM2MXHpZhugTUi1G1lPAhTBs9T13oPhna9Le1L2MmPnEZXcQgI3SEChznOpse28UJuQz7al4Ud1l+HmZgBexk3mN3AXDLlZplgOTFHVJxtZ/93ANXF7fL69J+A6Fx/jXEnv10ZOxqbpqPxOVbONSrOVeRBu4dN73sV4e412dW3xiM/BLCLPAX/DeVz9W1UHNEPd83BeWPU8fBphAg2WGXx+K3Dmu2mqmlfPriQoglx67y/gVmoGw0H8S1W/E5KbjYu3k9WuKyIjcA9vP+pCAjeQlUACClUdJCLfAB5V1SYloPBeTeHctXntTTQXIvIYzmMnNS9zCi5+UYM8tV5ZD8H1yMH1oBfg3Dnnq+pPGlH/OzgX0ji/ew9cWOFTcC+jB3Ajw+1VdZ9c6/Zlpu6l/tSNbpvU4/Tl5rUDUkxE5FCcK/eWuIBulcDl6jMGFrjuCZomj0Qe6+iGeyfl4iKclTY/R4BzS9w+Tu8dt5R7Y+ISVV0qLoNQmFzsunFDAh+FT0Dh6/5UXAKORuOVyz44RTAW1+7xuMBprZEBqvq9wP4V4hJ3RDEMF05kA4CI3Ip7QeyJ+y0aQ6yHT1y45ME4p4LD1C9ywy0sbMo6jrjuq7GRZsqA1Yws1bpFjfsCiEhzXct07+I7mjwF+4tgFZD3eYMkKIJcvDJqJRCgSlxmogZDphyHel+o6lPZxQqSgOIYYAec2+Tp4gKM3Z/lnJbMahHZU1XHw8YHPF2c++64BWgps1EnnNvdBhFplCklh9/9Idz8zgoRuVRcwpKrVHWaqo7IdnIGvihAzzbvHZAicyNuriPbsULQAfeO+XbgWJQzQWxEZDT1g2YOpaGTS5NJgiLIpff+a2C8uNDAggune04T64/rcliIBBSr1UU2rREX8mAxbsjcWjkfuNfb3gVYApyWRvbPuDAL47zsXsAfvIL9b4HbeamqPiL5jfoK8e+lXGiWDFiFRkR2w0WB3VTqhyGpxMX3KjiaW5KhuFwb+L8G+FBVP8l3JW1eEWjMJNFe9lnfe0utQP2Jqn7ZxCbEcjlU1WtF5ECcZ89gXK7kF5pY9xRvU7wTN1n+NW61cqtEVWcAO3ilhrowGulk/yEiY3F+9+CSk6RCOBQ03hLxU57mSuzw0nEQ54c6pgAdkGLQDjcCLKMubDe456lZQmaIyBa40UfKFPUqcFETX9wfAYtSnosi0kFE+qvqB01qbIgkTBbHShIdkk95pYxT1TFNrD+Wy2GhEZH+QKWqpssJ0OKRiIBzeI8wryTC8n1wv3fQbTjn2E25UiiXzELcS+LCfIzCmTMEeC4PHZCiISL9Ap2/EtzEd9oOQ57rfgEX3iUVqPJk4CRVPbAJZU4Bdk+5kYtIO1ymv7y6w7b5EQE5JFEXkatxISge8IcuEpHdVfVXTaj/9UyLTKR+cLR6H9H4iIWpsjeuIk71IKQFrSxuBCP8lrKTH4qLOX+eiDyqqn9OCYrIn4DjcXHpg73ngisCChf1NeO91EimActUtdCjpObijyJyHm5UNhmoFJEbVPWaZqh7U1W9O7B/j4jk7J0WokwDa4lUdZ1XBvlFW0CMjkJuOL9xcDGHSlL/p5GdmZLx+6U0PZnJO7gcA3NppkQZOBPYJv6au1MXo6Q/MKfYv0kTrusVArmCcaaAl3GTdLNDsnNJkye4tW6FuJdwwf1qgPfJUwKfIn9HqVhdJ+EsAeXNdT24uZuT/Xuj1P//YhPLfAE4PLB/RFPLjNqSMCJYJi7OBIkESQAAB2xJREFU0CvAAyKymECcmgi64SYhwa3ubSp59feNybnUrSyeSl16u5U4G2ZrZTPqh/RYD/RS1dURnkDzcS+BRi+2aoEU4l6Km3q0tVAuIuXAkcBNqro+NRHeDJyBe76uxz1vr5PemSEu5+HeWzf7/Y9xa1PyShIUwRG4YGM/pS5JdIPwEp4/4HyBX6LO0+SSplSuhUmQkq3OG4AbROS3wF/VuTH+BmevbrWTxTiT3SQRSa0MPgx40Hu6hM0l1TivobCHTV4TejQnhbiXinF/FpjbcHGl3gRe8S7gzTJHgHuvnKqqS2FjDpBrcQqiUajq+8BI35lFVb/OR0PDtPnJ4rj4iaVjcDP9qYmYN1T1s+K1qmmIj7fi3Rh/h7spf6uqrTKyJGxcXZvyynhNVSMXaInIqVHHVTVtonCjdZN6hlX1kcAxAUpVNSqTW77rn66qO2U7lmOZscOqNIU2qwgaMwkrIlO0aQt+WhSpm1BE/oiLJfNgU2/MYuB7VmlR1SWZPjeSQzGfYRF5E9gnNCJ4WVW3b0KZscOqNIU2qwgag/ca+pKG8e5b5YumUG6MzY1fFR7MtJa6aVNKfeuA7COqepxE56pGmxiXx2jZFPMZFhcp9FfUxbc6Fpf/PCrvedwyZ6jqjtmONRVTBAECL5x6BF80rQlpY5ElYWMvK5zX+eXA571VdZG3DTegDdrEjQD+GQ6jzfUMi8hQ6vJj/E+b6OorIhOAn2v9sCrXap6D25kiCCAiHYAf4uITKW6+4DZVTRfPxmhGJDqv8+vaetdFGEZGRGQHXJDIlAfjUtyEdF4XhpoiCCAij+A8DFILyk4EuqrqccVrlZHCm3t2Biaq6o4iMgT4Q9BeWsgFekbLx4+CRwF9VfUccZkGB2sTIwQ0N6FV9IILmgjO3KWq+pd81pcE99Fc2E7r5xx9SVzuAaNlkDWvs6q25siZRtO5G7d2Zne/vxBns29VioC6eEmDcZ2fJ3EK4WTgjXxXZoqgPtNEZKSqTgQQkV2BpsSPN/JL3vI6G22WAap6vIicAKCq1d6FtFWhqlcAiMgrwHBVXen3Lweeznd9pgjqU4WL5/KR3+8LzE15oJjHSXFR1aP8v5f7RX9dgWeL2CSj5bHOz/WlwmoPoHWvLu+FCyuSYp0/lldMEdSnGOEgjEYQ9BQyjACX4ToHW4rIA7jFh6cVtUVN4z7gDXFZ78CFzrgn35XYZLFhGG0KcfmiR+Js6hO16TlFiorPkfItv/uKqk7Pex2mCAzDaO34l2VaVHVac7WlNWKKwDCMVo+fMwK30HAELuicAMNwoejzugCrrVFS7AYYhmE0FVXdV1X3BRbhvGxGqGoVsBPOhdTIgCkCwzDaEoNVdVZqR1XfAr5ZxPa0CsxryDCMtsRMEfk7cL/fPwmXdc3IgM0RGIbRZhCRCuB86uL3vwLcqqpriteqlo8pAsMwjIRjpiHDMNoMPkzz5UA/Au+31hpKvrmwEYFhGG0GEZmDy08+FdiQOq6qXxWtUa0AGxEYhtGWWK6qzxS7Ea0NGxEYhtFm8KkqS4H/EAg2ZyuLM2OKwDCMNkNghXE4r/V+aU4xMNOQYRhti3ERx6y3mwVTBIZhtCW+DvxfARwKvFOktrQazDRkGEabRUTaA8+p6j7FbktLxmINGYbRlukIbFHsRrR0zDRkGEabIZVW1u+WApsCVxavRa0DMw0ZhtFmEJF+gd0a4HNVrSlWe1oLpggMwzASjs0RGIZhJBxTBIZhGAnHFIGRaETk1yLytojMFJEZIrJrAesaJyIjClW+YTQW8xoyEouI7IZbcDRcVdeKSE+gXZGbZRjNjo0IjCTTG/hSVdcCqOqXqvqpiPxWRCaLyFsicoeICGzs0V8vIlNE5B0R2VlE/iMi74nIVV6mv4jMEZEHvMy/RaRjuGIR+baITBCRaSLyqIh09sevFpHZfoRybTN+F0aCMUVgJJnngS1F5F0RuUVE9vbHb1LVnVV1O6ADbtSQYp2qjgBuA54EfgRsB5wmIj28zGDgFlX9JrAC+GGwUj/yuBQ4QFWHA1OAUf78o4BtVXUYcFUBrtkwGmCKwEgsqvo1UAWcA3wBPCwipwH7isgkvzhpP2DbwGlP+b+zgLdVdZEfUcwHtvSffayqr/n/7wf2DFU9EhgKvCYiM4BTcRm1lgNrgH+IyNFAdd4u1jAyYHMERqJR1Q24iJXj/Iv/XGAYMEJVPxaRy3HBy1KkYtzXBv5P7aeep/DinPC+AC+o6gnh9ojILsD+wDHABThFZBgFxUYERmIRkcEiMjBwaEdgrv//S2+3P6YRRff1E9EAJwLjQ59PBPYQkW18OzqJyCBfX1dVHYtLt7hDI+o2jJyxEYGRZDoDN4pIN1w4gnk4M9Ey4C3gM2ByI8qdC/xIRO4CZgO3Bj9U1S+8CeohHx0T3JzBSuBJEanAjRpGNaJuw8gZCzFhGHlERPoDY/xEs2G0Csw0ZBiGkXBsRGAYhpFwbERgGIaRcEwRGIZhJBxTBIZhGAnHFIFhGEbCMUVgGIaRcP4ftons3VvFE8YAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lI51xLyS03D3"
      },
      "source": [
        "**Calculate Sentence Scores**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v0xfD_y3j_aG"
      },
      "source": [
        "sentence_scores = {}\n",
        "\n",
        "for sent in sentence_list:\n",
        "    for word in nltk.word_tokenize(sent.lower()):\n",
        "        if word in word_frequencies.keys():\n",
        "            if len(sent.split(' ')) < 30:\n",
        "                if sent not in sentence_scores.keys():\n",
        "                    sentence_scores[sent] = word_frequencies[word]\n",
        "                else:\n",
        "                    sentence_scores[sent] += word_frequencies[word]"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EznHCyJkpglL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "685874ed-30b8-4117-90db-584770fc8ad7"
      },
      "source": [
        "sentence_scores"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'A coarse division is given below.': 0.14285714285714285,\n",
              " 'A major drawback of statistical methods is that they require elaborate feature engineering.': 0.8571428571428571,\n",
              " 'Challenges in natural language processing frequently involve speech recognition, natural language understanding, and natural-language generation.': 4.25,\n",
              " 'Cognition refers to \"the mental action or process of acquiring knowledge and understanding through thought, experience, and the senses.\"': 0.6071428571428571,\n",
              " 'Cognitive linguistics is an interdisciplinary branch of linguistics, combining knowledge and research from both psychology and linguistics.': 1.3571428571428572,\n",
              " 'Cognitive science is the interdisciplinary, scientific study of the mind and its processes.': 0.4642857142857142,\n",
              " 'For example, consider some of the meanings, in English, of the word “big”.': 0.5714285714285714,\n",
              " 'However, they continue to be relevant for contexts in which statistical interpretability and transparency is required.': 0.6785714285714285,\n",
              " 'Increasingly, however, research has focused on statistical models, which make soft, probabilistic decisions based on attaching real-valued weights to each input feature.': 2.035714285714285,\n",
              " 'Many different classes of machine-learning algorithms have been applied to natural-language-processing tasks.': 0.9285714285714285,\n",
              " 'Natural language processing has its roots in the 1950s.': 2.1785714285714284,\n",
              " 'Since the early 2010s, the field has thus largely abandoned statistical methods and shifted to neural networks for machine learning.': 2.107142857142857,\n",
              " 'Since the neural turn, statistical methods in NLP research have been largely replaced by neural networks.': 1.4642857142857142,\n",
              " 'Some of the earliest-used machine learning algorithms, such as decision trees, produced systems of hard if-then rules similar to existing hand-written rules.': 2.357142857142857,\n",
              " 'Some of these tasks have direct real-world applications, while others more commonly serve as subtasks that are used to aid in solving larger tasks.': 1.3571428571428568,\n",
              " 'Starting in the late 1980s, however, there was a revolution in natural language processing with the introduction of machine learning algorithms for language processing.': 5.178571428571428,\n",
              " 'The cache language models upon which many speech recognition systems now rely are examples of such statistical models.': 2.75,\n",
              " 'The following is a list of some of the most commonly researched tasks in natural language processing.': 2.6428571428571423,\n",
              " 'The intent behind other usages, like in ”She is a big person” will remain somewhat ambiguous to a person and a cognitive NLP algorithm alike without additional information.': 1.107142857142857,\n",
              " 'The result is a computer capable of \"understanding\" the contents of documents, including the contextual nuances of the language within them.': 1.7142857142857142,\n",
              " 'The technology can then accurately extract information and insights contained in the documents as well as categorize and organize the documents themselves.': 0.7142857142857142,\n",
              " 'These algorithms take as input a large set of \"features\" that are generated from the input data.': 1.2857142857142856,\n",
              " 'These examples are not presented to be complete, but merely as indicators of the implication of the idea of Conceptual metaphor.': 0.5357142857142857,\n",
              " \"This was due to both the steady increase in computational power (see Moore's law) and the gradual lessening of the dominance of Chomskyan theories of linguistics (e.g.\": 0.7142857142857142,\n",
              " 'Though natural language processing tasks are closely intertwined, they can be subdivided into categories for convenience.': 2.6071428571428563,\n",
              " 'Up to the 1980s, most natural language processing systems were based on complex sets of hand-written rules.': 3.0357142857142856,\n",
              " 'When used as a Stative verb, as in ”Tomorrow is a big day”, a likely inference of the author’s intent it that ”big” is being used to imply ”importance”.': 1.5714285714285712,\n",
              " 'transformational grammar), whose theoretical underpinnings discouraged the sort of corpus linguistics that underlies the machine-learning approach to language processing.': 2.25}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FtT4HuGw1vK3"
      },
      "source": [
        "**Extract Output Summary**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LgdWs7xYkHw7"
      },
      "source": [
        "summary_sentences = heapq.nlargest(7, sentence_scores, key=sentence_scores.get)\n",
        "summary = ' '.join(summary_sentences)"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HD05VC7_kUis",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "af9f2ba9-9ba6-45ec-f273-4f915e265276"
      },
      "source": [
        "summary"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Starting in the late 1980s, however, there was a revolution in natural language processing with the introduction of machine learning algorithms for language processing. Challenges in natural language processing frequently involve speech recognition, natural language understanding, and natural-language generation. Up to the 1980s, most natural language processing systems were based on complex sets of hand-written rules. The cache language models upon which many speech recognition systems now rely are examples of such statistical models. The following is a list of some of the most commonly researched tasks in natural language processing. Though natural language processing tasks are closely intertwined, they can be subdivided into categories for convenience. Some of the earliest-used machine learning algorithms, such as decision trees, produced systems of hard if-then rules similar to existing hand-written rules.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    }
  ]
}