Text Summarization with N-Grams.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Summarization With N-Grams\n",
    "### By RUTUJA SHINDE \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Web Scrapping Data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bs4 as bs                                                           # BeautifulSoup\n",
    "import urllib.request\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _scrape_webpage(url):\n",
    "       \n",
    "    scraped_textdata = urllib.request.urlopen(url)\n",
    "    textdata = scraped_textdata.read()\n",
    "    parsed_textdata = bs.BeautifulSoup(textdata,'lxml')\n",
    "    paragraphs = parsed_textdata.find_all('p')\n",
    "    formated_text = \"\"\n",
    "\n",
    "    for para in paragraphs:\n",
    "        formated_text += para.text\n",
    "    \n",
    "    return formated_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "mytext = _scrape_webpage('https://en.wikipedia.org/wiki/Natural_language_processing')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8602"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#print(mytext)\n",
    "len(mytext)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Tokenizing the Words**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of words after word tokenizing:  1466\n",
      "['Natural', 'language', 'processing', '(', 'NLP', ')', 'is', 'a', 'subfield', 'of', 'linguistics', ',', 'computer', 'science', ',', 'information', 'engineering', ',', 'and', 'artificial', 'intelligence', 'concerned', 'with', 'the', 'interactions', 'between', 'computers', 'and', 'human', '(', 'natural', ')', 'languages', ',', 'in', 'particular', 'how', 'to', 'program', 'computers', 'to', 'process', 'and', 'analyze', 'large', 'amounts', 'of', 'natural', 'language', 'data']\n",
      "8602\n"
     ]
    }
   ],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "tokens = word_tokenize(mytext)\n",
    "print(\"Number of words after word tokenizing: \", len(tokens))                 #tokenizing the words\n",
    "print(tokens[:50])\n",
    "\n",
    "print(len(mytext))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Sentence Tokenization**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Natural language processing (NLP) is a subfield of linguistics, computer science, information engineering, and artificial intelligence concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data.', 'Challenges in natural language processing frequently involve speech recognition, natural language understanding, and natural language generation.', 'The history of natural language processing (NLP) generally started in the 1950s, although work can be found from earlier periods.', 'In 1950, Alan Turing published an article titled \"Computing Machinery and Intelligence\" which proposed what is now called the Turing test as a criterion of intelligence[clarification needed].', 'The Georgetown experiment in 1954 involved fully automatic translation of more than sixty Russian sentences into English.', 'The authors claimed that within three or five years, machine translation would be a solved problem.', '[2]  However, real progress was much slower, and after the ALPAC report in 1966, which found that ten-year-long research had failed to fulfill the expectations, funding for machine translation was dramatically reduced.', 'Little further research in machine translation was conducted until the late 1980s when the first statistical machine translation systems were developed.', 'Some notably successful natural language processing systems developed in the 1960s were SHRDLU, a natural language system working in restricted \"blocks worlds\" with restricted vocabularies, and ELIZA, a simulation of a Rogerian psychotherapist, written by Joseph Weizenbaum between 1964 and 1966.', 'Using almost no information about human thought or emotion, ELIZA sometimes provided a startlingly human-like interaction.', 'When the \"patient\" exceeded the very small knowledge base, ELIZA might provide a generic response, for example, responding to \"My head hurts\" with \"Why do you say your head hurts?\".', 'During the 1970s, many programmers began to write \"conceptual ontologies\", which structured real-world information into computer-understandable data.', 'Examples are MARGIE (Schank, 1975), SAM (Cullingford, 1978), PAM (Wilensky, 1978), TaleSpin (Meehan, 1976), QUALM (Lehnert, 1977), Politics (Carbonell, 1979), and Plot Units (Lehnert 1981).', 'During this time, many chatterbots were written including PARRY, Racter, and Jabberwacky.', 'Up to the 1980s, most natural language processing systems were based on complex sets of hand-written rules.', 'Starting in the late 1980s, however, there was a revolution in natural language processing with the introduction of machine learning algorithms for language processing.', \"This was due to both the steady increase in computational power (see Moore's law) and the gradual lessening of the dominance of Chomskyan theories of linguistics (e.g.\", 'transformational grammar), whose theoretical underpinnings discouraged the sort of corpus linguistics that underlies the machine-learning approach to language processing.', '[3] Some of the earliest-used machine learning algorithms, such as decision trees, produced systems of hard if-then rules similar to existing hand-written rules.', 'However, part-of-speech tagging introduced the use of hidden Markov models to natural language processing, and increasingly, research has focused on statistical models, which make soft, probabilistic decisions based on attaching real-valued weights to the features making up the input data.', 'The cache language models upon which many speech recognition systems now rely are examples of such statistical models.', 'Such models are generally more robust when given unfamiliar input, especially input that contains errors (as is very common for real-world data), and produce more reliable results when integrated into a larger system comprising multiple subtasks.', 'Many of the notable early successes occurred in the field of machine translation, due especially to work at IBM Research, where successively more complicated statistical models were developed.', 'These systems were able to take advantage of existing multilingual textual corpora that had been produced by the Parliament of Canada and the European Union as a result of laws calling for the translation of all governmental proceedings into all official languages of the corresponding systems of government.', 'However, most other systems depended on corpora specifically developed for the tasks implemented by these systems, which was (and often continues to be) a major limitation in the success of these systems.', 'As a result, a great deal of research has gone into methods of more effectively learning from limited amounts of data.', 'Recent research has increasingly focused on unsupervised and semi-supervised learning algorithms.', 'Such algorithms can learn from data that has not been hand-annotated with the desired answers or using a combination of annotated and non-annotated data.', 'Generally, this task is much more difficult than supervised learning, and typically produces less accurate results for a given amount of input data.', 'However, there is an enormous amount of non-annotated data available (including, among other things, the entire content of the World Wide Web), which can often make up for the inferior results if the algorithm used has a low enough time complexity to be practical.', 'In the 2010s, representation learning and deep neural network-style machine learning methods became widespread in natural language processing, due in part to a flurry of results showing that such techniques[4][5] can achieve state-of-the-art results in many natural language tasks, for example in language modeling,[6] parsing,[7][8] and many others.', 'Popular techniques include the use of word embeddings to capture semantic properties of words, and an increase in end-to-end learning of a higher-level task (e.g., question answering) instead of relying on a pipeline of separate intermediate tasks (e.g., part-of-speech tagging and dependency parsing).', 'In some areas, this shift has entailed substantial changes in how NLP systems are designed, such that deep neural network-based approaches may be viewed as a new paradigm distinct from statistical natural language processing.', 'For instance, the term neural machine translation (NMT) emphasizes the fact that deep learning-based approaches to machine translation directly learn sequence-to-sequence transformations, obviating the need for intermediate steps such as word alignment and language modeling that was used in statistical machine translation (SMT).', 'In the early days, many language-processing systems were designed by hand-coding a set of rules:[9][10] such as by writing grammars or devising heuristic rules for stemming.', 'Since the so-called \"statistical revolution\"[11][12] in the late 1980s and mid-1990s, much natural language processing research has relied heavily on machine learning.', 'The machine-learning paradigm calls instead for using statistical inference to automatically learn such rules through the analysis of large corpora (the plural form of corpus, is a set of documents, possibly with human or computer annotations) of typical real-world examples.', 'Many different classes of machine-learning algorithms have been applied to natural-language-processing tasks.', 'These algorithms take as input a large set of \"features\" that are generated from the input data.', 'Some of the earliest-used algorithms, such as decision trees, produced systems of hard if-then rules similar to the systems of handwritten rules that were then common.', 'Increasingly, however, research has focused on statistical models, which make soft, probabilistic decisions based on attaching real-valued weights to each input feature.', 'Such models have the advantage that they can express the relative certainty of many different possible answers rather than only one, producing more reliable results when such a model is included as a component of a larger system.', 'Systems based on machine-learning algorithms have many advantages over hand-produced rules:\\nThe following is a list of some of the most commonly researched tasks in natural language processing.', 'Some of these tasks have direct real-world applications, while others more commonly serve as subtasks that are used to aid in solving larger tasks.', 'Though natural language processing tasks are closely intertwined, they are frequently subdivided into categories for convenience.', 'A coarse division is given below.', 'The first published work by an artificial intelligence was published in 2018, 1 the Road, marketed as a novel, contains sixty million words.']\n"
     ]
    }
   ],
   "source": [
    "from nltk.tokenize import sent_tokenize\n",
    "sent_tokens = sent_tokenize(mytext)\n",
    "print(sent_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Punctuation Removal**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of words after word tokenizing with removing punctuation:  1295\n",
      "['natural', 'language', 'processing', 'nlp', 'is', 'a', 'subfield', 'of', 'linguistics', 'computer', 'science', 'information', 'engineering', 'and', 'artificial', 'intelligence', 'concerned', 'with', 'the', 'interactions', 'between', 'computers', 'and', 'human', 'natural', 'languages', 'in', 'particular', 'how', 'to', 'program', 'computers', 'to', 'process', 'and', 'analyze', 'large', 'amounts', 'of', 'natural', 'language', 'data', 'challenges', 'in', 'natural', 'language', 'processing', 'frequently', 'involve', 'speech']\n",
      "8602\n",
      "1295\n"
     ]
    }
   ],
   "source": [
    "from nltk.tokenize import RegexpTokenizer\n",
    "tokenizer = RegexpTokenizer(r'\\w+')\n",
    "regexp_tokens = tokenizer.tokenize(mytext.lower())                                   #punctuation removal\n",
    "print(\"Number of words after word tokenizing with removing punctuation: \", len(regexp_tokens))\n",
    "print(regexp_tokens[0:50])\n",
    "\n",
    "print(len(mytext))\n",
    "print(len(regexp_tokens))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Convert to Lower case**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['natural', 'language', 'processing', 'nlp', 'is', 'a', 'subfield', 'of', 'linguistics', 'computer', 'science', 'information', 'engineering', 'and', 'artificial', 'intelligence', 'concerned', 'with', 'the', 'interactions', 'between', 'computers', 'and', 'human', 'natural', 'languages', 'in', 'particular', 'how', 'to', 'program', 'computers', 'to', 'process', 'and', 'analyze', 'large', 'amounts', 'of', 'natural', 'language', 'data', 'challenges', 'in', 'natural', 'language', 'processing', 'frequently', 'involve', 'speech', 'recognition', 'natural', 'language', 'understanding', 'and', 'natural', 'language', 'generation', 'the', 'history', 'of', 'natural', 'language', 'processing', 'nlp', 'generally', 'started', 'in', 'the', '1950s', 'although', 'work', 'can', 'be', 'found', 'from', 'earlier', 'periods', 'in', '1950', 'alan', 'turing', 'published', 'an', 'article', 'titled', 'computing', 'machinery', 'and', 'intelligence', 'which', 'proposed', 'what', 'is', 'now', 'called', 'the', 'turing', 'test', 'as', 'a', 'criterion', 'of', 'intelligence', 'clarification', 'needed', 'the', 'georgetown', 'experiment', 'in', '1954', 'involved', 'fully', 'automatic', 'translation', 'of', 'more', 'than', 'sixty', 'russian', 'sentences', 'into', 'english', 'the', 'authors', 'claimed', 'that', 'within', 'three', 'or', 'five', 'years', 'machine', 'translation', 'would', 'be', 'a', 'solved', 'problem', '2', 'however', 'real', 'progress', 'was', 'much', 'slower', 'and', 'after', 'the', 'alpac', 'report', 'in', '1966', 'which', 'found', 'that', 'ten', 'year', 'long', 'research', 'had', 'failed', 'to', 'fulfill', 'the', 'expectations', 'funding', 'for', 'machine', 'translation', 'was', 'dramatically', 'reduced', 'little', 'further', 'research', 'in', 'machine', 'translation', 'was', 'conducted', 'until', 'the', 'late', '1980s', 'when', 'the', 'first', 'statistical', 'machine', 'translation', 'systems', 'were', 'developed', 'some', 'notably', 'successful', 'natural', 'language', 'processing', 'systems', 'developed', 'in', 'the', '1960s', 'were', 'shrdlu', 'a', 'natural', 'language', 'system', 'working', 'in', 'restricted', 'blocks', 'worlds', 'with', 'restricted', 'vocabularies', 'and', 'eliza', 'a', 'simulation', 'of', 'a', 'rogerian', 'psychotherapist', 'written', 'by', 'joseph', 'weizenbaum', 'between', '1964', 'and', '1966', 'using', 'almost', 'no', 'information', 'about', 'human', 'thought', 'or', 'emotion', 'eliza', 'sometimes', 'provided', 'a', 'startlingly', 'human', 'like', 'interaction', 'when', 'the', 'patient', 'exceeded', 'the', 'very', 'small', 'knowledge', 'base', 'eliza', 'might', 'provide', 'a', 'generic', 'response', 'for', 'example', 'responding', 'to', 'my', 'head', 'hurts', 'with', 'why', 'do', 'you', 'say', 'your', 'head', 'hurts', 'during', 'the', '1970s', 'many', 'programmers', 'began', 'to', 'write', 'conceptual', 'ontologies', 'which', 'structured', 'real', 'world', 'information', 'into', 'computer', 'understandable', 'data', 'examples', 'are', 'margie', 'schank', '1975', 'sam', 'cullingford', '1978', 'pam', 'wilensky', '1978', 'talespin', 'meehan', '1976', 'qualm', 'lehnert', '1977', 'politics', 'carbonell', '1979', 'and', 'plot', 'units', 'lehnert', '1981', 'during', 'this', 'time', 'many', 'chatterbots', 'were', 'written', 'including', 'parry', 'racter', 'and', 'jabberwacky', 'up', 'to', 'the', '1980s', 'most', 'natural', 'language', 'processing', 'systems', 'were', 'based', 'on', 'complex', 'sets', 'of', 'hand', 'written', 'rules', 'starting', 'in', 'the', 'late', '1980s', 'however', 'there', 'was', 'a', 'revolution', 'in', 'natural', 'language', 'processing', 'with', 'the', 'introduction', 'of', 'machine', 'learning', 'algorithms', 'for', 'language', 'processing', 'this', 'was', 'due', 'to', 'both', 'the', 'steady', 'increase', 'in', 'computational', 'power', 'see', 'moore', 's', 'law', 'and', 'the', 'gradual', 'lessening', 'of', 'the', 'dominance', 'of', 'chomskyan', 'theories', 'of', 'linguistics', 'e', 'g', 'transformational', 'grammar', 'whose', 'theoretical', 'underpinnings', 'discouraged', 'the', 'sort', 'of', 'corpus', 'linguistics', 'that', 'underlies', 'the', 'machine', 'learning', 'approach', 'to', 'language', 'processing', '3', 'some', 'of', 'the', 'earliest', 'used', 'machine', 'learning', 'algorithms', 'such', 'as', 'decision', 'trees', 'produced', 'systems', 'of', 'hard', 'if', 'then', 'rules', 'similar', 'to', 'existing', 'hand', 'written', 'rules', 'however', 'part', 'of', 'speech', 'tagging', 'introduced', 'the', 'use', 'of', 'hidden', 'markov', 'models', 'to', 'natural', 'language', 'processing', 'and', 'increasingly', 'research', 'has', 'focused', 'on', 'statistical', 'models', 'which', 'make', 'soft', 'probabilistic', 'decisions', 'based', 'on', 'attaching', 'real', 'valued', 'weights', 'to', 'the', 'features', 'making', 'up', 'the', 'input', 'data', 'the', 'cache', 'language', 'models', 'upon', 'which', 'many', 'speech', 'recognition', 'systems', 'now', 'rely', 'are', 'examples', 'of', 'such', 'statistical', 'models', 'such', 'models', 'are', 'generally', 'more', 'robust', 'when', 'given', 'unfamiliar', 'input', 'especially', 'input', 'that', 'contains', 'errors', 'as', 'is', 'very', 'common', 'for', 'real', 'world', 'data', 'and', 'produce', 'more', 'reliable', 'results', 'when', 'integrated', 'into', 'a', 'larger', 'system', 'comprising', 'multiple', 'subtasks', 'many', 'of', 'the', 'notable', 'early', 'successes', 'occurred', 'in', 'the', 'field', 'of', 'machine', 'translation', 'due', 'especially', 'to', 'work', 'at', 'ibm', 'research', 'where', 'successively', 'more', 'complicated', 'statistical', 'models', 'were', 'developed', 'these', 'systems', 'were', 'able', 'to', 'take', 'advantage', 'of', 'existing', 'multilingual', 'textual', 'corpora', 'that', 'had', 'been', 'produced', 'by', 'the', 'parliament', 'of', 'canada', 'and', 'the', 'european', 'union', 'as', 'a', 'result', 'of', 'laws', 'calling', 'for', 'the', 'translation', 'of', 'all', 'governmental', 'proceedings', 'into', 'all', 'official', 'languages', 'of', 'the', 'corresponding', 'systems', 'of', 'government', 'however', 'most', 'other', 'systems', 'depended', 'on', 'corpora', 'specifically', 'developed', 'for', 'the', 'tasks', 'implemented', 'by', 'these', 'systems', 'which', 'was', 'and', 'often', 'continues', 'to', 'be', 'a', 'major', 'limitation', 'in', 'the', 'success', 'of', 'these', 'systems', 'as', 'a', 'result', 'a', 'great', 'deal', 'of', 'research', 'has', 'gone', 'into', 'methods', 'of', 'more', 'effectively', 'learning', 'from', 'limited', 'amounts', 'of', 'data', 'recent', 'research', 'has', 'increasingly', 'focused', 'on', 'unsupervised', 'and', 'semi', 'supervised', 'learning', 'algorithms', 'such', 'algorithms', 'can', 'learn', 'from', 'data', 'that', 'has', 'not', 'been', 'hand', 'annotated', 'with', 'the', 'desired', 'answers', 'or', 'using', 'a', 'combination', 'of', 'annotated', 'and', 'non', 'annotated', 'data', 'generally', 'this', 'task', 'is', 'much', 'more', 'difficult', 'than', 'supervised', 'learning', 'and', 'typically', 'produces', 'less', 'accurate', 'results', 'for', 'a', 'given', 'amount', 'of', 'input', 'data', 'however', 'there', 'is', 'an', 'enormous', 'amount', 'of', 'non', 'annotated', 'data', 'available', 'including', 'among', 'other', 'things', 'the', 'entire', 'content', 'of', 'the', 'world', 'wide', 'web', 'which', 'can', 'often', 'make', 'up', 'for', 'the', 'inferior', 'results', 'if', 'the', 'algorithm', 'used', 'has', 'a', 'low', 'enough', 'time', 'complexity', 'to', 'be', 'practical', 'in', 'the', '2010s', 'representation', 'learning', 'and', 'deep', 'neural', 'network', 'style', 'machine', 'learning', 'methods', 'became', 'widespread', 'in', 'natural', 'language', 'processing', 'due', 'in', 'part', 'to', 'a', 'flurry', 'of', 'results', 'showing', 'that', 'such', 'techniques', '4', '5', 'can', 'achieve', 'state', 'of', 'the', 'art', 'results', 'in', 'many', 'natural', 'language', 'tasks', 'for', 'example', 'in', 'language', 'modeling', '6', 'parsing', '7', '8', 'and', 'many', 'others', 'popular', 'techniques', 'include', 'the', 'use', 'of', 'word', 'embeddings', 'to', 'capture', 'semantic', 'properties', 'of', 'words', 'and', 'an', 'increase', 'in', 'end', 'to', 'end', 'learning', 'of', 'a', 'higher', 'level', 'task', 'e', 'g', 'question', 'answering', 'instead', 'of', 'relying', 'on', 'a', 'pipeline', 'of', 'separate', 'intermediate', 'tasks', 'e', 'g', 'part', 'of', 'speech', 'tagging', 'and', 'dependency', 'parsing', 'in', 'some', 'areas', 'this', 'shift', 'has', 'entailed', 'substantial', 'changes', 'in', 'how', 'nlp', 'systems', 'are', 'designed', 'such', 'that', 'deep', 'neural', 'network', 'based', 'approaches', 'may', 'be', 'viewed', 'as', 'a', 'new', 'paradigm', 'distinct', 'from', 'statistical', 'natural', 'language', 'processing', 'for', 'instance', 'the', 'term', 'neural', 'machine', 'translation', 'nmt', 'emphasizes', 'the', 'fact', 'that', 'deep', 'learning', 'based', 'approaches', 'to', 'machine', 'translation', 'directly', 'learn', 'sequence', 'to', 'sequence', 'transformations', 'obviating', 'the', 'need', 'for', 'intermediate', 'steps', 'such', 'as', 'word', 'alignment', 'and', 'language', 'modeling', 'that', 'was', 'used', 'in', 'statistical', 'machine', 'translation', 'smt', 'in', 'the', 'early', 'days', 'many', 'language', 'processing', 'systems', 'were', 'designed', 'by', 'hand', 'coding', 'a', 'set', 'of', 'rules', '9', '10', 'such', 'as', 'by', 'writing', 'grammars', 'or', 'devising', 'heuristic', 'rules', 'for', 'stemming', 'since', 'the', 'so', 'called', 'statistical', 'revolution', '11', '12', 'in', 'the', 'late', '1980s', 'and', 'mid', '1990s', 'much', 'natural', 'language', 'processing', 'research', 'has', 'relied', 'heavily', 'on', 'machine', 'learning', 'the', 'machine', 'learning', 'paradigm', 'calls', 'instead', 'for', 'using', 'statistical', 'inference', 'to', 'automatically', 'learn', 'such', 'rules', 'through', 'the', 'analysis', 'of', 'large', 'corpora', 'the', 'plural', 'form', 'of', 'corpus', 'is', 'a', 'set', 'of', 'documents', 'possibly', 'with', 'human', 'or', 'computer', 'annotations', 'of', 'typical', 'real', 'world', 'examples', 'many', 'different', 'classes', 'of', 'machine', 'learning', 'algorithms', 'have', 'been', 'applied', 'to', 'natural', 'language', 'processing', 'tasks', 'these', 'algorithms', 'take', 'as', 'input', 'a', 'large', 'set', 'of', 'features', 'that', 'are', 'generated', 'from', 'the', 'input', 'data', 'some', 'of', 'the', 'earliest', 'used', 'algorithms', 'such', 'as', 'decision', 'trees', 'produced', 'systems', 'of', 'hard', 'if', 'then', 'rules', 'similar', 'to', 'the', 'systems', 'of', 'handwritten', 'rules', 'that', 'were', 'then', 'common', 'increasingly', 'however', 'research', 'has', 'focused', 'on', 'statistical', 'models', 'which', 'make', 'soft', 'probabilistic', 'decisions', 'based', 'on', 'attaching', 'real', 'valued', 'weights', 'to', 'each', 'input', 'feature', 'such', 'models', 'have', 'the', 'advantage', 'that', 'they', 'can', 'express', 'the', 'relative', 'certainty', 'of', 'many', 'different', 'possible', 'answers', 'rather', 'than', 'only', 'one', 'producing', 'more', 'reliable', 'results', 'when', 'such', 'a', 'model', 'is', 'included', 'as', 'a', 'component', 'of', 'a', 'larger', 'system', 'systems', 'based', 'on', 'machine', 'learning', 'algorithms', 'have', 'many', 'advantages', 'over', 'hand', 'produced', 'rules', 'the', 'following', 'is', 'a', 'list', 'of', 'some', 'of', 'the', 'most', 'commonly', 'researched', 'tasks', 'in', 'natural', 'language', 'processing', 'some', 'of', 'these', 'tasks', 'have', 'direct', 'real', 'world', 'applications', 'while', 'others', 'more', 'commonly', 'serve', 'as', 'subtasks', 'that', 'are', 'used', 'to', 'aid', 'in', 'solving', 'larger', 'tasks', 'though', 'natural', 'language', 'processing', 'tasks', 'are', 'closely', 'intertwined', 'they', 'are', 'frequently', 'subdivided', 'into', 'categories', 'for', 'convenience', 'a', 'coarse', 'division', 'is', 'given', 'below', 'the', 'first', 'published', 'work', 'by', 'an', 'artificial', 'intelligence', 'was', 'published', 'in', '2018', '1', 'the', 'road', 'marketed', 'as', 'a', 'novel', 'contains', 'sixty', 'million', 'words']\n"
     ]
    }
   ],
   "source": [
    "from nltk.tokenize import RegexpTokenizer\n",
    "regexp_tokens = tokenizer.tokenize(mytext.lower())\n",
    "print(regexp_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\rutuj\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\rutuj\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Stopword Removal**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of words without stop words:  824\n",
      "['natural', 'language', 'processing', 'nlp', 'subfield', 'linguistics', 'computer', 'science', 'information', 'engineering', 'artificial', 'intelligence', 'concerned', 'interactions', 'computers', 'human', 'natural', 'languages', 'particular', 'program', 'computers', 'process', 'analyze', 'large', 'amounts', 'natural', 'language', 'data', 'challenges', 'natural', 'language', 'processing', 'frequently', 'involve', 'speech', 'recognition', 'natural', 'language', 'understanding', 'natural', 'language', 'generation', 'history', 'natural', 'language', 'processing', 'nlp', 'generally', 'started', '1950s']\n",
      "8602\n"
     ]
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "stopwords_tokens = [token for token in regexp_tokens if token not in stopwords.words('english')]\n",
    "print(\"# of words without stop words: \", len(stopwords_tokens))\n",
    "print(stopwords_tokens[0:50])                                                               #stopword removal\n",
    "print(len(mytext))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Digit Removal**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8499\n"
     ]
    }
   ],
   "source": [
    "a = re.sub( r'\\d+', '', mytext)\n",
    "#print(a)\n",
    "#print(len(mytext))                               # digit removal\n",
    "print(len(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Generate Unigrams**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['natural', 'language', 'processing', '(', 'nlp', ')', 'is', 'a', 'subfield', 'of', 'linguistics', ',', 'computer', 'science', ',', 'information', 'engineering', ',', 'and', 'artificial', 'intelligence', 'concerned', 'with', 'the', 'interactions', 'between', 'computers', 'and', 'human', '(', 'natural', ')', 'languages', ',', 'in', 'particular', 'how', 'to', 'program', 'computers', 'to', 'process', 'and', 'analyze', 'large', 'amounts', 'of', 'natural', 'language', 'data', '.', 'challenges', 'in', 'natural', 'language', 'processing', 'frequently', 'involve', 'speech', 'recognition', ',', 'natural', 'language', 'understanding', ',', 'and', 'natural', 'language', 'generation', '.', 'the', 'history', 'of', 'natural', 'language', 'processing', '(', 'nlp', ')', 'generally', 'started', 'in', 'the', 's', ',', 'although', 'work', 'can', 'be', 'found', 'from', 'earlier', 'periods', '.', 'in', ',', 'alan', 'turing', 'published', 'an', 'article', 'titled', '``', 'computing', 'machinery', 'and', 'intelligence', \"''\", 'which', 'proposed', 'what', 'is', 'now', 'called', 'the', 'turing', 'test', 'as', 'a', 'criterion', 'of', 'intelligence', '[', 'clarification', 'needed', ']', '.', 'the', 'georgetown', 'experiment', 'in', 'involved', 'fully', 'automatic', 'translation', 'of', 'more', 'than', 'sixty', 'russian', 'sentences', 'into', 'english', '.', 'the', 'authors', 'claimed', 'that', 'within', 'three', 'or', 'five', 'years', ',', 'machine', 'translation', 'would', 'be', 'a', 'solved', 'problem', '.', '[', ']', 'however', ',', 'real', 'progress', 'was', 'much', 'slower', ',', 'and', 'after', 'the', 'alpac', 'report', 'in', ',', 'which', 'found', 'that', 'ten-year-long', 'research', 'had', 'failed', 'to', 'fulfill', 'the', 'expectations', ',', 'funding', 'for', 'machine', 'translation', 'was', 'dramatically', 'reduced', '.', 'little', 'further', 'research', 'in', 'machine', 'translation', 'was', 'conducted', 'until', 'the', 'late', 's', 'when', 'the', 'first', 'statistical', 'machine', 'translation', 'systems', 'were', 'developed', '.', 'some', 'notably', 'successful', 'natural', 'language', 'processing', 'systems', 'developed', 'in', 'the', 's', 'were', 'shrdlu', ',', 'a', 'natural', 'language', 'system', 'working', 'in', 'restricted', '``', 'blocks', 'worlds', \"''\", 'with', 'restricted', 'vocabularies', ',', 'and', 'eliza', ',', 'a', 'simulation', 'of', 'a', 'rogerian', 'psychotherapist', ',', 'written', 'by', 'joseph', 'weizenbaum', 'between', 'and', '.', 'using', 'almost', 'no', 'information', 'about', 'human', 'thought', 'or', 'emotion', ',', 'eliza', 'sometimes', 'provided', 'a', 'startlingly', 'human-like', 'interaction', '.', 'when', 'the', '``', 'patient', \"''\", 'exceeded', 'the', 'very', 'small', 'knowledge', 'base', ',', 'eliza', 'might', 'provide', 'a', 'generic', 'response', ',', 'for', 'example', ',', 'responding', 'to', '``', 'my', 'head', 'hurts', \"''\", 'with', '``', 'why', 'do', 'you', 'say', 'your', 'head', 'hurts', '?', \"''\", '.', 'during', 'the', 's', ',', 'many', 'programmers', 'began', 'to', 'write', '``', 'conceptual', 'ontologies', \"''\", ',', 'which', 'structured', 'real-world', 'information', 'into', 'computer-understandable', 'data', '.', 'examples', 'are', 'margie', '(', 'schank', ',', ')', ',', 'sam', '(', 'cullingford', ',', ')', ',', 'pam', '(', 'wilensky', ',', ')', ',', 'talespin', '(', 'meehan', ',', ')', ',', 'qualm', '(', 'lehnert', ',', ')', ',', 'politics', '(', 'carbonell', ',', ')', ',', 'and', 'plot', 'units', '(', 'lehnert', ')', '.', 'during', 'this', 'time', ',', 'many', 'chatterbots', 'were', 'written', 'including', 'parry', ',', 'racter', ',', 'and', 'jabberwacky', '.', 'up', 'to', 'the', 's', ',', 'most', 'natural', 'language', 'processing', 'systems', 'were', 'based', 'on', 'complex', 'sets', 'of', 'hand-written', 'rules', '.', 'starting', 'in', 'the', 'late', 's', ',', 'however', ',', 'there', 'was', 'a', 'revolution', 'in', 'natural', 'language', 'processing', 'with', 'the', 'introduction', 'of', 'machine', 'learning', 'algorithms', 'for', 'language', 'processing', '.', 'this', 'was', 'due', 'to', 'both', 'the', 'steady', 'increase', 'in', 'computational', 'power', '(', 'see', 'moore', \"'s\", 'law', ')', 'and', 'the', 'gradual', 'lessening', 'of', 'the', 'dominance', 'of', 'chomskyan', 'theories', 'of', 'linguistics', '(', 'e.g', '.', 'transformational', 'grammar', ')', ',', 'whose', 'theoretical', 'underpinnings', 'discouraged', 'the', 'sort', 'of', 'corpus', 'linguistics', 'that', 'underlies', 'the', 'machine-learning', 'approach', 'to', 'language', 'processing', '.', '[', ']', 'some', 'of', 'the', 'earliest-used', 'machine', 'learning', 'algorithms', ',', 'such', 'as', 'decision', 'trees', ',', 'produced', 'systems', 'of', 'hard', 'if-then', 'rules', 'similar', 'to', 'existing', 'hand-written', 'rules', '.', 'however', ',', 'part-of-speech', 'tagging', 'introduced', 'the', 'use', 'of', 'hidden', 'markov', 'models', 'to', 'natural', 'language', 'processing', ',', 'and', 'increasingly', ',', 'research', 'has', 'focused', 'on', 'statistical', 'models', ',', 'which', 'make', 'soft', ',', 'probabilistic', 'decisions', 'based', 'on', 'attaching', 'real-valued', 'weights', 'to', 'the', 'features', 'making', 'up', 'the', 'input', 'data', '.', 'the', 'cache', 'language', 'models', 'upon', 'which', 'many', 'speech', 'recognition', 'systems', 'now', 'rely', 'are', 'examples', 'of', 'such', 'statistical', 'models', '.', 'such', 'models', 'are', 'generally', 'more', 'robust', 'when', 'given', 'unfamiliar', 'input', ',', 'especially', 'input', 'that', 'contains', 'errors', '(', 'as', 'is', 'very', 'common', 'for', 'real-world', 'data', ')', ',', 'and', 'produce', 'more', 'reliable', 'results', 'when', 'integrated', 'into', 'a', 'larger', 'system', 'comprising', 'multiple', 'subtasks', '.', 'many', 'of', 'the', 'notable', 'early', 'successes', 'occurred', 'in', 'the', 'field', 'of', 'machine', 'translation', ',', 'due', 'especially', 'to', 'work', 'at', 'ibm', 'research', ',', 'where', 'successively', 'more', 'complicated', 'statistical', 'models', 'were', 'developed', '.', 'these', 'systems', 'were', 'able', 'to', 'take', 'advantage', 'of', 'existing', 'multilingual', 'textual', 'corpora', 'that', 'had', 'been', 'produced', 'by', 'the', 'parliament', 'of', 'canada', 'and', 'the', 'european', 'union', 'as', 'a', 'result', 'of', 'laws', 'calling', 'for', 'the', 'translation', 'of', 'all', 'governmental', 'proceedings', 'into', 'all', 'official', 'languages', 'of', 'the', 'corresponding', 'systems', 'of', 'government', '.', 'however', ',', 'most', 'other', 'systems', 'depended', 'on', 'corpora', 'specifically', 'developed', 'for', 'the', 'tasks', 'implemented', 'by', 'these', 'systems', ',', 'which', 'was', '(', 'and', 'often', 'continues', 'to', 'be', ')', 'a', 'major', 'limitation', 'in', 'the', 'success', 'of', 'these', 'systems', '.', 'as', 'a', 'result', ',', 'a', 'great', 'deal', 'of', 'research', 'has', 'gone', 'into', 'methods', 'of', 'more', 'effectively', 'learning', 'from', 'limited', 'amounts', 'of', 'data', '.', 'recent', 'research', 'has', 'increasingly', 'focused', 'on', 'unsupervised', 'and', 'semi-supervised', 'learning', 'algorithms', '.', 'such', 'algorithms', 'can', 'learn', 'from', 'data', 'that', 'has', 'not', 'been', 'hand-annotated', 'with', 'the', 'desired', 'answers', 'or', 'using', 'a', 'combination', 'of', 'annotated', 'and', 'non-annotated', 'data', '.', 'generally', ',', 'this', 'task', 'is', 'much', 'more', 'difficult', 'than', 'supervised', 'learning', ',', 'and', 'typically', 'produces', 'less', 'accurate', 'results', 'for', 'a', 'given', 'amount', 'of', 'input', 'data', '.', 'however', ',', 'there', 'is', 'an', 'enormous', 'amount', 'of', 'non-annotated', 'data', 'available', '(', 'including', ',', 'among', 'other', 'things', ',', 'the', 'entire', 'content', 'of', 'the', 'world', 'wide', 'web', ')', ',', 'which', 'can', 'often', 'make', 'up', 'for', 'the', 'inferior', 'results', 'if', 'the', 'algorithm', 'used', 'has', 'a', 'low', 'enough', 'time', 'complexity', 'to', 'be', 'practical', '.', 'in', 'the', 's', ',', 'representation', 'learning', 'and', 'deep', 'neural', 'network-style', 'machine', 'learning', 'methods', 'became', 'widespread', 'in', 'natural', 'language', 'processing', ',', 'due', 'in', 'part', 'to', 'a', 'flurry', 'of', 'results', 'showing', 'that', 'such', 'techniques', '[', ']', '[', ']', 'can', 'achieve', 'state-of-the-art', 'results', 'in', 'many', 'natural', 'language', 'tasks', ',', 'for', 'example', 'in', 'language', 'modeling', ',', '[', ']', 'parsing', ',', '[', ']', '[', ']', 'and', 'many', 'others', '.', 'popular', 'techniques', 'include', 'the', 'use', 'of', 'word', 'embeddings', 'to', 'capture', 'semantic', 'properties', 'of', 'words', ',', 'and', 'an', 'increase', 'in', 'end-to-end', 'learning', 'of', 'a', 'higher-level', 'task', '(', 'e.g.', ',', 'question', 'answering', ')', 'instead', 'of', 'relying', 'on', 'a', 'pipeline', 'of', 'separate', 'intermediate', 'tasks', '(', 'e.g.', ',', 'part-of-speech', 'tagging', 'and', 'dependency', 'parsing', ')', '.', 'in', 'some', 'areas', ',', 'this', 'shift', 'has', 'entailed', 'substantial', 'changes', 'in', 'how', 'nlp', 'systems', 'are', 'designed', ',', 'such', 'that', 'deep', 'neural', 'network-based', 'approaches', 'may', 'be', 'viewed', 'as', 'a', 'new', 'paradigm', 'distinct', 'from', 'statistical', 'natural', 'language', 'processing', '.', 'for', 'instance', ',', 'the', 'term', 'neural', 'machine', 'translation', '(', 'nmt', ')', 'emphasizes', 'the', 'fact', 'that', 'deep', 'learning-based', 'approaches', 'to', 'machine', 'translation', 'directly', 'learn', 'sequence-to-sequence', 'transformations', ',', 'obviating', 'the', 'need', 'for', 'intermediate', 'steps', 'such', 'as', 'word', 'alignment', 'and', 'language', 'modeling', 'that', 'was', 'used', 'in', 'statistical', 'machine', 'translation', '(', 'smt', ')', '.', 'in', 'the', 'early', 'days', ',', 'many', 'language-processing', 'systems', 'were', 'designed', 'by', 'hand-coding', 'a', 'set', 'of', 'rules', ':', '[', ']', '[', ']', 'such', 'as', 'by', 'writing', 'grammars', 'or', 'devising', 'heuristic', 'rules', 'for', 'stemming', '.', 'since', 'the', 'so-called', '``', 'statistical', 'revolution', \"''\", '[', ']', '[', ']', 'in', 'the', 'late', 's', 'and', 'mid-s', ',', 'much', 'natural', 'language', 'processing', 'research', 'has', 'relied', 'heavily', 'on', 'machine', 'learning', '.', 'the', 'machine-learning', 'paradigm', 'calls', 'instead', 'for', 'using', 'statistical', 'inference', 'to', 'automatically', 'learn', 'such', 'rules', 'through', 'the', 'analysis', 'of', 'large', 'corpora', '(', 'the', 'plural', 'form', 'of', 'corpus', ',', 'is', 'a', 'set', 'of', 'documents', ',', 'possibly', 'with', 'human', 'or', 'computer', 'annotations', ')', 'of', 'typical', 'real-world', 'examples', '.', 'many', 'different', 'classes', 'of', 'machine-learning', 'algorithms', 'have', 'been', 'applied', 'to', 'natural-language-processing', 'tasks', '.', 'these', 'algorithms', 'take', 'as', 'input', 'a', 'large', 'set', 'of', '``', 'features', \"''\", 'that', 'are', 'generated', 'from', 'the', 'input', 'data', '.', 'some', 'of', 'the', 'earliest-used', 'algorithms', ',', 'such', 'as', 'decision', 'trees', ',', 'produced', 'systems', 'of', 'hard', 'if-then', 'rules', 'similar', 'to', 'the', 'systems', 'of', 'handwritten', 'rules', 'that', 'were', 'then', 'common', '.', 'increasingly', ',', 'however', ',', 'research', 'has', 'focused', 'on', 'statistical', 'models', ',', 'which', 'make', 'soft', ',', 'probabilistic', 'decisions', 'based', 'on', 'attaching', 'real-valued', 'weights', 'to', 'each', 'input', 'feature', '.', 'such', 'models', 'have', 'the', 'advantage', 'that', 'they', 'can', 'express', 'the', 'relative', 'certainty', 'of', 'many', 'different', 'possible', 'answers', 'rather', 'than', 'only', 'one', ',', 'producing', 'more', 'reliable', 'results', 'when', 'such', 'a', 'model', 'is', 'included', 'as', 'a', 'component', 'of', 'a', 'larger', 'system', '.', 'systems', 'based', 'on', 'machine-learning', 'algorithms', 'have', 'many', 'advantages', 'over', 'hand-produced', 'rules', ':', 'the', 'following', 'is', 'a', 'list', 'of', 'some', 'of', 'the', 'most', 'commonly', 'researched', 'tasks', 'in', 'natural', 'language', 'processing', '.', 'some', 'of', 'these', 'tasks', 'have', 'direct', 'real-world', 'applications', ',', 'while', 'others', 'more', 'commonly', 'serve', 'as', 'subtasks', 'that', 'are', 'used', 'to', 'aid', 'in', 'solving', 'larger', 'tasks', '.', 'though', 'natural', 'language', 'processing', 'tasks', 'are', 'closely', 'intertwined', ',', 'they', 'are', 'frequently', 'subdivided', 'into', 'categories', 'for', 'convenience', '.', 'a', 'coarse', 'division', 'is', 'given', 'below', '.', 'the', 'first', 'published', 'work', 'by', 'an', 'artificial', 'intelligence', 'was', 'published', 'in', ',', 'the', 'road', ',', 'marketed', 'as', 'a', 'novel', ',', 'contains', 'sixty', 'million', 'words', '.']\n"
     ]
    }
   ],
   "source": [
    "#function to generate unigrams from sentences\n",
    "from nltk.util import ngrams\n",
    "def extract_ngrams(data,num):\n",
    "    n_grams = ngrams(nltk.word_tokenize(data.lower()), num)\n",
    "    return [' '.join(grams) for grams in n_grams]\n",
    "gram_num = 1\n",
    "n_grams = extract_ngrams(a, gram_num)\n",
    "print(n_grams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Generate Bi Grams**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['natural language', 'language processing', 'processing (', '( nlp', 'nlp )', ') is', 'is a', 'a subfield', 'subfield of', 'of linguistics', 'linguistics ,', ', computer', 'computer science', 'science ,', ', information', 'information engineering', 'engineering ,', ', and', 'and artificial', 'artificial intelligence', 'intelligence concerned', 'concerned with', 'with the', 'the interactions', 'interactions between', 'between computers', 'computers and', 'and human', 'human (', '( natural', 'natural )', ') languages', 'languages ,', ', in', 'in particular', 'particular how', 'how to', 'to program', 'program computers', 'computers to', 'to process', 'process and', 'and analyze', 'analyze large', 'large amounts', 'amounts of', 'of natural', 'natural language', 'language data', 'data .', '. challenges', 'challenges in', 'in natural', 'natural language', 'language processing', 'processing frequently', 'frequently involve', 'involve speech', 'speech recognition', 'recognition ,', ', natural', 'natural language', 'language understanding', 'understanding ,', ', and', 'and natural', 'natural language', 'language generation', 'generation .', '. the', 'the history', 'history of', 'of natural', 'natural language', 'language processing', 'processing (', '( nlp', 'nlp )', ') generally', 'generally started', 'started in', 'in the', 'the s', 's ,', ', although', 'although work', 'work can', 'can be', 'be found', 'found from', 'from earlier', 'earlier periods', 'periods .', '. in', 'in ,', ', alan', 'alan turing', 'turing published', 'published an', 'an article', 'article titled', 'titled ``', '`` computing', 'computing machinery', 'machinery and', 'and intelligence', \"intelligence ''\", \"'' which\", 'which proposed', 'proposed what', 'what is', 'is now', 'now called', 'called the', 'the turing', 'turing test', 'test as', 'as a', 'a criterion', 'criterion of', 'of intelligence', 'intelligence [', '[ clarification', 'clarification needed', 'needed ]', '] .', '. the', 'the georgetown', 'georgetown experiment', 'experiment in', 'in involved', 'involved fully', 'fully automatic', 'automatic translation', 'translation of', 'of more', 'more than', 'than sixty', 'sixty russian', 'russian sentences', 'sentences into', 'into english', 'english .', '. the', 'the authors', 'authors claimed', 'claimed that', 'that within', 'within three', 'three or', 'or five', 'five years', 'years ,', ', machine', 'machine translation', 'translation would', 'would be', 'be a', 'a solved', 'solved problem', 'problem .', '. [', '[ ]', '] however', 'however ,', ', real', 'real progress', 'progress was', 'was much', 'much slower', 'slower ,', ', and', 'and after', 'after the', 'the alpac', 'alpac report', 'report in', 'in ,', ', which', 'which found', 'found that', 'that ten-year-long', 'ten-year-long research', 'research had', 'had failed', 'failed to', 'to fulfill', 'fulfill the', 'the expectations', 'expectations ,', ', funding', 'funding for', 'for machine', 'machine translation', 'translation was', 'was dramatically', 'dramatically reduced', 'reduced .', '. little', 'little further', 'further research', 'research in', 'in machine', 'machine translation', 'translation was', 'was conducted', 'conducted until', 'until the', 'the late', 'late s', 's when', 'when the', 'the first', 'first statistical', 'statistical machine', 'machine translation', 'translation systems', 'systems were', 'were developed', 'developed .', '. some', 'some notably', 'notably successful', 'successful natural', 'natural language', 'language processing', 'processing systems', 'systems developed', 'developed in', 'in the', 'the s', 's were', 'were shrdlu', 'shrdlu ,', ', a', 'a natural', 'natural language', 'language system', 'system working', 'working in', 'in restricted', 'restricted ``', '`` blocks', 'blocks worlds', \"worlds ''\", \"'' with\", 'with restricted', 'restricted vocabularies', 'vocabularies ,', ', and', 'and eliza', 'eliza ,', ', a', 'a simulation', 'simulation of', 'of a', 'a rogerian', 'rogerian psychotherapist', 'psychotherapist ,', ', written', 'written by', 'by joseph', 'joseph weizenbaum', 'weizenbaum between', 'between and', 'and .', '. using', 'using almost', 'almost no', 'no information', 'information about', 'about human', 'human thought', 'thought or', 'or emotion', 'emotion ,', ', eliza', 'eliza sometimes', 'sometimes provided', 'provided a', 'a startlingly', 'startlingly human-like', 'human-like interaction', 'interaction .', '. when', 'when the', 'the ``', '`` patient', \"patient ''\", \"'' exceeded\", 'exceeded the', 'the very', 'very small', 'small knowledge', 'knowledge base', 'base ,', ', eliza', 'eliza might', 'might provide', 'provide a', 'a generic', 'generic response', 'response ,', ', for', 'for example', 'example ,', ', responding', 'responding to', 'to ``', '`` my', 'my head', 'head hurts', \"hurts ''\", \"'' with\", 'with ``', '`` why', 'why do', 'do you', 'you say', 'say your', 'your head', 'head hurts', 'hurts ?', \"? ''\", \"'' .\", '. during', 'during the', 'the s', 's ,', ', many', 'many programmers', 'programmers began', 'began to', 'to write', 'write ``', '`` conceptual', 'conceptual ontologies', \"ontologies ''\", \"'' ,\", ', which', 'which structured', 'structured real-world', 'real-world information', 'information into', 'into computer-understandable', 'computer-understandable data', 'data .', '. examples', 'examples are', 'are margie', 'margie (', '( schank', 'schank ,', ', )', ') ,', ', sam', 'sam (', '( cullingford', 'cullingford ,', ', )', ') ,', ', pam', 'pam (', '( wilensky', 'wilensky ,', ', )', ') ,', ', talespin', 'talespin (', '( meehan', 'meehan ,', ', )', ') ,', ', qualm', 'qualm (', '( lehnert', 'lehnert ,', ', )', ') ,', ', politics', 'politics (', '( carbonell', 'carbonell ,', ', )', ') ,', ', and', 'and plot', 'plot units', 'units (', '( lehnert', 'lehnert )', ') .', '. during', 'during this', 'this time', 'time ,', ', many', 'many chatterbots', 'chatterbots were', 'were written', 'written including', 'including parry', 'parry ,', ', racter', 'racter ,', ', and', 'and jabberwacky', 'jabberwacky .', '. up', 'up to', 'to the', 'the s', 's ,', ', most', 'most natural', 'natural language', 'language processing', 'processing systems', 'systems were', 'were based', 'based on', 'on complex', 'complex sets', 'sets of', 'of hand-written', 'hand-written rules', 'rules .', '. starting', 'starting in', 'in the', 'the late', 'late s', 's ,', ', however', 'however ,', ', there', 'there was', 'was a', 'a revolution', 'revolution in', 'in natural', 'natural language', 'language processing', 'processing with', 'with the', 'the introduction', 'introduction of', 'of machine', 'machine learning', 'learning algorithms', 'algorithms for', 'for language', 'language processing', 'processing .', '. this', 'this was', 'was due', 'due to', 'to both', 'both the', 'the steady', 'steady increase', 'increase in', 'in computational', 'computational power', 'power (', '( see', 'see moore', \"moore 's\", \"'s law\", 'law )', ') and', 'and the', 'the gradual', 'gradual lessening', 'lessening of', 'of the', 'the dominance', 'dominance of', 'of chomskyan', 'chomskyan theories', 'theories of', 'of linguistics', 'linguistics (', '( e.g', 'e.g .', '. transformational', 'transformational grammar', 'grammar )', ') ,', ', whose', 'whose theoretical', 'theoretical underpinnings', 'underpinnings discouraged', 'discouraged the', 'the sort', 'sort of', 'of corpus', 'corpus linguistics', 'linguistics that', 'that underlies', 'underlies the', 'the machine-learning', 'machine-learning approach', 'approach to', 'to language', 'language processing', 'processing .', '. [', '[ ]', '] some', 'some of', 'of the', 'the earliest-used', 'earliest-used machine', 'machine learning', 'learning algorithms', 'algorithms ,', ', such', 'such as', 'as decision', 'decision trees', 'trees ,', ', produced', 'produced systems', 'systems of', 'of hard', 'hard if-then', 'if-then rules', 'rules similar', 'similar to', 'to existing', 'existing hand-written', 'hand-written rules', 'rules .', '. however', 'however ,', ', part-of-speech', 'part-of-speech tagging', 'tagging introduced', 'introduced the', 'the use', 'use of', 'of hidden', 'hidden markov', 'markov models', 'models to', 'to natural', 'natural language', 'language processing', 'processing ,', ', and', 'and increasingly', 'increasingly ,', ', research', 'research has', 'has focused', 'focused on', 'on statistical', 'statistical models', 'models ,', ', which', 'which make', 'make soft', 'soft ,', ', probabilistic', 'probabilistic decisions', 'decisions based', 'based on', 'on attaching', 'attaching real-valued', 'real-valued weights', 'weights to', 'to the', 'the features', 'features making', 'making up', 'up the', 'the input', 'input data', 'data .', '. the', 'the cache', 'cache language', 'language models', 'models upon', 'upon which', 'which many', 'many speech', 'speech recognition', 'recognition systems', 'systems now', 'now rely', 'rely are', 'are examples', 'examples of', 'of such', 'such statistical', 'statistical models', 'models .', '. such', 'such models', 'models are', 'are generally', 'generally more', 'more robust', 'robust when', 'when given', 'given unfamiliar', 'unfamiliar input', 'input ,', ', especially', 'especially input', 'input that', 'that contains', 'contains errors', 'errors (', '( as', 'as is', 'is very', 'very common', 'common for', 'for real-world', 'real-world data', 'data )', ') ,', ', and', 'and produce', 'produce more', 'more reliable', 'reliable results', 'results when', 'when integrated', 'integrated into', 'into a', 'a larger', 'larger system', 'system comprising', 'comprising multiple', 'multiple subtasks', 'subtasks .', '. many', 'many of', 'of the', 'the notable', 'notable early', 'early successes', 'successes occurred', 'occurred in', 'in the', 'the field', 'field of', 'of machine', 'machine translation', 'translation ,', ', due', 'due especially', 'especially to', 'to work', 'work at', 'at ibm', 'ibm research', 'research ,', ', where', 'where successively', 'successively more', 'more complicated', 'complicated statistical', 'statistical models', 'models were', 'were developed', 'developed .', '. these', 'these systems', 'systems were', 'were able', 'able to', 'to take', 'take advantage', 'advantage of', 'of existing', 'existing multilingual', 'multilingual textual', 'textual corpora', 'corpora that', 'that had', 'had been', 'been produced', 'produced by', 'by the', 'the parliament', 'parliament of', 'of canada', 'canada and', 'and the', 'the european', 'european union', 'union as', 'as a', 'a result', 'result of', 'of laws', 'laws calling', 'calling for', 'for the', 'the translation', 'translation of', 'of all', 'all governmental', 'governmental proceedings', 'proceedings into', 'into all', 'all official', 'official languages', 'languages of', 'of the', 'the corresponding', 'corresponding systems', 'systems of', 'of government', 'government .', '. however', 'however ,', ', most', 'most other', 'other systems', 'systems depended', 'depended on', 'on corpora', 'corpora specifically', 'specifically developed', 'developed for', 'for the', 'the tasks', 'tasks implemented', 'implemented by', 'by these', 'these systems', 'systems ,', ', which', 'which was', 'was (', '( and', 'and often', 'often continues', 'continues to', 'to be', 'be )', ') a', 'a major', 'major limitation', 'limitation in', 'in the', 'the success', 'success of', 'of these', 'these systems', 'systems .', '. as', 'as a', 'a result', 'result ,', ', a', 'a great', 'great deal', 'deal of', 'of research', 'research has', 'has gone', 'gone into', 'into methods', 'methods of', 'of more', 'more effectively', 'effectively learning', 'learning from', 'from limited', 'limited amounts', 'amounts of', 'of data', 'data .', '. recent', 'recent research', 'research has', 'has increasingly', 'increasingly focused', 'focused on', 'on unsupervised', 'unsupervised and', 'and semi-supervised', 'semi-supervised learning', 'learning algorithms', 'algorithms .', '. such', 'such algorithms', 'algorithms can', 'can learn', 'learn from', 'from data', 'data that', 'that has', 'has not', 'not been', 'been hand-annotated', 'hand-annotated with', 'with the', 'the desired', 'desired answers', 'answers or', 'or using', 'using a', 'a combination', 'combination of', 'of annotated', 'annotated and', 'and non-annotated', 'non-annotated data', 'data .', '. generally', 'generally ,', ', this', 'this task', 'task is', 'is much', 'much more', 'more difficult', 'difficult than', 'than supervised', 'supervised learning', 'learning ,', ', and', 'and typically', 'typically produces', 'produces less', 'less accurate', 'accurate results', 'results for', 'for a', 'a given', 'given amount', 'amount of', 'of input', 'input data', 'data .', '. however', 'however ,', ', there', 'there is', 'is an', 'an enormous', 'enormous amount', 'amount of', 'of non-annotated', 'non-annotated data', 'data available', 'available (', '( including', 'including ,', ', among', 'among other', 'other things', 'things ,', ', the', 'the entire', 'entire content', 'content of', 'of the', 'the world', 'world wide', 'wide web', 'web )', ') ,', ', which', 'which can', 'can often', 'often make', 'make up', 'up for', 'for the', 'the inferior', 'inferior results', 'results if', 'if the', 'the algorithm', 'algorithm used', 'used has', 'has a', 'a low', 'low enough', 'enough time', 'time complexity', 'complexity to', 'to be', 'be practical', 'practical .', '. in', 'in the', 'the s', 's ,', ', representation', 'representation learning', 'learning and', 'and deep', 'deep neural', 'neural network-style', 'network-style machine', 'machine learning', 'learning methods', 'methods became', 'became widespread', 'widespread in', 'in natural', 'natural language', 'language processing', 'processing ,', ', due', 'due in', 'in part', 'part to', 'to a', 'a flurry', 'flurry of', 'of results', 'results showing', 'showing that', 'that such', 'such techniques', 'techniques [', '[ ]', '] [', '[ ]', '] can', 'can achieve', 'achieve state-of-the-art', 'state-of-the-art results', 'results in', 'in many', 'many natural', 'natural language', 'language tasks', 'tasks ,', ', for', 'for example', 'example in', 'in language', 'language modeling', 'modeling ,', ', [', '[ ]', '] parsing', 'parsing ,', ', [', '[ ]', '] [', '[ ]', '] and', 'and many', 'many others', 'others .', '. popular', 'popular techniques', 'techniques include', 'include the', 'the use', 'use of', 'of word', 'word embeddings', 'embeddings to', 'to capture', 'capture semantic', 'semantic properties', 'properties of', 'of words', 'words ,', ', and', 'and an', 'an increase', 'increase in', 'in end-to-end', 'end-to-end learning', 'learning of', 'of a', 'a higher-level', 'higher-level task', 'task (', '( e.g.', 'e.g. ,', ', question', 'question answering', 'answering )', ') instead', 'instead of', 'of relying', 'relying on', 'on a', 'a pipeline', 'pipeline of', 'of separate', 'separate intermediate', 'intermediate tasks', 'tasks (', '( e.g.', 'e.g. ,', ', part-of-speech', 'part-of-speech tagging', 'tagging and', 'and dependency', 'dependency parsing', 'parsing )', ') .', '. in', 'in some', 'some areas', 'areas ,', ', this', 'this shift', 'shift has', 'has entailed', 'entailed substantial', 'substantial changes', 'changes in', 'in how', 'how nlp', 'nlp systems', 'systems are', 'are designed', 'designed ,', ', such', 'such that', 'that deep', 'deep neural', 'neural network-based', 'network-based approaches', 'approaches may', 'may be', 'be viewed', 'viewed as', 'as a', 'a new', 'new paradigm', 'paradigm distinct', 'distinct from', 'from statistical', 'statistical natural', 'natural language', 'language processing', 'processing .', '. for', 'for instance', 'instance ,', ', the', 'the term', 'term neural', 'neural machine', 'machine translation', 'translation (', '( nmt', 'nmt )', ') emphasizes', 'emphasizes the', 'the fact', 'fact that', 'that deep', 'deep learning-based', 'learning-based approaches', 'approaches to', 'to machine', 'machine translation', 'translation directly', 'directly learn', 'learn sequence-to-sequence', 'sequence-to-sequence transformations', 'transformations ,', ', obviating', 'obviating the', 'the need', 'need for', 'for intermediate', 'intermediate steps', 'steps such', 'such as', 'as word', 'word alignment', 'alignment and', 'and language', 'language modeling', 'modeling that', 'that was', 'was used', 'used in', 'in statistical', 'statistical machine', 'machine translation', 'translation (', '( smt', 'smt )', ') .', '. in', 'in the', 'the early', 'early days', 'days ,', ', many', 'many language-processing', 'language-processing systems', 'systems were', 'were designed', 'designed by', 'by hand-coding', 'hand-coding a', 'a set', 'set of', 'of rules', 'rules :', ': [', '[ ]', '] [', '[ ]', '] such', 'such as', 'as by', 'by writing', 'writing grammars', 'grammars or', 'or devising', 'devising heuristic', 'heuristic rules', 'rules for', 'for stemming', 'stemming .', '. since', 'since the', 'the so-called', 'so-called ``', '`` statistical', 'statistical revolution', \"revolution ''\", \"'' [\", '[ ]', '] [', '[ ]', '] in', 'in the', 'the late', 'late s', 's and', 'and mid-s', 'mid-s ,', ', much', 'much natural', 'natural language', 'language processing', 'processing research', 'research has', 'has relied', 'relied heavily', 'heavily on', 'on machine', 'machine learning', 'learning .', '. the', 'the machine-learning', 'machine-learning paradigm', 'paradigm calls', 'calls instead', 'instead for', 'for using', 'using statistical', 'statistical inference', 'inference to', 'to automatically', 'automatically learn', 'learn such', 'such rules', 'rules through', 'through the', 'the analysis', 'analysis of', 'of large', 'large corpora', 'corpora (', '( the', 'the plural', 'plural form', 'form of', 'of corpus', 'corpus ,', ', is', 'is a', 'a set', 'set of', 'of documents', 'documents ,', ', possibly', 'possibly with', 'with human', 'human or', 'or computer', 'computer annotations', 'annotations )', ') of', 'of typical', 'typical real-world', 'real-world examples', 'examples .', '. many', 'many different', 'different classes', 'classes of', 'of machine-learning', 'machine-learning algorithms', 'algorithms have', 'have been', 'been applied', 'applied to', 'to natural-language-processing', 'natural-language-processing tasks', 'tasks .', '. these', 'these algorithms', 'algorithms take', 'take as', 'as input', 'input a', 'a large', 'large set', 'set of', 'of ``', '`` features', \"features ''\", \"'' that\", 'that are', 'are generated', 'generated from', 'from the', 'the input', 'input data', 'data .', '. some', 'some of', 'of the', 'the earliest-used', 'earliest-used algorithms', 'algorithms ,', ', such', 'such as', 'as decision', 'decision trees', 'trees ,', ', produced', 'produced systems', 'systems of', 'of hard', 'hard if-then', 'if-then rules', 'rules similar', 'similar to', 'to the', 'the systems', 'systems of', 'of handwritten', 'handwritten rules', 'rules that', 'that were', 'were then', 'then common', 'common .', '. increasingly', 'increasingly ,', ', however', 'however ,', ', research', 'research has', 'has focused', 'focused on', 'on statistical', 'statistical models', 'models ,', ', which', 'which make', 'make soft', 'soft ,', ', probabilistic', 'probabilistic decisions', 'decisions based', 'based on', 'on attaching', 'attaching real-valued', 'real-valued weights', 'weights to', 'to each', 'each input', 'input feature', 'feature .', '. such', 'such models', 'models have', 'have the', 'the advantage', 'advantage that', 'that they', 'they can', 'can express', 'express the', 'the relative', 'relative certainty', 'certainty of', 'of many', 'many different', 'different possible', 'possible answers', 'answers rather', 'rather than', 'than only', 'only one', 'one ,', ', producing', 'producing more', 'more reliable', 'reliable results', 'results when', 'when such', 'such a', 'a model', 'model is', 'is included', 'included as', 'as a', 'a component', 'component of', 'of a', 'a larger', 'larger system', 'system .', '. systems', 'systems based', 'based on', 'on machine-learning', 'machine-learning algorithms', 'algorithms have', 'have many', 'many advantages', 'advantages over', 'over hand-produced', 'hand-produced rules', 'rules :', ': the', 'the following', 'following is', 'is a', 'a list', 'list of', 'of some', 'some of', 'of the', 'the most', 'most commonly', 'commonly researched', 'researched tasks', 'tasks in', 'in natural', 'natural language', 'language processing', 'processing .', '. some', 'some of', 'of these', 'these tasks', 'tasks have', 'have direct', 'direct real-world', 'real-world applications', 'applications ,', ', while', 'while others', 'others more', 'more commonly', 'commonly serve', 'serve as', 'as subtasks', 'subtasks that', 'that are', 'are used', 'used to', 'to aid', 'aid in', 'in solving', 'solving larger', 'larger tasks', 'tasks .', '. though', 'though natural', 'natural language', 'language processing', 'processing tasks', 'tasks are', 'are closely', 'closely intertwined', 'intertwined ,', ', they', 'they are', 'are frequently', 'frequently subdivided', 'subdivided into', 'into categories', 'categories for', 'for convenience', 'convenience .', '. a', 'a coarse', 'coarse division', 'division is', 'is given', 'given below', 'below .', '. the', 'the first', 'first published', 'published work', 'work by', 'by an', 'an artificial', 'artificial intelligence', 'intelligence was', 'was published', 'published in', 'in ,', ', the', 'the road', 'road ,', ', marketed', 'marketed as', 'as a', 'a novel', 'novel ,', ', contains', 'contains sixty', 'sixty million', 'million words', 'words .']\n"
     ]
    }
   ],
   "source": [
    "#function to generate bigrams from sentences\n",
    "from nltk.util import ngrams\n",
    "def extract_ngrams(data,num):\n",
    "    n_grams = ngrams(nltk.word_tokenize(data.lower()), num)\n",
    "    return [' '.join(grams) for grams in n_grams]\n",
    "gram_num = 2\n",
    "n_grams = extract_ngrams(a, gram_num)\n",
    "print(n_grams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Generate TriGrams**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['natural language processing', 'language processing (', 'processing ( nlp', '( nlp )', 'nlp ) is', ') is a', 'is a subfield', 'a subfield of', 'subfield of linguistics', 'of linguistics ,', 'linguistics , computer', ', computer science', 'computer science ,', 'science , information', ', information engineering', 'information engineering ,', 'engineering , and', ', and artificial', 'and artificial intelligence', 'artificial intelligence concerned', 'intelligence concerned with', 'concerned with the', 'with the interactions', 'the interactions between', 'interactions between computers', 'between computers and', 'computers and human', 'and human (', 'human ( natural', '( natural )', 'natural ) languages', ') languages ,', 'languages , in', ', in particular', 'in particular how', 'particular how to', 'how to program', 'to program computers', 'program computers to', 'computers to process', 'to process and', 'process and analyze', 'and analyze large', 'analyze large amounts', 'large amounts of', 'amounts of natural', 'of natural language', 'natural language data', 'language data .', 'data . challenges', '. challenges in', 'challenges in natural', 'in natural language', 'natural language processing', 'language processing frequently', 'processing frequently involve', 'frequently involve speech', 'involve speech recognition', 'speech recognition ,', 'recognition , natural', ', natural language', 'natural language understanding', 'language understanding ,', 'understanding , and', ', and natural', 'and natural language', 'natural language generation', 'language generation .', 'generation . the', '. the history', 'the history of', 'history of natural', 'of natural language', 'natural language processing', 'language processing (', 'processing ( nlp', '( nlp )', 'nlp ) generally', ') generally started', 'generally started in', 'started in the', 'in the s', 'the s ,', 's , although', ', although work', 'although work can', 'work can be', 'can be found', 'be found from', 'found from earlier', 'from earlier periods', 'earlier periods .', 'periods . in', '. in ,', 'in , alan', ', alan turing', 'alan turing published', 'turing published an', 'published an article', 'an article titled', 'article titled ``', 'titled `` computing', '`` computing machinery', 'computing machinery and', 'machinery and intelligence', \"and intelligence ''\", \"intelligence '' which\", \"'' which proposed\", 'which proposed what', 'proposed what is', 'what is now', 'is now called', 'now called the', 'called the turing', 'the turing test', 'turing test as', 'test as a', 'as a criterion', 'a criterion of', 'criterion of intelligence', 'of intelligence [', 'intelligence [ clarification', '[ clarification needed', 'clarification needed ]', 'needed ] .', '] . the', '. the georgetown', 'the georgetown experiment', 'georgetown experiment in', 'experiment in involved', 'in involved fully', 'involved fully automatic', 'fully automatic translation', 'automatic translation of', 'translation of more', 'of more than', 'more than sixty', 'than sixty russian', 'sixty russian sentences', 'russian sentences into', 'sentences into english', 'into english .', 'english . the', '. the authors', 'the authors claimed', 'authors claimed that', 'claimed that within', 'that within three', 'within three or', 'three or five', 'or five years', 'five years ,', 'years , machine', ', machine translation', 'machine translation would', 'translation would be', 'would be a', 'be a solved', 'a solved problem', 'solved problem .', 'problem . [', '. [ ]', '[ ] however', '] however ,', 'however , real', ', real progress', 'real progress was', 'progress was much', 'was much slower', 'much slower ,', 'slower , and', ', and after', 'and after the', 'after the alpac', 'the alpac report', 'alpac report in', 'report in ,', 'in , which', ', which found', 'which found that', 'found that ten-year-long', 'that ten-year-long research', 'ten-year-long research had', 'research had failed', 'had failed to', 'failed to fulfill', 'to fulfill the', 'fulfill the expectations', 'the expectations ,', 'expectations , funding', ', funding for', 'funding for machine', 'for machine translation', 'machine translation was', 'translation was dramatically', 'was dramatically reduced', 'dramatically reduced .', 'reduced . little', '. little further', 'little further research', 'further research in', 'research in machine', 'in machine translation', 'machine translation was', 'translation was conducted', 'was conducted until', 'conducted until the', 'until the late', 'the late s', 'late s when', 's when the', 'when the first', 'the first statistical', 'first statistical machine', 'statistical machine translation', 'machine translation systems', 'translation systems were', 'systems were developed', 'were developed .', 'developed . some', '. some notably', 'some notably successful', 'notably successful natural', 'successful natural language', 'natural language processing', 'language processing systems', 'processing systems developed', 'systems developed in', 'developed in the', 'in the s', 'the s were', 's were shrdlu', 'were shrdlu ,', 'shrdlu , a', ', a natural', 'a natural language', 'natural language system', 'language system working', 'system working in', 'working in restricted', 'in restricted ``', 'restricted `` blocks', '`` blocks worlds', \"blocks worlds ''\", \"worlds '' with\", \"'' with restricted\", 'with restricted vocabularies', 'restricted vocabularies ,', 'vocabularies , and', ', and eliza', 'and eliza ,', 'eliza , a', ', a simulation', 'a simulation of', 'simulation of a', 'of a rogerian', 'a rogerian psychotherapist', 'rogerian psychotherapist ,', 'psychotherapist , written', ', written by', 'written by joseph', 'by joseph weizenbaum', 'joseph weizenbaum between', 'weizenbaum between and', 'between and .', 'and . using', '. using almost', 'using almost no', 'almost no information', 'no information about', 'information about human', 'about human thought', 'human thought or', 'thought or emotion', 'or emotion ,', 'emotion , eliza', ', eliza sometimes', 'eliza sometimes provided', 'sometimes provided a', 'provided a startlingly', 'a startlingly human-like', 'startlingly human-like interaction', 'human-like interaction .', 'interaction . when', '. when the', 'when the ``', 'the `` patient', \"`` patient ''\", \"patient '' exceeded\", \"'' exceeded the\", 'exceeded the very', 'the very small', 'very small knowledge', 'small knowledge base', 'knowledge base ,', 'base , eliza', ', eliza might', 'eliza might provide', 'might provide a', 'provide a generic', 'a generic response', 'generic response ,', 'response , for', ', for example', 'for example ,', 'example , responding', ', responding to', 'responding to ``', 'to `` my', '`` my head', 'my head hurts', \"head hurts ''\", \"hurts '' with\", \"'' with ``\", 'with `` why', '`` why do', 'why do you', 'do you say', 'you say your', 'say your head', 'your head hurts', 'head hurts ?', \"hurts ? ''\", \"? '' .\", \"'' . during\", '. during the', 'during the s', 'the s ,', 's , many', ', many programmers', 'many programmers began', 'programmers began to', 'began to write', 'to write ``', 'write `` conceptual', '`` conceptual ontologies', \"conceptual ontologies ''\", \"ontologies '' ,\", \"'' , which\", ', which structured', 'which structured real-world', 'structured real-world information', 'real-world information into', 'information into computer-understandable', 'into computer-understandable data', 'computer-understandable data .', 'data . examples', '. examples are', 'examples are margie', 'are margie (', 'margie ( schank', '( schank ,', 'schank , )', ', ) ,', ') , sam', ', sam (', 'sam ( cullingford', '( cullingford ,', 'cullingford , )', ', ) ,', ') , pam', ', pam (', 'pam ( wilensky', '( wilensky ,', 'wilensky , )', ', ) ,', ') , talespin', ', talespin (', 'talespin ( meehan', '( meehan ,', 'meehan , )', ', ) ,', ') , qualm', ', qualm (', 'qualm ( lehnert', '( lehnert ,', 'lehnert , )', ', ) ,', ') , politics', ', politics (', 'politics ( carbonell', '( carbonell ,', 'carbonell , )', ', ) ,', ') , and', ', and plot', 'and plot units', 'plot units (', 'units ( lehnert', '( lehnert )', 'lehnert ) .', ') . during', '. during this', 'during this time', 'this time ,', 'time , many', ', many chatterbots', 'many chatterbots were', 'chatterbots were written', 'were written including', 'written including parry', 'including parry ,', 'parry , racter', ', racter ,', 'racter , and', ', and jabberwacky', 'and jabberwacky .', 'jabberwacky . up', '. up to', 'up to the', 'to the s', 'the s ,', 's , most', ', most natural', 'most natural language', 'natural language processing', 'language processing systems', 'processing systems were', 'systems were based', 'were based on', 'based on complex', 'on complex sets', 'complex sets of', 'sets of hand-written', 'of hand-written rules', 'hand-written rules .', 'rules . starting', '. starting in', 'starting in the', 'in the late', 'the late s', 'late s ,', 's , however', ', however ,', 'however , there', ', there was', 'there was a', 'was a revolution', 'a revolution in', 'revolution in natural', 'in natural language', 'natural language processing', 'language processing with', 'processing with the', 'with the introduction', 'the introduction of', 'introduction of machine', 'of machine learning', 'machine learning algorithms', 'learning algorithms for', 'algorithms for language', 'for language processing', 'language processing .', 'processing . this', '. this was', 'this was due', 'was due to', 'due to both', 'to both the', 'both the steady', 'the steady increase', 'steady increase in', 'increase in computational', 'in computational power', 'computational power (', 'power ( see', '( see moore', \"see moore 's\", \"moore 's law\", \"'s law )\", 'law ) and', ') and the', 'and the gradual', 'the gradual lessening', 'gradual lessening of', 'lessening of the', 'of the dominance', 'the dominance of', 'dominance of chomskyan', 'of chomskyan theories', 'chomskyan theories of', 'theories of linguistics', 'of linguistics (', 'linguistics ( e.g', '( e.g .', 'e.g . transformational', '. transformational grammar', 'transformational grammar )', 'grammar ) ,', ') , whose', ', whose theoretical', 'whose theoretical underpinnings', 'theoretical underpinnings discouraged', 'underpinnings discouraged the', 'discouraged the sort', 'the sort of', 'sort of corpus', 'of corpus linguistics', 'corpus linguistics that', 'linguistics that underlies', 'that underlies the', 'underlies the machine-learning', 'the machine-learning approach', 'machine-learning approach to', 'approach to language', 'to language processing', 'language processing .', 'processing . [', '. [ ]', '[ ] some', '] some of', 'some of the', 'of the earliest-used', 'the earliest-used machine', 'earliest-used machine learning', 'machine learning algorithms', 'learning algorithms ,', 'algorithms , such', ', such as', 'such as decision', 'as decision trees', 'decision trees ,', 'trees , produced', ', produced systems', 'produced systems of', 'systems of hard', 'of hard if-then', 'hard if-then rules', 'if-then rules similar', 'rules similar to', 'similar to existing', 'to existing hand-written', 'existing hand-written rules', 'hand-written rules .', 'rules . however', '. however ,', 'however , part-of-speech', ', part-of-speech tagging', 'part-of-speech tagging introduced', 'tagging introduced the', 'introduced the use', 'the use of', 'use of hidden', 'of hidden markov', 'hidden markov models', 'markov models to', 'models to natural', 'to natural language', 'natural language processing', 'language processing ,', 'processing , and', ', and increasingly', 'and increasingly ,', 'increasingly , research', ', research has', 'research has focused', 'has focused on', 'focused on statistical', 'on statistical models', 'statistical models ,', 'models , which', ', which make', 'which make soft', 'make soft ,', 'soft , probabilistic', ', probabilistic decisions', 'probabilistic decisions based', 'decisions based on', 'based on attaching', 'on attaching real-valued', 'attaching real-valued weights', 'real-valued weights to', 'weights to the', 'to the features', 'the features making', 'features making up', 'making up the', 'up the input', 'the input data', 'input data .', 'data . the', '. the cache', 'the cache language', 'cache language models', 'language models upon', 'models upon which', 'upon which many', 'which many speech', 'many speech recognition', 'speech recognition systems', 'recognition systems now', 'systems now rely', 'now rely are', 'rely are examples', 'are examples of', 'examples of such', 'of such statistical', 'such statistical models', 'statistical models .', 'models . such', '. such models', 'such models are', 'models are generally', 'are generally more', 'generally more robust', 'more robust when', 'robust when given', 'when given unfamiliar', 'given unfamiliar input', 'unfamiliar input ,', 'input , especially', ', especially input', 'especially input that', 'input that contains', 'that contains errors', 'contains errors (', 'errors ( as', '( as is', 'as is very', 'is very common', 'very common for', 'common for real-world', 'for real-world data', 'real-world data )', 'data ) ,', ') , and', ', and produce', 'and produce more', 'produce more reliable', 'more reliable results', 'reliable results when', 'results when integrated', 'when integrated into', 'integrated into a', 'into a larger', 'a larger system', 'larger system comprising', 'system comprising multiple', 'comprising multiple subtasks', 'multiple subtasks .', 'subtasks . many', '. many of', 'many of the', 'of the notable', 'the notable early', 'notable early successes', 'early successes occurred', 'successes occurred in', 'occurred in the', 'in the field', 'the field of', 'field of machine', 'of machine translation', 'machine translation ,', 'translation , due', ', due especially', 'due especially to', 'especially to work', 'to work at', 'work at ibm', 'at ibm research', 'ibm research ,', 'research , where', ', where successively', 'where successively more', 'successively more complicated', 'more complicated statistical', 'complicated statistical models', 'statistical models were', 'models were developed', 'were developed .', 'developed . these', '. these systems', 'these systems were', 'systems were able', 'were able to', 'able to take', 'to take advantage', 'take advantage of', 'advantage of existing', 'of existing multilingual', 'existing multilingual textual', 'multilingual textual corpora', 'textual corpora that', 'corpora that had', 'that had been', 'had been produced', 'been produced by', 'produced by the', 'by the parliament', 'the parliament of', 'parliament of canada', 'of canada and', 'canada and the', 'and the european', 'the european union', 'european union as', 'union as a', 'as a result', 'a result of', 'result of laws', 'of laws calling', 'laws calling for', 'calling for the', 'for the translation', 'the translation of', 'translation of all', 'of all governmental', 'all governmental proceedings', 'governmental proceedings into', 'proceedings into all', 'into all official', 'all official languages', 'official languages of', 'languages of the', 'of the corresponding', 'the corresponding systems', 'corresponding systems of', 'systems of government', 'of government .', 'government . however', '. however ,', 'however , most', ', most other', 'most other systems', 'other systems depended', 'systems depended on', 'depended on corpora', 'on corpora specifically', 'corpora specifically developed', 'specifically developed for', 'developed for the', 'for the tasks', 'the tasks implemented', 'tasks implemented by', 'implemented by these', 'by these systems', 'these systems ,', 'systems , which', ', which was', 'which was (', 'was ( and', '( and often', 'and often continues', 'often continues to', 'continues to be', 'to be )', 'be ) a', ') a major', 'a major limitation', 'major limitation in', 'limitation in the', 'in the success', 'the success of', 'success of these', 'of these systems', 'these systems .', 'systems . as', '. as a', 'as a result', 'a result ,', 'result , a', ', a great', 'a great deal', 'great deal of', 'deal of research', 'of research has', 'research has gone', 'has gone into', 'gone into methods', 'into methods of', 'methods of more', 'of more effectively', 'more effectively learning', 'effectively learning from', 'learning from limited', 'from limited amounts', 'limited amounts of', 'amounts of data', 'of data .', 'data . recent', '. recent research', 'recent research has', 'research has increasingly', 'has increasingly focused', 'increasingly focused on', 'focused on unsupervised', 'on unsupervised and', 'unsupervised and semi-supervised', 'and semi-supervised learning', 'semi-supervised learning algorithms', 'learning algorithms .', 'algorithms . such', '. such algorithms', 'such algorithms can', 'algorithms can learn', 'can learn from', 'learn from data', 'from data that', 'data that has', 'that has not', 'has not been', 'not been hand-annotated', 'been hand-annotated with', 'hand-annotated with the', 'with the desired', 'the desired answers', 'desired answers or', 'answers or using', 'or using a', 'using a combination', 'a combination of', 'combination of annotated', 'of annotated and', 'annotated and non-annotated', 'and non-annotated data', 'non-annotated data .', 'data . generally', '. generally ,', 'generally , this', ', this task', 'this task is', 'task is much', 'is much more', 'much more difficult', 'more difficult than', 'difficult than supervised', 'than supervised learning', 'supervised learning ,', 'learning , and', ', and typically', 'and typically produces', 'typically produces less', 'produces less accurate', 'less accurate results', 'accurate results for', 'results for a', 'for a given', 'a given amount', 'given amount of', 'amount of input', 'of input data', 'input data .', 'data . however', '. however ,', 'however , there', ', there is', 'there is an', 'is an enormous', 'an enormous amount', 'enormous amount of', 'amount of non-annotated', 'of non-annotated data', 'non-annotated data available', 'data available (', 'available ( including', '( including ,', 'including , among', ', among other', 'among other things', 'other things ,', 'things , the', ', the entire', 'the entire content', 'entire content of', 'content of the', 'of the world', 'the world wide', 'world wide web', 'wide web )', 'web ) ,', ') , which', ', which can', 'which can often', 'can often make', 'often make up', 'make up for', 'up for the', 'for the inferior', 'the inferior results', 'inferior results if', 'results if the', 'if the algorithm', 'the algorithm used', 'algorithm used has', 'used has a', 'has a low', 'a low enough', 'low enough time', 'enough time complexity', 'time complexity to', 'complexity to be', 'to be practical', 'be practical .', 'practical . in', '. in the', 'in the s', 'the s ,', 's , representation', ', representation learning', 'representation learning and', 'learning and deep', 'and deep neural', 'deep neural network-style', 'neural network-style machine', 'network-style machine learning', 'machine learning methods', 'learning methods became', 'methods became widespread', 'became widespread in', 'widespread in natural', 'in natural language', 'natural language processing', 'language processing ,', 'processing , due', ', due in', 'due in part', 'in part to', 'part to a', 'to a flurry', 'a flurry of', 'flurry of results', 'of results showing', 'results showing that', 'showing that such', 'that such techniques', 'such techniques [', 'techniques [ ]', '[ ] [', '] [ ]', '[ ] can', '] can achieve', 'can achieve state-of-the-art', 'achieve state-of-the-art results', 'state-of-the-art results in', 'results in many', 'in many natural', 'many natural language', 'natural language tasks', 'language tasks ,', 'tasks , for', ', for example', 'for example in', 'example in language', 'in language modeling', 'language modeling ,', 'modeling , [', ', [ ]', '[ ] parsing', '] parsing ,', 'parsing , [', ', [ ]', '[ ] [', '] [ ]', '[ ] and', '] and many', 'and many others', 'many others .', 'others . popular', '. popular techniques', 'popular techniques include', 'techniques include the', 'include the use', 'the use of', 'use of word', 'of word embeddings', 'word embeddings to', 'embeddings to capture', 'to capture semantic', 'capture semantic properties', 'semantic properties of', 'properties of words', 'of words ,', 'words , and', ', and an', 'and an increase', 'an increase in', 'increase in end-to-end', 'in end-to-end learning', 'end-to-end learning of', 'learning of a', 'of a higher-level', 'a higher-level task', 'higher-level task (', 'task ( e.g.', '( e.g. ,', 'e.g. , question', ', question answering', 'question answering )', 'answering ) instead', ') instead of', 'instead of relying', 'of relying on', 'relying on a', 'on a pipeline', 'a pipeline of', 'pipeline of separate', 'of separate intermediate', 'separate intermediate tasks', 'intermediate tasks (', 'tasks ( e.g.', '( e.g. ,', 'e.g. , part-of-speech', ', part-of-speech tagging', 'part-of-speech tagging and', 'tagging and dependency', 'and dependency parsing', 'dependency parsing )', 'parsing ) .', ') . in', '. in some', 'in some areas', 'some areas ,', 'areas , this', ', this shift', 'this shift has', 'shift has entailed', 'has entailed substantial', 'entailed substantial changes', 'substantial changes in', 'changes in how', 'in how nlp', 'how nlp systems', 'nlp systems are', 'systems are designed', 'are designed ,', 'designed , such', ', such that', 'such that deep', 'that deep neural', 'deep neural network-based', 'neural network-based approaches', 'network-based approaches may', 'approaches may be', 'may be viewed', 'be viewed as', 'viewed as a', 'as a new', 'a new paradigm', 'new paradigm distinct', 'paradigm distinct from', 'distinct from statistical', 'from statistical natural', 'statistical natural language', 'natural language processing', 'language processing .', 'processing . for', '. for instance', 'for instance ,', 'instance , the', ', the term', 'the term neural', 'term neural machine', 'neural machine translation', 'machine translation (', 'translation ( nmt', '( nmt )', 'nmt ) emphasizes', ') emphasizes the', 'emphasizes the fact', 'the fact that', 'fact that deep', 'that deep learning-based', 'deep learning-based approaches', 'learning-based approaches to', 'approaches to machine', 'to machine translation', 'machine translation directly', 'translation directly learn', 'directly learn sequence-to-sequence', 'learn sequence-to-sequence transformations', 'sequence-to-sequence transformations ,', 'transformations , obviating', ', obviating the', 'obviating the need', 'the need for', 'need for intermediate', 'for intermediate steps', 'intermediate steps such', 'steps such as', 'such as word', 'as word alignment', 'word alignment and', 'alignment and language', 'and language modeling', 'language modeling that', 'modeling that was', 'that was used', 'was used in', 'used in statistical', 'in statistical machine', 'statistical machine translation', 'machine translation (', 'translation ( smt', '( smt )', 'smt ) .', ') . in', '. in the', 'in the early', 'the early days', 'early days ,', 'days , many', ', many language-processing', 'many language-processing systems', 'language-processing systems were', 'systems were designed', 'were designed by', 'designed by hand-coding', 'by hand-coding a', 'hand-coding a set', 'a set of', 'set of rules', 'of rules :', 'rules : [', ': [ ]', '[ ] [', '] [ ]', '[ ] such', '] such as', 'such as by', 'as by writing', 'by writing grammars', 'writing grammars or', 'grammars or devising', 'or devising heuristic', 'devising heuristic rules', 'heuristic rules for', 'rules for stemming', 'for stemming .', 'stemming . since', '. since the', 'since the so-called', 'the so-called ``', 'so-called `` statistical', '`` statistical revolution', \"statistical revolution ''\", \"revolution '' [\", \"'' [ ]\", '[ ] [', '] [ ]', '[ ] in', '] in the', 'in the late', 'the late s', 'late s and', 's and mid-s', 'and mid-s ,', 'mid-s , much', ', much natural', 'much natural language', 'natural language processing', 'language processing research', 'processing research has', 'research has relied', 'has relied heavily', 'relied heavily on', 'heavily on machine', 'on machine learning', 'machine learning .', 'learning . the', '. the machine-learning', 'the machine-learning paradigm', 'machine-learning paradigm calls', 'paradigm calls instead', 'calls instead for', 'instead for using', 'for using statistical', 'using statistical inference', 'statistical inference to', 'inference to automatically', 'to automatically learn', 'automatically learn such', 'learn such rules', 'such rules through', 'rules through the', 'through the analysis', 'the analysis of', 'analysis of large', 'of large corpora', 'large corpora (', 'corpora ( the', '( the plural', 'the plural form', 'plural form of', 'form of corpus', 'of corpus ,', 'corpus , is', ', is a', 'is a set', 'a set of', 'set of documents', 'of documents ,', 'documents , possibly', ', possibly with', 'possibly with human', 'with human or', 'human or computer', 'or computer annotations', 'computer annotations )', 'annotations ) of', ') of typical', 'of typical real-world', 'typical real-world examples', 'real-world examples .', 'examples . many', '. many different', 'many different classes', 'different classes of', 'classes of machine-learning', 'of machine-learning algorithms', 'machine-learning algorithms have', 'algorithms have been', 'have been applied', 'been applied to', 'applied to natural-language-processing', 'to natural-language-processing tasks', 'natural-language-processing tasks .', 'tasks . these', '. these algorithms', 'these algorithms take', 'algorithms take as', 'take as input', 'as input a', 'input a large', 'a large set', 'large set of', 'set of ``', 'of `` features', \"`` features ''\", \"features '' that\", \"'' that are\", 'that are generated', 'are generated from', 'generated from the', 'from the input', 'the input data', 'input data .', 'data . some', '. some of', 'some of the', 'of the earliest-used', 'the earliest-used algorithms', 'earliest-used algorithms ,', 'algorithms , such', ', such as', 'such as decision', 'as decision trees', 'decision trees ,', 'trees , produced', ', produced systems', 'produced systems of', 'systems of hard', 'of hard if-then', 'hard if-then rules', 'if-then rules similar', 'rules similar to', 'similar to the', 'to the systems', 'the systems of', 'systems of handwritten', 'of handwritten rules', 'handwritten rules that', 'rules that were', 'that were then', 'were then common', 'then common .', 'common . increasingly', '. increasingly ,', 'increasingly , however', ', however ,', 'however , research', ', research has', 'research has focused', 'has focused on', 'focused on statistical', 'on statistical models', 'statistical models ,', 'models , which', ', which make', 'which make soft', 'make soft ,', 'soft , probabilistic', ', probabilistic decisions', 'probabilistic decisions based', 'decisions based on', 'based on attaching', 'on attaching real-valued', 'attaching real-valued weights', 'real-valued weights to', 'weights to each', 'to each input', 'each input feature', 'input feature .', 'feature . such', '. such models', 'such models have', 'models have the', 'have the advantage', 'the advantage that', 'advantage that they', 'that they can', 'they can express', 'can express the', 'express the relative', 'the relative certainty', 'relative certainty of', 'certainty of many', 'of many different', 'many different possible', 'different possible answers', 'possible answers rather', 'answers rather than', 'rather than only', 'than only one', 'only one ,', 'one , producing', ', producing more', 'producing more reliable', 'more reliable results', 'reliable results when', 'results when such', 'when such a', 'such a model', 'a model is', 'model is included', 'is included as', 'included as a', 'as a component', 'a component of', 'component of a', 'of a larger', 'a larger system', 'larger system .', 'system . systems', '. systems based', 'systems based on', 'based on machine-learning', 'on machine-learning algorithms', 'machine-learning algorithms have', 'algorithms have many', 'have many advantages', 'many advantages over', 'advantages over hand-produced', 'over hand-produced rules', 'hand-produced rules :', 'rules : the', ': the following', 'the following is', 'following is a', 'is a list', 'a list of', 'list of some', 'of some of', 'some of the', 'of the most', 'the most commonly', 'most commonly researched', 'commonly researched tasks', 'researched tasks in', 'tasks in natural', 'in natural language', 'natural language processing', 'language processing .', 'processing . some', '. some of', 'some of these', 'of these tasks', 'these tasks have', 'tasks have direct', 'have direct real-world', 'direct real-world applications', 'real-world applications ,', 'applications , while', ', while others', 'while others more', 'others more commonly', 'more commonly serve', 'commonly serve as', 'serve as subtasks', 'as subtasks that', 'subtasks that are', 'that are used', 'are used to', 'used to aid', 'to aid in', 'aid in solving', 'in solving larger', 'solving larger tasks', 'larger tasks .', 'tasks . though', '. though natural', 'though natural language', 'natural language processing', 'language processing tasks', 'processing tasks are', 'tasks are closely', 'are closely intertwined', 'closely intertwined ,', 'intertwined , they', ', they are', 'they are frequently', 'are frequently subdivided', 'frequently subdivided into', 'subdivided into categories', 'into categories for', 'categories for convenience', 'for convenience .', 'convenience . a', '. a coarse', 'a coarse division', 'coarse division is', 'division is given', 'is given below', 'given below .', 'below . the', '. the first', 'the first published', 'first published work', 'published work by', 'work by an', 'by an artificial', 'an artificial intelligence', 'artificial intelligence was', 'intelligence was published', 'was published in', 'published in ,', 'in , the', ', the road', 'the road ,', 'road , marketed', ', marketed as', 'marketed as a', 'as a novel', 'a novel ,', 'novel , contains', ', contains sixty', 'contains sixty million', 'sixty million words', 'million words .']\n"
     ]
    }
   ],
   "source": [
    "#function to generate trigrams from sentences\n",
    "from nltk.util import ngrams\n",
    "def extract_ngrams(data,num):\n",
    "    n_grams = ngrams(nltk.word_tokenize(data.lower()), num)\n",
    "    return [' '.join(grams) for grams in n_grams]\n",
    "gram_num = 3\n",
    "n_grams = extract_ngrams(a, gram_num)\n",
    "print(n_grams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Generate N Grams and frequencies**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#function to generate ngrams from mytext\n",
    "def generate_ngrams(text,n):\n",
    " \n",
    "    # split sentences into tokens\n",
    "    tokens=re.split(\"\\\\s+\",text)\n",
    "    ngrams=[]\n",
    " \n",
    "    # collect the n-grams\n",
    "    for i in range(len(tokens)-n+1):\n",
    "       temp=[tokens[j] for j in range(i,i+n)]\n",
    "       ngrams.append(\" \".join(temp))\n",
    " \n",
    "    return ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "#defining a function for computing ngrams from mytext and finding frequency for ngram words\n",
    "def compute_ngrams(tokens,n):\n",
    "    ngram_dict={}\n",
    "    for sentence in tokens:\n",
    "        mytext = re.sub(r'\\d+', '', sentence.lower())\n",
    "        generated_ngrams = ngrams(stopwords_tokens,n)\n",
    "        ngram_dict[sentence] = generated_ngrams\n",
    "        length=len(stopwords_tokens)\n",
    "        p=Counter(generated_ngrams)\n",
    "        for x,y in p.items():\n",
    "            p[x]=y/length\n",
    "            ngram_dict[sentence]=p\n",
    "    return ngram_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter                  #declare Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "#defining functions for computing ngram sentence frequencies \n",
    "def compute_score(computed_ngram_Frequency):\n",
    "    sentencescore={}\n",
    "    for sentence,freq in computed_ngram_Frequency.items():\n",
    "        words_in_sentence = len(freq)\n",
    "        totalscore = 0\n",
    "        for gram,score in freq.items():\n",
    "            totalscore += score\n",
    "            sentencescore[sentence] = totalscore / words_in_sentence\n",
    "    return sentencescore\n",
    "sentence_ngrams=compute_ngrams(sent_tokens,2)\n",
    "sent_scores=compute_score(sentence_ngrams)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Summary Generation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Natural language processing (NLP) is a subfield of linguistics, computer science, information engineering, and artificial intelligence concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data. Challenges in natural language processing frequently involve speech recognition, natural language understanding, and natural language generation. The history of natural language processing (NLP) generally started in the 1950s, although work can be found from earlier periods.\n"
     ]
    }
   ],
   "source": [
    "#summarizing the text based on ngrams sentence frequencies\n",
    "import heapq  \n",
    "summary_sentences = heapq.nlargest(3, sent_scores, key=sent_scores.get)\n",
    "summary = ' '.join(summary_sentences)\n",
    "print(summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **N-GRAM SUMMARY**\n",
    "\n",
    "**Natural language processing (NLP) is a subfield of linguistics, computer science, information engineering, and artificial intelligence concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data. Challenges in natural language processing frequently involve speech recognition, natural language understanding, and natural language generation. The history of natural language processing (NLP) generally started in the 1950s, although work can be found from earlier periods.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **COMPARISON WITH LAB 1**\n",
    "**The summary in Lab 1 was lengthy as compared to N-Grams and has punctuations but the summary for N-Grams is small and precise with minimum punctuations comparatively. It is to the point and precise with n-grams. The N-Gram helps us capture the context information of the data better than TF-IDF approach as it uses the frequencies of the sentences.As Number of N increases, the summary becomes more precise and upto the point** "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}