Part_01_NLPPresentationJobHunt_DemoWordFreq.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "02d10fc3",
   "metadata": {},
   "source": [
    "# Job-Hunt NLP Demo - Part 1\n",
    "\n",
    "Which demo will also be useful in doing some quick NLP work to see how my résumé's word distribution matches that from job descriptions found at Ancestry®.\n",
    "\n",
    "There's a wonderful project out there, [MyBinder](https://mybinder.org), which allows you to interactively run a Jupyter notebook completely online. It's nice to have when you'd like to play with code and see better the outputs that come from running that code. I've had some problems with images going down, but I'm going to work to keep this one up.\n",
    "\n",
    "The link to the online, interactive notebook - the binder - is at the badge you see right here\n",
    "\n",
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/bballdave025/job-app-word-freq/ancestry_hwr?labpath=Part_01_NLPPresentationJobHunt_DemoWordFreq.ipynb)\n",
    "\n",
    "Note that you're seeing the link to the Part 1 Jupyter notebook.\n",
    "\n",
    "<hr/>\n",
    "\n",
    "## We are calling this version 0.1.004\n",
    "\n",
    "It's the Ancestry jobs applied for in October 2023, especially the <i>Sr. Machine Learning Scientist (R002744)</i> working with Handwritten Text Recognition (HTR), but we're splitting it into smaller notebooks. Hopefully, MyBinder can load each more quickly. We'll see how things work with `pickle`ing variables between the parts.\n",
    "\n",
    "<hr/>\n",
    "\n",
    "## What we are doing in Part 1\n",
    "\n",
    "First of all, let's give you a MyBinder badge link which specifies the version and the part.\n",
    "\n",
    "[![Binder](./badge_logo_dwb_v_0-1-004_part_1.png)](https://mybinder.org/v2/gh/bballdave025/job-app-word-freq/ancestry_hwr?labpath=Part_01_NLPPresentationJobHunt_DemoWordFreq.ipynb)\n",
    "\n",
    "The main point of Part 1 is to prepare text from the job description and my résumé. This includes obtaining the text, converting it to a string object, and cleaning it up so as to be more useful both linguistically and computationally. These will later be used to create word-frequency counts for the documents and to compare them. A good match should correspond to a good application. The goodness-of-word-frequency comparison is the main point of this whole project.\n",
    "\n",
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "206f69f9",
   "metadata": {},
   "source": [
    "## Link to setup from the Conda Prompt\n",
    "\n",
    "The instructions for setting up the conda environment from Windows is in I'm figuring out how to make the [MyBinder](https://mybinder.org) server consistent. Now, you sometimes need several tries before it comes up. I might just leave ONLY the setup stuff, so you can look at it there. I've done that, and the server seems a lot more consistent.\n",
    "\n",
    "Here is the binder with just the `conda`/`pip` setup parts\n",
    "\n",
    "[![Binder](./badge_logo_dwb_v_0-1-001_merged_small.png)](https://mybinder.org/v2/gh/bballdave025/job-app-word-freq/original-timed-freq?labpath=CondaSetup_v01_NLP_Presentation_Job_Hunt_NLP.ipynb)\n",
    "\n",
    "If you want to look at my full first try, when I tried to keep myself under a time limit in doing the NLP Presentation, you can look at [this MyBinder](https://mybinder.org/v2/gh/bballdave025/job-app-word-freq/original-timed-freq?labpath=A_v01_NLP_Presentation_Job_Hunt_NLP_Useful_Demo_Word_Freq.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b0d3fc8",
   "metadata": {},
   "source": [
    "## The Third Iteration\n",
    "\n",
    "### Splitting up the long notebook\n",
    "\n",
    "These are comparisons between my résumé for some Ancestry jobs and the job descriptions. For this version, I'm splitting the work into several shorter notebooks that should be more easily handled by MyBinder."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2201e56",
   "metadata": {},
   "source": [
    "## Texts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2d411d8",
   "metadata": {},
   "source": [
    "The text of my résumé for these jobs is in the local file,\n",
    "\n",
    "```\n",
    "res_Ancestry_HTR.txt\n",
    "```\n",
    "\n",
    "The job description is at\n",
    "\n",
    "https://ancestry.wd5.myworkdayjobs.com/en-US/Careers/job/Remote/Sr-Machine-Learning-Scientist_R002744\n",
    "\n",
    "and archived at\n",
    "\n",
    "https://web.archive.org/web/20231024161419/https://ancestry.wd5.myworkdayjobs.com/en-US/Careers/job/Remote/Sr-Machine-Learning-Scientist_R002744\n",
    "\n",
    "I'll pull from the archived version, so there is no problem with anyone trying to reproduce this. I also won't worry about the similar jobs listed at the bottom.\n",
    "\n",
    "I've used archive.org and its Wayback Machine to get an archived version that should stay available.\n",
    "\n",
    "```\n",
    "desc_CJKV_dev3.txt\n",
    "desc_CJKV_dev4.txt\n",
    "desc_CJKV_dev5.txt\n",
    "desc_CJKV_devInTest3.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f0e85c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Try 1 - from the archive\n",
    "\n",
    "# Let's test download, html parsing, and find contractions, numbers\n",
    "\n",
    "source_url = ( r\"https://web.archive.org/web/20230724185343/\"\n",
    "               r\"https://epej.fa.us2.oraclecloud.com/hcmUI/\"\n",
    "               r\"CandidateExperience/en/sites/CX_1001/job/356975/\"\n",
    "             )\n",
    "\n",
    "source_url = ( r\"https://epej.fa.us2.oraclecloud.com/hcmUI/\"\n",
    "               r\"CandidateExperience/en/sites/CX_1001/job/356975/\"\n",
    "             )\n",
    "\n",
    "complete_html_code_response = requests.get(source_url)\n",
    "\n",
    "print(f\"resp: {str(complete_html_code_response)}\")\n",
    "print()\n",
    "print()\n",
    "print()\n",
    "\n",
    "complete_html_as_text = complete_html_code_response.text\n",
    "\n",
    "print(f\"as text:\\n----------\\n{str(complete_html_as_text)}\\n----------\")\n",
    "print()\n",
    "print()\n",
    "print()\n",
    "\n",
    "soup_cleaner = BeautifulSoup(complete_html_as_text, 'html.parser')\n",
    "\n",
    "# Check encoding (and hope it starts out as utf-8)\n",
    "print(f\"original encoding: {str(soup_cleaner.original_encoding)}\")\n",
    "print()\n",
    "print()\n",
    "print()\n",
    "\n",
    "job_desc_init_str = soup_cleaner.body.get_text()\n",
    "\n",
    "print(f\"job_desc_init_str:\\n----------\\n{str(job_desc_init_str)}\\n----------\")\n",
    "print()\n",
    "print()\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5b18797",
   "metadata": {},
   "outputs": [],
   "source": [
    "application_text_filenames = \\\n",
    "  [\"res_Ancestry_HTR.txt\",\n",
    "  ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fb355d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "job_description_text_filenames = \\\n",
    "  [\"desc_CJKV_dev5.txt\",\n",
    "   \"desc_CJKV_dev4.txt\",\n",
    "   \"desc_CJKV_dev3.txt\",\n",
    "   \"desc_CJKV_devInTest3.txt\",\n",
    "  ]\n",
    "\n",
    "# The \"dev5\" is the nicest job - and it's with Java, which I know best."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05ca5e0c",
   "metadata": {},
   "source": [
    "`######################################################`\n",
    "\n",
    "The job description page looks to contain something like `JavaScript`, `ajax`, etc.\n",
    "\n",
    "Rather than writing in a webscraper or looking through the code and finding what gets pulled from the database, I'm just going to copy/paste the text into the text files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a71eaaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "##  Code to get current timestamp, if needed.\n",
    "##+ Meant to be run once, then commented out.\n",
    "# #######################\n",
    "# # No need to run again\n",
    "# #####\n",
    "# !powershell -c (Get-Date -UFormat \"%s_%Y%m%dT%H%M%S%Z00\") -replace '[.][0-9]*_', '_'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43c57519",
   "metadata": {},
   "outputs": [],
   "source": [
    "local_job_desc_filenames = job_description_text_filenames\n",
    "local_job_appl_filenames = application_text_filenames\n",
    "\n",
    "import pprint\n",
    "\n",
    "pprint.pprint(local_job_desc_filenames)\n",
    "print()\n",
    "pprint.pprint(local_job_appl_filenames)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22ceffac",
   "metadata": {},
   "source": [
    "Output was\n",
    "\n",
    "```\n",
    "['desc_CJKV_dev5.txt',\n",
    " 'desc_CJKV_dev4.txt',\n",
    " 'desc_CJKV_dev3.txt',\n",
    " 'desc_CJKV_devInTest3.txt']\n",
    "\n",
    "['res_CJKV.txt']\n",
    "```\n",
    "\n",
    "at `1691423942_20230807T155902-0600`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "417a0dce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# read in texts to original strings\n",
    "\n",
    "def read_in_texts(local_desc_fnames, local_appl_fnames,\n",
    "                  do_combine_desc_files = False,\n",
    "                  do_combine_appl_files = True\n",
    "                 ):\n",
    "    for_pairwise_desc_texts = []\n",
    "    for_pairwise_appl_texts = []\n",
    "    \n",
    "    complete_description_text = \"\"\n",
    "    complete_application_text = \"\"\n",
    "    \n",
    "    if do_combine_desc_files:\n",
    "        complete_description_text = \" \"\n",
    "    ##endof:  if do_combine-desc_files\n",
    "    \n",
    "    if do_combine_appl_files:\n",
    "        complete_application_text = \" \"\n",
    "    ##endof:  if do_combine_appl_files\n",
    "    \n",
    "    for this_description_filename in local_job_desc_filenames:\n",
    "        with open(this_description_filename, 'r', encoding='utf-8') as dfh:\n",
    "            this_desc_file_content_str = dfh.read()\n",
    "            if do_combine_desc_files:\n",
    "                complete_description_text += \" \" + this_desc_file_content_str\n",
    "            else:\n",
    "                this_desc_in_array_str = \" \" + this_desc_file_content_str + \" \"\n",
    "                for_pairwise_desc_texts.append(this_desc_in_array_str)\n",
    "            ##endof:  if/else do_combine_desc_files\n",
    "        ##endof:  with open ... dfh\n",
    "    ##endof:  for this_description_filename in local_job_desc_filenames\n",
    "    \n",
    "    for this_application_filename in local_job_appl_filenames:\n",
    "        with open(this_application_filename, 'r', encoding='utf-8') as afh:\n",
    "            this_appl_file_content_str = afh.read()\n",
    "            if do_combine_appl_files:\n",
    "                complete_application_text += \" \" + this_appl_file_content_str\n",
    "            else:\n",
    "                this_appl_in_array_str = \" \" + this_appl_file_content_str + \" \"\n",
    "                for_pairwise_appl_texts.append(this_appl_in_array_str)\n",
    "            ##endof:  if/else do_combine_appl_files\n",
    "        ##endof:  with open ... afh\n",
    "    ##endof:  for this_application_filename in local_job_appl_filenames\n",
    "    \n",
    "    complete_description_text += \" \"\n",
    "    complete_application_text += \" \"\n",
    "    \n",
    "    if do_combine_desc_files:\n",
    "        for_pairwise_desc_texts = [complete_description_text]\n",
    "    ##endof:  if do_combine-desc_files\n",
    "    \n",
    "    if do_combine_appl_files:\n",
    "        for_pairwise_appl_texts = [complete_application_text]\n",
    "    ##endof:  if do_combine_appl_files\n",
    "    \n",
    "    return for_pairwise_desc_texts, for_pairwise_appl_texts\n",
    "    \n",
    "##endof:  read_in_texts(<params>)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d8cb215",
   "metadata": {},
   "source": [
    "#### This next, make_it_one_line_single_spaced function will be very useful as we go forward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8df24df7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "import re\n",
    "\n",
    "def make_it_one_line_single_spaced(input_str):\n",
    "    processing_str = input_str\n",
    "    \n",
    "    processing_str = ' '.join(processing_str.split())\n",
    "    processing_str = processing_str.replace(\"\\t\", \" \")\n",
    "    processing_str = processing_str.replace(\"\\n\", \" \")\n",
    "    processing_str = re.sub(r\"(^|[^ ])[ ][ ]+($|[^ ])\",\n",
    "                            r\"\\g<1> \\g<2>\",\n",
    "                            processing_str,\n",
    "                            flags=re.IGNORECASE\n",
    "                           )\n",
    "    \n",
    "    return processing_str\n",
    "##endof:  make_it_one_line_single_spaced(input_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd0b2ce5",
   "metadata": {},
   "source": [
    "### The actual reading in of the texts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68ca193f",
   "metadata": {},
   "outputs": [],
   "source": [
    "complete_description_text, complete_application_text = \\\n",
    "                    read_in_texts(local_job_desc_filenames,\n",
    "                                  local_job_appl_filenames)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b27e6c18",
   "metadata": {},
   "source": [
    "### Code for cleaning text\n",
    "\n",
    "We will iterate a bit, so as not to have to write a text normalizer for the whole world. Rather than putting together regexes to test for things like which contractions are there and which other things might need changing (especially things like dashes), I'm doing simple regexes. Q&R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8c83bf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import string\n",
    "\n",
    "#from bs4 import BeautifulSoup\n",
    "#from bs4 import UnicodeDammit\n",
    "\n",
    "def clean_text_string_quickly(input_str):\n",
    "    processing_str = input_str\n",
    "    \n",
    "    # ## one line, single-spaced\n",
    "    # processing_str = ' '.join(processing_str.split())\n",
    "    # processing_str = processing_str.replace(\"\\t\", \" \")\n",
    "    # processing_str = processing_str.replace(\"\\n\", \" \")\n",
    "    # processing_str = re.sub(r\"(^|[^ ])[ ][ ]+($|[^ ])\",\n",
    "    #                         r\"\\g<1> \\g<2>\",\n",
    "    #                         processing_str,\n",
    "    #                         flags=re.IGNORECASE\n",
    "    #                       )\n",
    "    \n",
    "    ## one line, single-spaced\n",
    "    processing_str = make_it_one_line_single_spaced(processing_str)\n",
    "    \n",
    "    \n",
    "    ## get rid of outside-ascii (or control character)\n",
    "    processing_str = re.sub(r\"[^\\u0020-\\u007E]\",\n",
    "                            \" \",\n",
    "                            processing_str,\n",
    "                            flags=re.IGNORECASE\n",
    "                           )\n",
    "    \n",
    "    ## my stuff\n",
    "    processing_str = re.sub(r\"[ ][|]+[ ]\",\n",
    "                            \" \",\n",
    "                            processing_str,\n",
    "                            flags=re.IGNORECASE\n",
    "                           )\n",
    "    processing_str = processing_str.replace(r\"&\", \"and\")\n",
    "    processing_str = processing_str.replace(r\"U.S.\", \"U S \")\n",
    "    \n",
    "    ## get rid of punctuation\n",
    "    processing_str = re.sub(r\"(([^0-9 ])[.,!?:\\\"']([) ]|$))\",\n",
    "                            r\"\\g<2>\\g<3>\",\n",
    "                            processing_str,\n",
    "                            flags=re.IGNORECASE\n",
    "                           )\n",
    "    processing_str = re.sub(r\"(([0-9 ])[.,!?:\\\"']([ ]|$))\",\n",
    "                            r\"\\g<2>\\g<3>\",\n",
    "                            processing_str,\n",
    "                            flags=re.IGNORECASE\n",
    "                           )\n",
    "    # parentheses\n",
    "    processing_str = processing_str.replace(r\"(\", \" \")\n",
    "    processing_str = processing_str.replace(r\")\", \" \")\n",
    "    # dashes\n",
    "    processing_str = re.sub(r\"[ ][-]+[ ]\",\n",
    "                            \" \",\n",
    "                            processing_str,\n",
    "                            flags=re.IGNORECASE\n",
    "                           )\n",
    "    \n",
    "    ##  lowercase - to skip until a few iterations through\n",
    "    ##+ cleaning the text\n",
    "    processing_str = processing_str.casefold()\n",
    "    \n",
    "    ## fixes found by iterating this cleaning function\n",
    "    processing_str = re.sub(r\"[ ][/][ ]\",\n",
    "                            \" \",\n",
    "                            processing_str,\n",
    "                            flags=re.IGNORECASE\n",
    "                           )\n",
    "    \n",
    "    ## What's found in the documents\n",
    "    # My inspection\n",
    "    processing_str = processing_str.replace(r\"s.r\", \"s r\")\n",
    "    processing_str = processing_str.replace(r\"c++/perl\", \"c++ perl\")\n",
    "    \n",
    "    # From the automated looking, below, here for version 0.1.002\n",
    "    processing_str = processing_str.replace(\n",
    "                                 r\"monitors/equipment\", \n",
    "                                  \"monitors equipment\"\n",
    "    )\n",
    "    processing_str = processing_str.replace(\n",
    "                                 r\"product/engineering\", \n",
    "                                  \"product engineering\"\n",
    "    )\n",
    "    processing_str = processing_str.replace(\n",
    "                                 r\"engineering/troubleshooting\", \n",
    "                                  \"engineering troubleshooting\"\n",
    "    )\n",
    "    processing_str = processing_str.replace(\n",
    "                                 r\"engineering/programming\", \n",
    "                                  \"engineering programming\"\n",
    "    )\n",
    "    processing_str = processing_str.replace(\n",
    "                         r\"analytical/diagnostic/troubleshooting\", \n",
    "                          \"analytical diagnostic troubleshooting\"\n",
    "    )\n",
    "    processing_str = processing_str.replace(\n",
    "                                 r\"integration/continuous\", \n",
    "                                  \"integration continuous\"\n",
    "    )\n",
    "    \n",
    "    processing_str = processing_str.replace(r\"net/powershell\", \n",
    "                                                \"net powershell\")\n",
    "    processing_str = processing_str.replace(r\"c/c\", \"c c\")\n",
    "    \n",
    "    processing_str = re.sub(r\"\\b\\w{1}\\b\", \"\", processing_str)\n",
    "    \n",
    "    # KEEP THESE 3 EXAMPLES IN THE CODE FOR COPY/PASTE, WHATEVER\n",
    "    # processing_str = processing_str.replace(r\"notice/more\", \n",
    "    #                                               \"notice more\")\n",
    "    # processing_str = processing_str.replace(r\"s.r\", \"s r\")\n",
    "    # processing_str = processing_str.replace(\n",
    "    #                              r\"monitors/equipment\", \n",
    "    #                               \"monitors equipment\"\n",
    "    # )\n",
    "    \n",
    "    # ##spacing fix at the end\n",
    "    # processing_str = re.sub(r\"(^|[^ ])[ ][ ]+($|[^ ])\",\n",
    "    #                         r\"\\g<1> \\g<2>\",\n",
    "    #                         processing_str,\n",
    "    #                         flags=re.IGNORECASE\n",
    "    #                        )\n",
    "    \n",
    "    ## spacing fix at the end\n",
    "    processing_str = make_it_one_line_single_spaced(processing_str)\n",
    "    \n",
    "    ## Let's give it back\n",
    "    return processing_str\n",
    "\n",
    "##endof:  clean_text_string_quickly(input_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d6bb8f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def remove_stopwords(input_str):\n",
    "    ##  From https://www.nltk.org/book/ch02.html\n",
    "    ##+ > [Stopwords are] high-frequency words like the, to and also that we \n",
    "    ##+ > sometimes want to filter out of a document before further processing. \n",
    "    ##+ > Stopwords usually have little lexical content, and their presence in \n",
    "    ##+ > a text fails to distinguish it from other texts.\n",
    "    \n",
    "    processing_str = input_str\n",
    "    \n",
    "    stopwords_to_remove = [\n",
    "'i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', 'your', 'yours',\n",
    "'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', 'her', 'hers',\n",
    "'herself', 'it', 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves',\n",
    "'what', 'which', 'who', 'whom', 'this', 'that', 'these', 'those', 'am', 'is', 'are',\n",
    "'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does',\n",
    "'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until',\n",
    "'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into',\n",
    "'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down',\n",
    "'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here',\n",
    "'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more',\n",
    "'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so',\n",
    "'than', 'too', 'very', 'can', 'will', 'just', 'should', 'now'\n",
    "]\n",
    "    \n",
    "    # No attempt to optimize code, here. Q&R\n",
    "    for my_stopword in stopwords_to_remove:\n",
    "        #processing_str = processing_str.replace(my_stopword, \" \")\n",
    "        word_with_boundaries = r\"\\b\" + my_stopword + r\"\\b\"\n",
    "        processing_str = re.sub(word_with_boundaries, \" \", \n",
    "                                processing_str, \n",
    "                                flags=re.IGNORECASE)\n",
    "    ##endof:  for my_stopword in stopwords_to_remove\n",
    "    \n",
    "    return processing_str\n",
    "##endof:  remove_stopwords(input_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "caa9f136",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We'll usually keep these two.\n",
    "do_look_at_description_text = True\n",
    "do_look_at_application_text = True\n",
    "\n",
    "#  this one can go (False) if you don't want the big strings\n",
    "#+ i.e. you don't want the complete file contents\n",
    "do_print_the_big_strings = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b98b8eb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "if do_look_at_description_text:\n",
    "    test1 = []\n",
    "    for desc_text_str in complete_description_text:\n",
    "        test1.append(clean_text_string_quickly(desc_text_str))\n",
    "        if do_print_the_big_strings:\n",
    "            import pprint\n",
    "            pprint.pprint(test1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b6a1163",
   "metadata": {},
   "outputs": [],
   "source": [
    "if do_look_at_application_text:\n",
    "    test2 = []\n",
    "    for appl_text_str in complete_application_text:\n",
    "        test2.append(clean_text_string_quickly(appl_text_str))\n",
    "        if do_print_the_big_strings:\n",
    "            import pprint\n",
    "            pprint.pprint(test2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b746d1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Without stopwords\n",
    "test1 = [make_it_one_line_single_spaced(\n",
    "                                remove_stopwords(their_text)\n",
    "                                       ) for their_text in test1\n",
    "        ]\n",
    "\n",
    "if do_print_the_big_strings:\n",
    "    import pprint\n",
    "    pprint.pprint(test1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "400fcd23",
   "metadata": {},
   "outputs": [],
   "source": [
    "test2 = [make_it_one_line_single_spaced(\n",
    "                                remove_stopwords(their_text)\n",
    "                                       ) for their_text in test2\n",
    "        ]\n",
    "\n",
    "if do_print_the_big_strings:\n",
    "    import pprint\n",
    "    pprint.pprint(test2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f03a2ca6",
   "metadata": {},
   "source": [
    "### For these next few cells, we are finding things to search and replace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9dcfaa52",
   "metadata": {},
   "outputs": [],
   "source": [
    "# looking at contractions\n",
    "if do_look_at_description_text:\n",
    "    for their_text in test1:\n",
    "        print()\n",
    "        print(re.findall(r\"\\b('[\\w']+\\b|[\\w']+'[\\w']+|[\\w']+')\\b\",\n",
    "                         their_text)\n",
    "             )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15afe99f",
   "metadata": {},
   "source": [
    "First run-through had\n",
    "\n",
    "```\n",
    "[\"organization's\", \"bachelor's\", \"master's\"]\n",
    "\n",
    "[\"bachelor's\"]\n",
    "\n",
    "[\"bachelor's\"]\n",
    "\n",
    "[\"bachelor's\"]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "511e55de",
   "metadata": {},
   "outputs": [],
   "source": [
    "if do_look_at_application_text:\n",
    "    for my_text in test2:\n",
    "        print()\n",
    "        print(re.findall(r\"\\b('[\\w']+\\b|[\\w']+'[\\w']+|[\\w']+')\\b\", \n",
    "                         my_text)\n",
    "             )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29c0c8af",
   "metadata": {},
   "source": [
    "First run-through had\n",
    "\n",
    "```\n",
    "[\"workplace's\", \"wife's\", \"nist's\", \"container's\", \"mission's\"]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e9c271e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# looking at all slashes\n",
    "if do_look_at_description_text:\n",
    "    for their_text in test1:\n",
    "        print()\n",
    "        print(re.findall(r\"\\b[\\w/]+/[\\w/]+\\b\", \n",
    "                         their_text)\n",
    "             )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcb06226",
   "metadata": {},
   "source": [
    "First run-through had\n",
    "\n",
    "```\n",
    "['monitors/equipment']\n",
    "\n",
    "['product/engineering', 'engineering/troubleshooting', 'monitors/equipment']\n",
    "\n",
    "['monitors/equipment']\n",
    "\n",
    "['engineering/programming', 'analytical/diagnostic/troubleshooting', 'monitors/equipment', 'integration/continuous']\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10eaeff9",
   "metadata": {},
   "outputs": [],
   "source": [
    "if do_look_at_application_text:\n",
    "    for my_text in test2:\n",
    "        print()\n",
    "        print(re.findall(r\"\\b[\\w./]+/[\\w/]+\\b\",\n",
    "                         my_text)\n",
    "             )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "872dd7f6",
   "metadata": {},
   "source": [
    "First run-through had\n",
    "\n",
    "```\n",
    "['github.com/bballdave025', 'stackexchange.com/users/8693193', 'net/powershell', 'c/c']\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90c2d018",
   "metadata": {},
   "source": [
    "### Let's clean things up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "129ca3ca",
   "metadata": {},
   "source": [
    "<b>We'll give you the chance to look at the originals, if you want.</b>\n",
    "\n",
    "<b>Only do the two cells below if you want a big preview! What I'm saying is, \"The two cells below will give you long outputs if uncommented.\"</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f97d508",
   "metadata": {},
   "outputs": [],
   "source": [
    "#complete_description_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb43100b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#complete_application_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52e0aa8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_sorted_word_counts(*cleaned_strings):\n",
    "                           #,\n",
    "                           #do_output_sorted_file=False,\n",
    "                           #sorted_filename=\\\n",
    "                           #    \"sorted_words_from_strings.txt\"):\n",
    "    '''\n",
    "    @return  OrderedDict\n",
    "    '''\n",
    "    \n",
    "    from collections import OrderedDict \n",
    "      # Do I need imports inside the function for pickle(?)\n",
    "    \n",
    "    EXIT_NOWORDSWEREFOUND = -1\n",
    "    \n",
    "    work_with_str = combine_strings(cleaned_strings)\n",
    "    \n",
    "    list_of_words_in_str = work_with_str.split()\n",
    "    \n",
    "    if len(list_of_words_in_str) <= 0:\n",
    "        print(\"No words were found.\", file=sys.stderr)\n",
    "        print(\"The program will exit.\", file=sys.stderr)\n",
    "        #sys.exit(EXIT_NOWORDSWEREFOUND)\n",
    "        return EXIT_NOWORDSWEREFOUND\n",
    "    ##endof:  if len(list_of_words_in_str) <= 0\n",
    "    \n",
    "    word_count_ordered_dict = OrderedDict()\n",
    "    \n",
    "    for this_word in list_of_words_in_str:\n",
    "        if this_word in word_count_ordered_dict:\n",
    "            word_count_ordered_dict[this_word] += 1\n",
    "        else:\n",
    "            word_count_ordered_dict[this_word] = 1\n",
    "        ##endof:  if/else this_word in list_of_words_in_str\n",
    "    ##endof:  for this_word in list_of_words_in_str\n",
    "    \n",
    "    ## DWB note ##\n",
    "    ##  At this point, the OrderedDict is sorted by the\n",
    "    ##+ order in which keys were inserted, not by their\n",
    "    ##+ count.\n",
    "    \n",
    "    for key, _ in \\\n",
    "          sorted(word_count_ordered_dict.items(),\n",
    "                 key=lambda word_and_count: word_and_count[1],\n",
    "                 reverse=True):\n",
    "        word_count_ordered_dict.move_to_end(key)\n",
    "    ##endof:  for myword, _ ...\n",
    "    \n",
    "    return word_count_ordered_dict\n",
    "    \n",
    "##endof:  get_sorted_word_counts(*cleaned_strings)\n",
    "\n",
    "def combine_strings(tuple_of_strings):\n",
    "                    #, \n",
    "                    #do_output_raw_file=False,\n",
    "                    #raw_filename='raw_words_from_strings.txt'):\n",
    "    '''\n",
    "    @return  string\n",
    "    '''\n",
    "    \n",
    "    returned_str = \" \"\n",
    "    \n",
    "    for this_str in tuple_of_strings:\n",
    "        returned_str += this_str + \" \"\n",
    "    ##endof:  for this_str in tuple_of_string\n",
    "    \n",
    "    ## one line, single-spaced\n",
    "    returned_str = ' '.join(returned_str.split())\n",
    "    returned_str = returned_str.replace(\"\\t\", \" \")\n",
    "    returned_str = returned_str.replace(\"\\n\", \" \")\n",
    "    returned_str = re.sub(r\"([^ ])[ ][ ]+($|[^ ])\",\n",
    "                            r\"\\g<1> \\g<2>\",\n",
    "                            returned_str,\n",
    "                            flags=re.IGNORECASE\n",
    "                           )\n",
    "    \n",
    "    return returned_str\n",
    "##endof:  combine_strings(tuple_of_strings)\n",
    "\n",
    "## Some future maybes\n",
    "# @TODO: add a sort-by-word as well as sort-by-count flag\n",
    "# @TODO:  also, print out the pre-sorted and sorted files\n",
    "#       + with word lists, frequency, and in-order-of-\n",
    "#       + highest-count stuff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05a47128",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Cleaning and Counting ##\n",
    "\n",
    "description_strings_pre = \\\n",
    "  [clean_text_string_quickly(this_desc_text_str) \n",
    "           for this_desc_text_str in complete_description_text]\n",
    "application_strings_pre = \\\n",
    "  [clean_text_string_quickly(this_appl_text_str)\n",
    "           for this_appl_text_str in complete_application_text]\n",
    "\n",
    "description_strings = \\\n",
    "  [make_it_one_line_single_spaced(remove_stopwords(this_desc)) \n",
    "           for this_desc in description_strings_pre]\n",
    "application_strings = \\\n",
    "  [make_it_one_line_single_spaced(remove_stopwords(this_appl)) \n",
    "           for this_appl in application_strings_pre]\n",
    "\n",
    "description_word_counts = \\\n",
    "  [get_sorted_word_counts(description_str) \\\n",
    "              for description_str in description_strings]\n",
    "application_word_counts = \\\n",
    "  [get_sorted_word_counts(application_str) \\\n",
    "              for application_str in application_strings]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcb1781c",
   "metadata": {},
   "source": [
    "<b>Once again, the four cells below will give you long outputs if uncommented.</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49c0f225",
   "metadata": {},
   "outputs": [],
   "source": [
    "#description_strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27519a4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#description_word_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbceac80",
   "metadata": {},
   "outputs": [],
   "source": [
    "#application_strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e2f4c5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "#application_word_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e924cc",
   "metadata": {},
   "source": [
    "## We are ready for FormattedWord Frequency Counts in Part 2\n",
    "\n",
    "### First, though, we'll pickle the things we need.\n",
    "\n",
    "And the link for the Part 2 MyBinder will be included after the pickling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae198da2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "pickle_filename_1_to_2 = \"important_part_1_vars.pkl\"\n",
    "\n",
    "things_to_pickle_1 = [\n",
    "    local_job_desc_filenames,\n",
    "    local_job_appl_filenames,\n",
    "    complete_description_text, \n",
    "    complete_application_text,\n",
    "    description_strings,\n",
    "    description_word_counts,\n",
    "    application_strings,\n",
    "    application_word_counts,\n",
    "]\n",
    "\n",
    "with open(pickle_filename_1_to_2, 'wb') as pfh:\n",
    "    pickle.dump(things_to_pickle_1, pfh)\n",
    "##endof:  with open ... as pfh # (pickle file handle)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3478e5ff",
   "metadata": {},
   "source": [
    "[Part 2 On GitHub](https://github.com/bballdave025/job-app-word-freq/blob/ancestry_hwr/Part_02_NLPPresentationJobHunt_DemoWordFreq.ipynb)\n",
    "\n",
    "[Part 2 On MyBinder](https://mybinder.org/v2/gh/bballdave025/job-app-word-freq/ancestry_hwr?labpath=Part_02_NLPPresentationJobHunt_DemoWordFreq.ipynb)\n",
    "\n",
    "Or, alternatively, use the badge as a link for the MyBinder version.\n",
    "\n",
    "[![Binder](./badge_logo_dwb_v_0-1-004_part_2.png)](https://mybinder.org/v2/gh/bballdave025/job-app-word-freq/ancestry_hwr?labpath=Part_02_NLPPresentationJobHunt_DemoWordFreq.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be6bf8cb",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}