Part_03_NLPPresentationJobHunt_DemoWordFreq.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "02d10fc3",
   "metadata": {},
   "source": [
    "# Job-Hunt NLP Demo - Part 3\n",
    "\n",
    "Which demo will also be useful in doing some quick NLP work to see how my résumé's word distribution matches that from job descriptions.\n",
    "\n",
    "There's a wonderful project out there, [MyBinder](https://mybinder.org), which allows you to interactively run a Jupyter notebook completely online. It's nice to have when you'd like to play with code and see better the outputs that come from running that code. I've had some problems with images going down, but I'm going to work to keep this one up.\n",
    "\n",
    "The link to the online, interactive notebook - the binder - is at the badge you see right here\n",
    "\n",
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/bballdave025/job-app-word-freq/polished-5part-presented?labpath=Part_03_NLPPresentationJobHunt_DemoWordFreq.ipynb)\n",
    "\n",
    "<hr/>\n",
    "\n",
    "## We are calling this version 0.1.003\n",
    "\n",
    "It's the FamilySearch CJKV jobs applied for in August 2023, but we're splitting it into smaller notebooks. Hopefully, MyBinder can load each more quickly. We'll see how things work with pickling variables between the parts.\n",
    "\n",
    "<hr/>\n",
    "\n",
    "## What we are doing in Part 3\n",
    "\n",
    "First of all, let's give you a MyBinder badge link which specifies the version and the part.\n",
    "\n",
    "[![Binder](./badge_logo_dwb_v_0-1-003_part_3.png)](https://mybinder.org/v2/gh/bballdave025/job-app-word-freq/polished-5part-presented?labpath=Part_03_NLPPresentationJobHunt_DemoWordFreq.ipynb)\n",
    "\n",
    "@TODO : write a few dinglehoppers about what we're doing in Part 3\n",
    "\n",
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dbd5a12",
   "metadata": {},
   "source": [
    "## Let's start by un-pickle-ing the things we'll need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae198da2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "pickle_filename_2_to_3 = \"important_part_2_vars.pkl\"\n",
    "\n",
    "unpickled_array = []\n",
    "\n",
    "with open(pickle_filename_2_to_3, 'rb') as pfh:\n",
    "    unpickled_array = pickle.load(pfh)\n",
    "##endof:  with open ... as pfh # (pickle file handle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b80b50d",
   "metadata": {},
   "outputs": [],
   "source": [
    "local_job_desc_filenames = unpickled_array[0]\n",
    "local_job_appl_filenames = unpickled_array[1]\n",
    "list_of_display_table_desc = unpickled_array[2]\n",
    "list_of_display_table_appl = unpickled_array[3]\n",
    "description_word_counts = unpickled_array[4]\n",
    "application_word_counts = unpickled_array[5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b41410d4",
   "metadata": {},
   "source": [
    "## Seeing words at certain ranks and comparing description & application"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b865afb",
   "metadata": {},
   "source": [
    "### First, a review of where we've been @TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22ac3a13",
   "metadata": {},
   "source": [
    "@TODO Review what we've done."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74cfb514",
   "metadata": {},
   "source": [
    "Good things are what we've done."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "820e7922",
   "metadata": {},
   "source": [
    "## The Top-25 From Both"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "673bf68b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_description_word_at_rank(this_rank = 1, \n",
    "                                 this_desc_fname_idx=0,\n",
    "                                 do_print_details=False\n",
    "                                ):\n",
    "    this_idx = this_rank # the header is index 0\n",
    "    this_table_to_use = \\\n",
    "      list_of_display_table_desc[this_desc_fname_idx]\n",
    "    this_word = this_table_to_use[this_rank][0]\n",
    "    if do_print_details:\n",
    "        print()\n",
    "        print(f\"  The job description word at rank {this_rank},\")\n",
    "        print( \n",
    "          ( \"  from file:\"\n",
    "           f\" '{local_job_desc_filenames[this_desc_fname_idx]}',\"\n",
    "          )\n",
    ")\n",
    "        print(f\"  is '{this_word}'.\")\n",
    "        print()\n",
    "    ##endof: if do_print_details\n",
    "    \n",
    "    return this_word\n",
    "##endof:  get_description_word_at_rank(<params>)\n",
    "\n",
    "def get_application_word_at_rank(this_rank = 1, \n",
    "                                 do_print_details=False\n",
    "                                ):\n",
    "    this_idx = this_rank # the header is index 0\n",
    "    this_appl_fname_idx=0\n",
    "    this_table_to_use = \\\n",
    "      list_of_display_table_appl[this_appl_fname_idx]\n",
    "    this_word = this_table_to_use[this_rank][0]\n",
    "    if do_print_details:\n",
    "        print()\n",
    "        print(f\"  The job application word at rank {this_rank},\")\n",
    "        print( \n",
    "          ( \"  from file:\"\n",
    "           f\" '{local_job_appl_filenames[this_appl_fname_idx]}',\"\n",
    "          )\n",
    ")\n",
    "        print(f\"  is '{this_word}'.\")\n",
    "        print()\n",
    "    ##endof: if do_print_details\n",
    "    \n",
    "    return this_word\n",
    "##endof:  get_description_word_at_rank(<params>)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df466e57",
   "metadata": {},
   "outputs": [],
   "source": [
    "get_description_word_at_rank(1, do_print_details=True);\n",
    "get_application_word_at_rank(1, do_print_details=True);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db47d24b",
   "metadata": {},
   "source": [
    "## Time for top-25 histograms (or whatever the discretized version is)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca502880",
   "metadata": {},
   "source": [
    "I'm going to go through these histograms one at a time. Basically, I'll compare each of the four job descriptions to my job application."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1fc230c",
   "metadata": {},
   "source": [
    "### Choices for the job description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d4a1fb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "str_for_choices = f\"Choices are any of: {list(range(len(local_job_desc_filenames)))}\"\n",
    "print(str_for_choices.replace(r\"[\", r\"{\").replace(r\"]\", r\"}\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d15a78e",
   "metadata": {},
   "source": [
    "Output was most recently\n",
    "\n",
    "Choices are any of: `{0, 1, 2, 3}`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "259d2ea4",
   "metadata": {},
   "source": [
    "### Make your choice in the next cell, if you want to see only one description\n",
    "\n",
    "Otherwise, you should leave this index as `0`, as it's part of my process of going through all four job descriptions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2359396",
   "metadata": {},
   "outputs": [],
   "source": [
    "##  It's your turn to choose which one you want.\n",
    "##+ Just do this if you are pressed for time and\n",
    "##+ want to see a certain result; I will be displaying\n",
    "##+ all four, here.\n",
    "the_choice_of_description_index = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60dbc3e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "desc_idx_00 = the_choice_of_description_index # smaller variable name.\n",
    "the_chosen_filename = local_job_desc_filenames[desc_idx_00]\n",
    "print(f\"We will be looking at: {the_chosen_filename}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08225819",
   "metadata": {},
   "source": [
    "### One value for the job application\n",
    "\n",
    "This is how I want to structure things in general. Even if I have a résumé and a cover letter and a list of skills from the application and whatever questions they want me to answer, I want to combine them. That is possible in one of the functions above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b870b6a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "#  You can't choose a value for now (or it least doing \n",
    "#+ so won't give you anything useful).\n",
    "the_only_application_index_value = 0\n",
    "appl_idx = the_only_application_index_value\n",
    "the_only_application_filename = local_job_appl_filenames[appl_idx]\n",
    "\n",
    "print(f\"And the comparison will be to: {the_only_application_filename}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2186063a",
   "metadata": {},
   "source": [
    "### Code for one histogram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1847441",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "## Next line only for Jupyter notebook.\n",
    "%matplotlib inline\n",
    "\n",
    "def get_histo_from_freq_dict(word_count_ordered_dict,\n",
    "                             n_top_words = 25,\n",
    "                             do_show_frac_not_count=False,\n",
    "                             do_show_wd_cnt_or_frac_lists=False,\n",
    "                             axx=None\n",
    "                            ):\n",
    "    '''\n",
    "    @return  an axis from matplotlab (with the object - histogram - in it)\n",
    "    '''\n",
    "    \n",
    "    if axx is None:\n",
    "        fig = plt.figure(figsize=(10, 3))\n",
    "        axx = fig.add_subplot(111)\n",
    "    ##endof:  if axx is None\n",
    "    \n",
    "    counts = None\n",
    "    fractions = None\n",
    "    \n",
    "    if do_show_frac_not_count: \n",
    "        frac_wd_cnt_list_of_tuples = \\\n",
    "          [ ( k, (v / len(word_count_ordered_dict)) )\n",
    "                  for k, v in word_count_ordered_dict.items()]\n",
    "        fractions_pre = [ this_item[1] \n",
    "                           for this_item in frac_wd_cnt_list_of_tuples ]\n",
    "        \n",
    "        fractions = fractions_pre[:n_top_words]\n",
    "    else:\n",
    "        counts_pre = list(word_count_ordered_dict.values())\n",
    "        counts = counts_pre[:n_top_words]\n",
    "    ##endof:  if/else do_show_frac_not_count\n",
    "    \n",
    "    words_pre  = list(word_count_ordered_dict.keys())\n",
    "    words  = words_pre[:n_top_words]\n",
    "    \n",
    "    ## making sure things were working\n",
    "    if do_show_wd_cnt_or_frac_lists:\n",
    "        if do_show_frac_not_count:\n",
    "            print (f\"fractions: {fractions}\")\n",
    "        else:\n",
    "            print(f\"counts: {counts}\")\n",
    "        ##endof:  if do_show_frac_not_count\n",
    "        print(f\"words:  {words}\")\n",
    "    ##endof:  if do_show_word_and_count_lists\n",
    "    \n",
    "    x_words_coords = np.arange(len(words))\n",
    "    \n",
    "    if do_show_frac_not_count:\n",
    "        axx.bar(x_words_coords, fractions, align='center')\n",
    "    else:\n",
    "        axx.bar(x_words_coords, counts, align='center')\n",
    "    ##endof:  if/else do_show_frac_not_count\n",
    "    \n",
    "    axx.set_xticks(x_words_coords)\n",
    "    axx.set_xticklabels(words, rotation=45, ha='right')\n",
    "    \n",
    "##endof:  get_histo_from_freq_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eef44ce",
   "metadata": {},
   "source": [
    "### Let's see a histogram for the job description with word counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d223537a",
   "metadata": {},
   "outputs": [],
   "source": [
    "get_histo_from_freq_dict(description_word_counts[desc_idx_00],\n",
    "                         do_show_wd_cnt_or_frac_lists=True)\n",
    "\n",
    "desc_top_25_hist_fname = \"top_25_description_words.png\"\n",
    "plt.savefig(desc_top_25_hist_fname,\n",
    "            bbox_inches='tight')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a1c0ec9",
   "metadata": {},
   "source": [
    "### Now, let's see one for the job description with word frequency as a fraction of total words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e23d64e",
   "metadata": {},
   "outputs": [],
   "source": [
    "get_histo_from_freq_dict(description_word_counts[desc_idx_00],\n",
    "                         do_show_frac_not_count=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4b047b4",
   "metadata": {},
   "source": [
    "### Here comes the histogram for the job application with word counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6195381",
   "metadata": {},
   "outputs": [],
   "source": [
    "get_histo_from_freq_dict(application_word_counts[appl_idx],\n",
    "                         do_show_wd_cnt_or_frac_lists=True)\n",
    "\n",
    "appl_top_25_hist_fname = \"top_25_application_words.png\"\n",
    "plt.savefig(appl_top_25_hist_fname,\n",
    "            bbox_inches='tight')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d6e897d",
   "metadata": {},
   "source": [
    "### And the histogram for the job application  with word frequency as a fraction of total words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a916bea",
   "metadata": {},
   "outputs": [],
   "source": [
    "get_histo_from_freq_dict(application_word_counts[appl_idx],\n",
    "                         do_show_frac_not_count=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37b5c149",
   "metadata": {},
   "source": [
    "### Change the img src values and img alt values, then see the histograms together\n",
    "\n",
    "You might need to double-click on the image to get the html source. You also might need to click on the image (or image-not-found icon) just to make things reload."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f7b313c",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"  img src values for the two images:\")\n",
    "print(f'\"{desc_top_25_hist_fname}\"')\n",
    "print(f'\"{appl_top_25_hist_fname}\"')\n",
    "\n",
    "print()\n",
    "print(\"  img alt values for the two images:\")\n",
    "wd_count_alt_text_1 = '\"The histogram for the job description with word frequencies\"'\n",
    "wd_count_alt_text_2 = '\"The histogram for the job application with word frequencies\"'\n",
    "\n",
    "print(wd_count_alt_text_1)\n",
    "print(wd_count_alt_text_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1aa4275d",
   "metadata": {},
   "source": [
    "The output histograms, stacked for easier view.\n",
    "\n",
    "_Remember that you might need to double click on the images to change the img src and img alt values._\n",
    "\n",
    "<br/>\n",
    "<div>\n",
    "  <img src=\"top_25_description_words.png\"\n",
    "       alt=\"The histogram for the job description with word frequencies\"\n",
    "       width=\"auto\">\n",
    "</div>\n",
    "<br/>\n",
    "\n",
    "<br/>\n",
    "<div>\n",
    "  <img src=\"top_25_application_words.png\"\n",
    "       alt=\"The histogram for the job application with word frequencies\"\n",
    "       width=\"auto\">\n",
    "</div>\n",
    "<br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "608b547f",
   "metadata": {},
   "source": [
    "Sometimes, I'll grab a printscreen of the above two images and draw green lines between words that match. However, from the time when I allowed the view of the match and three surrounding words, this step hasn't seemed as vital.\n",
    "\n",
    "If this is going to happen, double click on this cell to see the now-commented HTML, get your saved filename, change the HTML accordingly, and uncomment everything. (HTML Comments start with `<!--` and end with `-->`\n",
    "\n",
    "<!--\n",
    "<br/>\n",
    "<div>\n",
    "  <img src=\"word_frequency_plots_w_link_lines.jpg\"\n",
    "       alt=\"Word matches for the pair of histograms.\"\n",
    "       width=\"100%\">\n",
    "</div>\n",
    "<br/>\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be691ca8",
   "metadata": {},
   "source": [
    "## A better way to compare words in the documents - in Part 4\n",
    "\n",
    "### Well, for me, it will at least be better to be able to calculate a metric from it,\n",
    "\n",
    "### and I think the visualizations tell more about how the documents compare\n",
    "\n",
    "In the next section, rather than using lines between words that match, a messy thing that ends up looking something like the image that follows.\n",
    "\n",
    "<br/>\n",
    "<div>\n",
    "  <img src=\"first_QandR_word_frequency_plots_w_link_lines.jpg\"\n",
    "       alt=\"Two word-count histograms with lines marking the same word from each document.\"\n",
    "       width=\"300px\">\n",
    "</div>\n",
    "<br/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4100246e",
   "metadata": {},
   "source": [
    "### But first, as usual, it's pickle time.\n",
    "\n",
    "And the link for the Part 4 MyBinder will be included after the pickling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13f1d2c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "pickle_filename_3_to_4 = \"important_part_3_vars.pkl\"\n",
    "\n",
    "things_to_pickle_3 = [\n",
    "    local_job_desc_filenames,\n",
    "    local_job_appl_filenames,\n",
    "    description_word_counts,\n",
    "    application_word_counts,\n",
    "    list_of_display_table_desc\n",
    "]\n",
    "\n",
    "with open(pickle_filename_3_to_4, 'wb') as pfh:\n",
    "    pickle.dump(things_to_pickle_3, pfh)\n",
    "##endof:  with open ... as pfh # (pickle file handle)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebff28ba",
   "metadata": {},
   "source": [
    "[Part 4 On GitHub](https://github.com/bballdave025/job-app-word-freq/blob/polished-5part-presented/Part_04_NLPPresentationJobHunt_DemoWordFreq.ipynb)\n",
    "\n",
    "[Part 4 On MyBinder](https://mybinder.org/v2/gh/bballdave025/job-app-word-freq/polished-5part-presented?labpath=Part_04_NLPPresentationJobHunt_DemoWordFreq.ipynb)\n",
    "\n",
    "Or, alternatively<strike>/eventually</strike>, use the badge as a link for the MyBinder version.\n",
    "\n",
    "[![Binder](./badge_logo_dwb_v_0-1-003_part_4.png)](https://mybinder.org/v2/gh/bballdave025/job-app-word-freq/polished-5part-presented?labpath=Part_04_NLPPresentationJobHunt_DemoWordFreq.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50ac1b42",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}