Sparkify.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sparkify Project Workspace\n",
    "This workspace contains a tiny subset (128MB) of the full dataset available (12GB). Feel free to use this workspace to build your project, or to explore a smaller subset with Spark before deploying your cluster on the cloud. Instructions for setting up your Spark cluster is included in the last lesson of the Extracurricular Spark Course content.\n",
    "\n",
    "You can follow the steps below to guide your data analysis and model building portion of this project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import libraries\n",
    "import re\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "import pyspark\n",
    "from pyspark.ml.evaluation import MulticlassClassificationEvaluator\n",
    "from pyspark.sql.types import IntegerType\n",
    "from pyspark.ml.classification import LogisticRegression, GBTClassifier,LinearSVC\n",
    "from pyspark.sql.functions import col, udf\n",
    "from pyspark.ml.feature import VectorAssembler, StandardScaler\n",
    "from pyspark.ml.tuning import CrossValidator, ParamGridBuilder\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a Spark session\n",
    "spark  = pyspark.sql.SparkSession.builder.getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load and Clean Dataset\n",
    "In this workspace, the mini-dataset file is `mini_sparkify_event_data.json`. Load and clean the dataset, checking for invalid or missing data - for example, records without userids or sessionids. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = spark.read.json(\"mini_sparkify_event_data.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- artist: string (nullable = true)\n",
      " |-- auth: string (nullable = true)\n",
      " |-- firstName: string (nullable = true)\n",
      " |-- gender: string (nullable = true)\n",
      " |-- itemInSession: long (nullable = true)\n",
      " |-- lastName: string (nullable = true)\n",
      " |-- length: double (nullable = true)\n",
      " |-- level: string (nullable = true)\n",
      " |-- location: string (nullable = true)\n",
      " |-- method: string (nullable = true)\n",
      " |-- page: string (nullable = true)\n",
      " |-- registration: long (nullable = true)\n",
      " |-- sessionId: long (nullable = true)\n",
      " |-- song: string (nullable = true)\n",
      " |-- status: long (nullable = true)\n",
      " |-- ts: long (nullable = true)\n",
      " |-- userAgent: string (nullable = true)\n",
      " |-- userId: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.createOrReplaceTempView(\"df_log\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+\n",
      "|level|\n",
      "+-----+\n",
      "| free|\n",
      "| paid|\n",
      "+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "spark.sql(\"SELECT DISTINCT level FROM df_log\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+------+\n",
      "|                page|   num|\n",
      "+--------------------+------+\n",
      "|            NextSong|228108|\n",
      "|                Home| 14457|\n",
      "|           Thumbs Up| 12551|\n",
      "|     Add to Playlist|  6526|\n",
      "|          Add Friend|  4277|\n",
      "|         Roll Advert|  3933|\n",
      "|               Login|  3241|\n",
      "|              Logout|  3226|\n",
      "|         Thumbs Down|  2546|\n",
      "|           Downgrade|  2055|\n",
      "|                Help|  1726|\n",
      "|            Settings|  1514|\n",
      "|               About|   924|\n",
      "|             Upgrade|   499|\n",
      "|       Save Settings|   310|\n",
      "|               Error|   258|\n",
      "|      Submit Upgrade|   159|\n",
      "|    Submit Downgrade|    63|\n",
      "|              Cancel|    52|\n",
      "|Cancellation Conf...|    52|\n",
      "+--------------------+------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "spark.sql(\"SELECT page, COUNT(*) AS num FROM df_log GROUP BY page ORDER BY num DESC\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### I will use the cancellation confirmation page to identify whether the user churn or not"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+----------+\n",
      "|userID|occurrence|\n",
      "+------+----------+\n",
      "|      |      8346|\n",
      "|    10|       795|\n",
      "|   100|      3214|\n",
      "|100001|       187|\n",
      "|100002|       218|\n",
      "|100003|        78|\n",
      "|100004|      1245|\n",
      "|100005|       216|\n",
      "|100006|        44|\n",
      "|100007|       520|\n",
      "|100008|       940|\n",
      "|100009|       671|\n",
      "|100010|       381|\n",
      "|100011|        23|\n",
      "|100012|       600|\n",
      "|100013|      1392|\n",
      "|100014|       310|\n",
      "|100015|      1050|\n",
      "|100016|       638|\n",
      "|100017|        75|\n",
      "+------+----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "spark.sql(\"SELECT DISTINCT userID, COUNT(*) AS occurrence FROM df_log GROUP BY userID ORDER BY userID ASC\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### There are  some missing userIDs "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+----------+\n",
      "|sessionID|occurrence|\n",
      "+---------+----------+\n",
      "|        1|       714|\n",
      "|        2|         7|\n",
      "|        3|        38|\n",
      "|        4|        24|\n",
      "|        5|        58|\n",
      "|        6|        71|\n",
      "|        7|         1|\n",
      "|        8|        49|\n",
      "|        9|       197|\n",
      "|       10|       176|\n",
      "|       11|        47|\n",
      "|       12|        60|\n",
      "|       13|       122|\n",
      "|       15|       220|\n",
      "|       16|        39|\n",
      "|       17|       171|\n",
      "|       18|        30|\n",
      "|       19|       158|\n",
      "|       20|        29|\n",
      "|       21|       138|\n",
      "+---------+----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "spark.sql(\"SELECT DISTINCT sessionID, COUNT(*) as occurrence FROM df_log GROUP BY sessionID ORDER BY sessionID ASC\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### There is no missing sessionID\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.filter(df.userId != '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "278154"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploratory Data Analysis\n",
    "When you're working with the full dataset, perform EDA by loading a small subset of the data and doing basic manipulations within Spark. In this workspace, you are already provided a small subset of data you can explore.\n",
    "\n",
    "### Define Churn\n",
    "\n",
    "Once you've done some preliminary analysis, create a column `Churn` to use as the label for your model. I suggest using the `Cancellation Confirmation` events to define your churn, which happen for both paid and free users. As a bonus task, you can also look into the `Downgrade` events.\n",
    "\n",
    "### Explore Data\n",
    "Once you've defined churn, perform some exploratory data analysis to observe the behavior for users who stayed vs users who churned. You can start by exploring aggregates on these two groups of users, observing how much of a specific action they experienced per a certain time unit or number of songs played."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DataFrame[artist: string, auth: string, firstName: string, gender: string, itemInSession: bigint, lastName: string, length: double, level: string, location: string, method: string, page: string, registration: bigint, sessionId: bigint, song: string, status: bigint, ts: bigint, userAgent: string, userId: string, churn: int]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Define Churn\n",
    "churn_event = udf(lambda x: 1 if x == \"Cancellation Confirmation\" else 0, IntegerType())\n",
    "df_churn = df.withColumn(\"churn\", churn_event(\"page\"))\n",
    "df_churn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_churn.createOrReplaceTempView(\"df_churn_log\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------+\n",
      "|num_user_churn|\n",
      "+--------------+\n",
      "|            52|\n",
      "+--------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "spark.sql(\"SELECT COUNT(DISTINCT userID) AS num_user_churn FROM df_churn_log WHERE churn=1\").show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------+\n",
      "|num_user|\n",
      "+--------+\n",
      "|     225|\n",
      "+--------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "spark.sql(\"SELECT COUNT(DISTINCT userID) AS num_user FROM df_churn_log\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(artist='Sleeping With Sirens', auth='Logged In', firstName='Darianna', gender='F', itemInSession=0, lastName='Carpenter', length=202.97098, level='free', location='Bridgeport-Stamford-Norwalk, CT', method='PUT', page='NextSong', registration=1538016340000, sessionId=31, song='Captain Tyin Knots VS Mr Walkway (No Way)', status=200, ts=1539003534000, userAgent='\"Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53\"', userId='100010', churn=0, churn_d=0)]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_log_new = spark.sql(\"SELECT a.*, b.churn_d\\\n",
    "                          FROM \\\n",
    "                              df_churn_log AS a \\\n",
    "                          INNER JOIN \\\n",
    "                              (SELECT userID, SUM(churn) AS churn_d \\\n",
    "                               FROM df_churn_log \\\n",
    "                               GROUP BY userID) AS b \\\n",
    "                          ON a.userID=b.userID\")\n",
    "data_log_new.createOrReplaceTempView(\"data_log_new\")\n",
    "data_log_new.take(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+------------------+\n",
      "|churn_d|  avg(listen_time)|\n",
      "+-------+------------------+\n",
      "|      0|276166.93746780336|\n",
      "|      1| 174014.2685513462|\n",
      "+-------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#avg time to listen\n",
    "p1 = data_log_new \\\n",
    "        .select('userID','length','churn_d') \\\n",
    "        .groupBy('userID','churn_d') \\\n",
    "        .agg({'length':'sum'}) \\\n",
    "        .withColumnRenamed('sum(length)', 'listen_time') \\\n",
    "        .select('churn_d','listen_time') \\\n",
    "        .groupBy('churn_d') \\\n",
    "        .agg({'listen_time':'avg'})\n",
    "p1.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbQAAAEWCAYAAAAO4GKjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAHkpJREFUeJzt3Xu8VVW99/HPF1A0JW+gIhnbu2mZIppmKZZZiqWkpaaJaRknrWPZheo55/FUdrSOHfXok7cM85ZaYaSZhMrFygsgIuhBTfGOioqKtwx/zx9jLJku19qbvdnstRl+36/XejHXnHONOcaca6/vGmNO5lJEYGZmtrLr0+oKmJmZdQcHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmb2uS3i1psaS+ra5LM5IOkzSx1fUw6+0caNaQpMmSnpXUv9V16Q65PV+snx8RD0XEmhGxpIPXHynpphVUt7k5VBdLWiLplcrz70XEJRGx94rYdoO6hKQX87YXSrpM0trL+NoVto+6g6SzK/v1H5Jeqzy/dgVut7+kH0i6V9JLkh6RdLWkj6yobb5dOdDsLSS1AR8GAvjUCtpGvxVR7sooIrbNobomMA04rvY8In7cgiq9P9dlU2Ad4MQW1KHbRcSYyn7+MXB5ZT/vsyK2KUnABGBv4FBgLWAz4OfAyCav8d9GFznQrJEjgJuBccDo2kxJu0haUB2ekzRK0uw83UfSWEl/l/S0pCskrZuXteVv/0dLegi4Ic+/Mpf5nKSpkratlL2epD9Iel7SbZJ+VO0BSNpa0p8lPSNpnqTPdrahlXr1y8+PlHS/pBckPZCH+94DnA3smr/NL8rr9pf0X5IekvRE7gGsnpeNyN/ET5D0pKTHJX2hs/Wr1Kna7pD0lfyN/wVJP5S0maS/5X11haRVK+vvJ2mWpEWS/ippu2XZbkQ8T/ow3qZS1lqSfpHb82g+Jn0b7SNJm+R/++TXni/pyUpZF0s6vr1yK+seJelupVGD6yQNrdsfY/L+eFbSWTlIOrufL5d0bJ7ePJd7VH7+XkkLKuseW3mf/07SBk2KHQl8ENg/IqZHxGsR8WpEXBMRJ1TKWyDpm5LmAs/nee+TNC3vw9mS9qmsf7OkwyvPx0ialKdXy3U/TtJ8SU9JOqm2T/LfzU35b+4pSb/q7L7qrRxo1sgRwCX58fHaH2tE3Ay8CFSHSj4HXJqnvwYcAOwBbAQ8C5xVV/YewHuAj+fn1wJbAOsDM/M2a87K29uQFKzVcF0D+HPe9vqkb7//rxqInZXLPAPYJyIGkD6IZkXE3cAY4G/523xtCO4UYEtge2BzYAjw75UiNyR9Ix8CHA2cJWmdrtavzieAHYFdgG8D5wKHARsD7yXtDyQNAy4AvgysB5wDTNAyDCXnuh5A+nJTcyHwT1J7dyD1PL7YaB9FxAOkD+cd8ms/DCzO4QewOzClvXJzPQ4Avgd8GhhE6sVeVlfd/YCdgPcDn2Xp+6szpgAjKnW7n/R+fVNdJe0L/BswinRsFwIXNylzL+AvEfHEMmz/YOBjwHqSVgOuBq4itflbwJWSNulEez5Jem/uTHo/HJbn/2cud23g3aT3RBkiwg8/3ngAHwJeAwbm5/8LfL2y/EfABXl6AClwhubndwMfraw7OJfVD2gjDWFu2s62187rrAX0za/dqm7bN+Xpg4Fpda8/B/i/TcqeTPrgrZ9fq1c/YA1gEXAgsHrdekfWtp2fK7d9s8q8XYEH8vQI4GWgX2X5k8AuHez/t9SzwbYD2K3yfAbwncrzU4HT8vTPgR/WlTcP2KPJ9oMUQouAJfn4D8nLNgBere4b0gfljY3qmeddBHyDFO7zgJ+Qgm+TvI0+y1DutcDRlWV9gJcq77sAPlRZfgUwtoP9fCJwcd28bYEn8vQ44EuV43k58JU8fQnwg7r37evAhg22czEwrvJ8o9zu54BFlfkLgM9Vnn8MeBBQZd74WrtIXzIOrywbA0zK06vlfTKisvwbwDWV/XMmMLijz4OV7eEemtUbDUyMiIX5+aVUekb5+afzN/xPAzMj4sG8bCgwPg+RLCIF3BLSB1bNw7WJPFR1ch66eR6YnxcNJH0r7Vddv256KPCB2rby9g4jfXB2SUS8SArKMcDjkq6RtHWT1QcB7wBmVLb/pzy/5umI+Gfl+UvAml2tX53qN/6XGzyvbWcocELdftqY9MHazLBIvdDVSIE4LfcYhgKrkPZNraxzSD3kZmq9nt2BqaTA3iM/pkXE68tQ7lDg9MqyZ0hfKIZUtrOgMt3V/XwX0EfSNqQvduOBF/LwZrU3uREpbACIiEWkLwFDeKunSV/saus+lvftbqT9W1V9f28EPBQ5gbIHm2yjmWp5D7L0mH+d9N69PQ9lHv6WV66kfPLR3qB0/uezQN/K+YL+wNqS3h8Rd0TEXZIeBPbhzcONkP6AjoqIvzQouy1PVv9APwfsTxqWmU/qmT1L+rB6ijQE9S7gnrz+xnXbmhIRH+tSY5uIiOuA6/K++BFwHksvkKlaSAqObSPi0e6sQzd7GDgpIk7q7Asj4jVJ5wOnkYYxHyb1pAbWBfUbL2kwbwrwU+CRPH0T6VzbKywNiI7KrbXhkgbLuk1EhKSppC9Gr0TEQklTSMO1/UiBB/AYKWSBdP4PeCfQ6H1wPXCUpA2i42HH6v57jDQcWPVu4NY8/SIplGoafZHbGPh75bWPAeT361H5nNoewERJUyPioQ7q1+u5h2ZVB5B6VNuQxt63J53vmkY6r1ZzKel82e7AlZX5ZwMn1U7YSxokaf92tjeA9EH2NOmP840r+iJdRv874ERJ78g9pWodrga2lPR5Savkx06V8zON9MsnzGuPVaoLJW0g6VP5XNqrwOK8PyD1gN6lfLFF7lmcB/y3pPXz64dI6sq5mxXpPGCMpA8oWUPSSEkDOnqh0kUZXyAF9/0R8TgwEThV0juVLgLaTFLtPNOb9hFARNybX384MDXShSZPkIZ1p+R1Oir3bOC7tfOjSheQfGZ5d0wTU4CvsjRsJwPH5brXAucy4EtKF4qsRjqXekNELKgvjPQ+vQX4vaTh+X26KvCBDuoxjdRbPF5SP0kfI51XrP29zQIOyu/jrUnDvfW+k/dVW27D5QCSDpa0UW7Porxuoy8SKx0HmlWNBn4Z6f9mLag9SOPth2np5cSXkYaRbqgMTQKcTroqbqKkF0jj/O394f6KNBTyKOnb7811y48j9doWkM7FXEYKGiLiBdIf+CGkb54LSB8s7V3s8HPSh2vt8cu65X2AE3J5z5C+vX4lL7sBmAsskFRr83eA+4Cb85DpJGCrdrbf4yJiOulc0Jmk3u99NP7wq7pD0uK8/mhgVEQ8k5cdAaxKOl7PAr9h6ZBao30EKRyervQAppB64bdX1mlabkSMJx3bX+f9PIc0QrAiTCF90Zqan08lDV/WnhMRV5MurJhAeq9sCHy+UWH5i89+pPfG5aRzZ38nDdfv26wSEfFKft1BpC98PwMOjohaj+snpF7jU6QLghpdlHINcAcwnRSEtXV2JQ2VL87zj4mIx5rVZWWiNw/RmvVekk4hnXgf3eHKZm9Tudf4MrBxRDzS6vr0JPfQrNfK/19muzxUtjPp0vfxra6XmfVOvijEerMBpGHGjUiXvJ8K/L6lNTKzXstDjmZmVgQPOZqZWRE85NiDBg4cGG1tba2uhpnZSmXGjBkLI2JQR+s50HpQW1sb06dPb3U1zMxWKvlmDh3ykKOZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBP/H6h5056PP0Tb2mlZXw2ylMf/kka2ugq1E3EMzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrQ6wNN0oaSfi3p75LukvRHSVv2wHYXd3L9EyV9c0XVx8zM2terA02SgPHA5IjYLCK2Ab4HbNDampmZWW/TqwMN2BN4LSLOrs2IiFnA7ZKulzRT0p2S9geQ1CbpbknnSZoraaKk1fOyzSVNknRHft1mef63JN0mabak/2hUiWbrSPq+pHmSJgFbrcD9YGZmHejtgfZeYEaD+a8AoyJiGCn0Ts29OYAtgLMiYltgEXBgnn9Jnv9+4IPA45L2zuvvDGwP7Chp9+qGmq0jaUfgEGAH4NPATo0aIOkYSdMlTV/y0nNd2glmZtaxfq2uQBcJ+HEOn9eBISwdhnwg9+IghWGbpAHAkIgYDxARr8AbYbU3cHtef01SeE2tbKvZOgOA8RHxUi5rQqOKRsS5wLkA/QdvEcvRZjMza0dvD7S5wEEN5h8GDAJ2jIjXJM0HVsvLXq2stwRYnRSAjQj4z4g4p506NFxH0vGAA8rMrJfo7UOONwD9JX2pNkPSTsBQ4MkcZnvm501FxPPAI5IOyGX0l/QO4DrgKElr5vlDJK1f9/Jm60wFRklaPfcAP9kdDTYzs67p1T20iAhJo4DTJI0lnTubD5wInCFpOjAL+N9lKO7zwDmSfgC8BnwmIiZKeg/wt3wKbjFwOPBkpQ4N14mImZIuz9t/EJjWDU02M7MuUoRHzXpK/8FbxODRp7W6GmYrjfknj2x1FawXkDQjIoZ3tF5vH3I0MzNbJg40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrQr9WV+Dt5H1D1mL6ySNbXQ0zsyK5h2ZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZF6NfqCryd3Pnoc7SNvabV1TAz61HzTx7ZI9txD83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK0KHgSbp+5LmSpotaZakD+T5x0t6R2W9P0paO08vzv+2SZrToMw+ks6QNEfSnZJuk7RJB/UYJ+mgzjawgzI/JWlsd5S/IupnZmbLrl97CyXtCuwHDIuIVyUNBFbNi48HLgZeAoiIfTux3YOBjYDtIuJ1Se8CXuxs5ZeHpH4RMQGY0JPbNTOzFaOjHtpgYGFEvAoQEQsj4jFJXyMF0o2SbgSQND8H3rIYDDweEa/nch+JiGdzOYtrK0k6SNK4yuv2kjRN0j2S9svrbCvp1tx7nC1pizz/iPz8DkkX5XnjJP0s1/kUSUdKOrOD8vtK+mnuRc6W9OU8X5LOlHSXpGuA9Zex7WZmtgK020MDJgL/LukeYBJweURMiYgzJH0D2DMiFnZhu1cAN0n6MHA9cHFE3L4Mr2sD9gA2I4Xp5sAY4PSIuETSqkBfSdsC3wd2i4iFktatlLElsFdELJF05DKUfwTwXETsJKk/8BdJE4EdgK2A9wEbAHcBF9RXWNIxwDEAfd85aBmaaGZmXdFuDy0iFgM7kj6QnwIubxACnRYRj5DC4LvA68D1kj66DC+9IiJej4h7gfuBrYG/Ad+T9B1gaES8DHwE+E0tbCPimUoZV0bEkk6UvzdwhKRZwC3AesAWwO7AZRGxJCIeA25o0tZzI2J4RAzv+461lqGJZmbWFR310Mgf/pOByZLuBEYD45Z3w3kY81rgWklPAAeQemtRWW21+pe9tZi4VNItwEjgOklfBNRg3Zr2ztW9pfxc1lcj4rrqAkn7trMNMzPrYe320CRtVTsnlW0PPJinXwAGdGWjkoZJ2ihP9wG2q5T7hKT35Pmj6l76mXyF5GbApsA8SZsC90fEGaQLPLYjBeNnJa2Xt7Euy+Yt5QPXAf8iaZVc1paS1gCmAofkc2yDgT27si/MzKx7dNRDWxP4n3w5/j+B+8jng4BzSb2rxyOisx/m6wPn5XNSALcCtYszxgJXAw8Dc3IdauYBU0jnrMZExCuSDgYOl/QasAD4QUQ8I+kkYIqkJcDtwJHLUK9G5Z9POrc2U5JIQ68HAONJQ5t3Avfk15mZWYsowqNmPaX/4C1i8OjTWl0NM7MeNf/kkcv1ekkzImJ4R+v5TiFmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRWhX6sr8HbyviFrMf3kka2uhplZkdxDMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyIoIlpdh7cNSS8A81pdjx4yEFjY6kr0ELe1TG+Xtq4M7RwaEYM6Wsm3vupZ8yJieKsr0RMkTXdby+O2lqekdnrI0czMiuBAMzOzIjjQeta5ra5AD3Jby+S2lqeYdvqiEDMzK4J7aGZmVgQHmpmZFcGB1kMkfULSPEn3SRrb6vp0N0nzJd0paZak6XneupL+LOne/O86ra5nV0i6QNKTkuZU5jVsm5Iz8nGeLWlY62reOU3aeaKkR/NxnSVp38qy7+Z2zpP08dbUumskbSzpRkl3S5or6V/z/BKPa7O2lndsI8KPFfwA+gJ/BzYFVgXuALZpdb26uY3zgYF1834CjM3TY4FTWl3PLrZtd2AYMKejtgH7AtcCAnYBbml1/ZeznScC32yw7jb5fdwf2CS/v/u2ug2daOtgYFieHgDck9tU4nFt1tbijq17aD1jZ+C+iLg/Iv4B/BrYv8V16gn7Axfm6QuBA1pYly6LiKnAM3Wzm7Vtf+BXkdwMrC1pcM/UdPk0aWcz+wO/johXI+IB4D7S+3ylEBGPR8TMPP0CcDcwhDKPa7O2NrPSHlsHWs8YAjxcef4I7b+hVkYBTJQ0Q9Ixed4GEfE4pD8qYP2W1a77NWtbicf6uDzMdkFl2LiYdkpqA3YAbqHw41rXVijs2DrQeoYazCvt/0vsFhHDgH2AYyXt3uoKtUhpx/rnwGbA9sDjwKl5fhHtlLQm8Fvg+Ih4vr1VG8xbqdrboK3FHVsHWs94BNi48vxdwGMtqssKERGP5X+fBMaThiieqA3L5H+fbF0Nu12zthV1rCPiiYhYEhGvA+exdOhppW+npFVIH/CXRMTv8uwij2ujtpZ4bB1oPeM2YAtJm0haFTgEmNDiOnUbSWtIGlCbBvYG5pDaODqvNhr4fWtquEI0a9sE4Ih8VdwuwHO1IayVUd15olGk4wqpnYdI6i9pE2AL4Naerl9XSRLwC+DuiPhZZVFxx7VZW4s8tq2+KuXt8iBdJXUP6Yqh77e6Pt3ctk1JV0XdAcyttQ9YD7geuDf/u26r69rF9l1GGpJ5jfTt9ehmbSMN15yVj/OdwPBW138523lRbsds0gfd4Mr638/tnAfs0+r6d7KtHyINo80GZuXHvoUe12ZtLe7Y+tZXZmZWBA85mplZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhWHEmjJIWkrVtdl45IGiHp6gbzz5e0TTuvO1LSRt2w/fdV7rb+jKQH8vQkSRtJ+s3ybqOyreMlHZGnJ0sa3l1lryiSFnewfJJW0l+RKJEDzUp0KHAT6T+wLzdJfbujnM6IiC9GxF3trHIksNyBFhF3RsT2EbE96f8ifSs/3ysiHouIg5Z3GwCS+gFHAZd2R3m9yEXAV1pdCUscaFaUfL+63Uj/KfiQyvzL637vaZykAyX1lfRTSbflm7R+OS8fkX9D6lLSfz5F0lX55stzKzdgRtLRku7JvY7zJJ2Z5w+S9Ntc9m2SdutEOyZLGp7rN07SHKXfm/u6pIOA4cAluTe1uqQdJU3J9buucvumyZJOkXRrruOHO1GHNuXfRss9wqsk/SH34o6T9A1Jt0u6WdK6eb3NJP0p12NapZf8EWBmRPyzbht9JF0o6UcNtn+ypLvycfmv9vappDUl/TLvo9mSDszzD83z5kg6pVL2YkknSboj13+DPH8TSX/LZf+wsv5gSVPz/p5T2Y8TSF+grDdo9f/s9sOP7nwAhwO/yNN/ZenvQI0CLszTq5LuJr46cAzwf/L8/sB00m9AjQBeBDaplF27a8TqpNsErUfqJc0H1gVWAaYBZ+b1LgU+lKffTbr1UH19RwBXN5g/mRRaOwJ/rsxfu7o8T6+S2zooPz8YuKCy3ql5el9gUjv7bhxwUOV5G/m30Ug9wvtIv6c1CHgOGJOX/TfphreQ7q6xRZ7+AHBDnv4P4Kt17duFdHeSt9w5J+/PefDGzR9q7W64T4FTgNMqr18nH5uHcn37ATcAB+TlAXwyT/+k8h6YAByRp48FFufpE1h6B5y+wIDKtu4F1mv1e9+PoB9mZTkUOC1P/zo/n0n6ccYzJPUHPgFMjYiXJe0NbJd7PQBrke5d9w/g1ki/B1XzNUmj8vTGeb0NgSkR8QyApCuBLfM6ewHbSG/cvPydkgZE+k2qZXU/sKmk/wGuASY2WGcr4L3An/O2+pJuYVVTu/HuDFJIddWNue4vSHoO+EOefydpH64JfBC4stLm/vnfwaTf4ao6B7giIk5qsK3ngVeA8yVdA9TOMzbcp3n+Gz3yiHhW6RcfJkfEUwCSLiH9iOlVpONbK3MG8LE8vRtwYJ6+iBSUkO7HeoHSTX6viohZlbo+SQrPpxu0w3qQA82KIWk90tDWeyUF6YM9JH07Il6RNBn4OKkHc1ntZaSew3V1ZY0g9dCqz/cCdo2Il3JZq9H4pzZq+uT1X+5qm/IH8/tzvY8FPks6F/Wm6gJzI2LXJsW8mv9dwvL9zb9amX698vz1XG4fYFGk83H1Xibtr6q/AntKOjUiXqkuiIh/StoZ+CgpqI4jHduG+1Qp4erv49fesXktcveKt+6Xt9wPMCKm5oAcCVwk6acR8au8eLXcPmsxn0OzkhxE+lXhoRHRFhEbAw+Qbs4Kqcf2BeDDQC3ArgP+JX/zRtKWSr8YUG8t4NkcZluThssg3YV8D0nrKF34cGDlNRNJH8Tksht90LdL0kCgT0T8Fvg3YFhe9AJp+A/S0NwgSbvm16wiadvObmt5RfqNrQckfSbXQzmMIfXONq97yS+AP5J6dG8K2tzbWysi/ggcT/rNLmi+T+vnr0P6Ecs9JA1UurDnUGBKB834C0t7eodVyhsKPBkR5+V6D6u1kdRLn99BudYDHGhWkkNJv8VW9Vvgc3l6ImnIaVJE/CPPOx+4C5iZL4A4h8a9mD8B/STNBn4I3AwQEY8CPyZ9eE7KZT2XX/M1YHi+SOEuYEyTen9U0iOVR7WnNQSYLGkW6RzXd/P8ccDZeX5fUpifIukO0t3UP9hkWyvaYcDRuR5zgf3z/GtJ+/5NIv2cyUxSr6f6eTQAuDrv7ynA1/P8Zvv0R8A6+YKNO4A9I/28y3eBG0m/BDEzIjr6CaN/Jf1A7W2kLzE1I4BZkm4nfWk5Pc/fEbg56i52sdbw3fbNlpOkNSNice5ljCddkFEfrG97ksYD346Ie1tdl+4i6XRgQkRc3+q6mHtoZt3hxNxTmkMa4ryqxfXprcaSLg4pyRyHWe/hHpqZmRXBPTQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK8L/B6CIobYbFfdPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7ff4704869b0>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "churn_d = ['Still Subscribed', 'Canceled']\n",
    "importances = [271.166,174.014]\n",
    "y_pos = np.arange(len(churn_d))\n",
    "\n",
    "plt.barh(y_pos,importances,align ='center')\n",
    "plt.yticks(y_pos,churn_d)\n",
    "plt.xlabel('Average Listen Time(k seconds)')\n",
    "plt.title('Average Listen Time Between Two Groups')\n",
    "plt.savefig('Average Listen Time Between Two Groups', dpi=300)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------------+\n",
      "|churn_d|avg(num_thumb_down)|\n",
      "+-------+-------------------+\n",
      "|      0| 13.225806451612904|\n",
      "|      1| 10.333333333333334|\n",
      "+-------+-------------------+\n",
      "\n",
      "+-------+-----------------+\n",
      "|churn_d|avg(num_thumb_up)|\n",
      "+-------+-----------------+\n",
      "|      0|62.89411764705882|\n",
      "|      1|            37.18|\n",
      "+-------+-----------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Number of thumb-up and thumb_down\n",
    "data_log_new \\\n",
    "        .select('userID','page','churn_d') \\\n",
    "        .where(data_log_new.page == 'Thumbs Down') \\\n",
    "        .groupBy('userID','churn_d') \\\n",
    "        .agg({'page':'count'}) \\\n",
    "        .withColumnRenamed('count(page)', 'num_thumb_down') \\\n",
    "        .select('churn_d','num_thumb_down') \\\n",
    "        .groupBy('churn_d') \\\n",
    "        .agg({'num_thumb_down':'avg'}).show()\n",
    "\n",
    "data_log_new \\\n",
    "        .select('userID','page','churn_d') \\\n",
    "        .where(data_log_new.page == 'Thumbs Up') \\\n",
    "        .groupBy('userID','churn_d') \\\n",
    "        .agg({'page':'count'}) \\\n",
    "        .withColumnRenamed('count(page)', 'num_thumb_up') \\\n",
    "        .select('churn_d','num_thumb_up') \\\n",
    "        .groupBy('churn_d') \\\n",
    "        .agg({'num_thumb_up':'avg'}).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbQAAAEWCAYAAAAO4GKjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAHspJREFUeJzt3XmYXFWd//H3h0TCahDCEoKmBcK+hk0GWUVRQAVEWSVxAXQEZEZFBJfIAD8cHh1gcEaBUUbZF8NkWCNgEnAIELaEXYXEhC1EIBBAhOT7++OcMpeiuru60+mqPnxez9NP193PuXW7Pvece/uWIgIzM7OBbplWF8DMzKwvONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONCuapHGSLmrh9k+VNE/Ss71YdqakPZdGuVq5LbOlxYHW5iRNkvSipCGtLktfyPX5q6T3V8btKWlmC4u1VOQ6fgPYJCLWqpt2mKQF+ed1SYsqwwtaU+LWqBwTCyTNlzRF0uZNLtshKSQNXtrl7I1Wvs+SPi9pmqRXJT0n6Q5JRy3NbbaaA62NSeoAdgYC+NRS2kYrPgheBb7Xgu0ukV7sq5HAXyJibv2EiLg4IlaKiJWATwBP14bzuHebY3K9VwMmAb9ubXH6RqveZ0knAz8CTgXWBIYDxwKdtsIlDVpa5ekvDrT2dgQwFbgQGFMbKelDkp6tHoCS9pc0Pb9eRtKJkv4k6S+SrpC0ap5WO6P9kqQ/A7fm8VfmddbOkDetrHs1Sf8r6WVJd+dutNsr0zeS9FtJL0h6TNLnuqnXOcAhktZvNDGXb/3K8IWSTs2vd5M0R9IJkuZKekbSfpL2lvR4LsNJdatcTtLlkl6RdK+kLSvrXlvS1ZKel/SkpOMq08ZJukrSRZJeBsY2KOtQSb/Ky8+S9N28//cEfgusnc/GL+xmn3RmK0nT8/tyuaTl8nbHVt+D+v2W99l/SLohb//3ktaSdFZu8T8qaeu6bW0n6eE8/Ze1bTXS5Ht0klJ360xJhzVT2Yh4C7gM2KSy7k6PZ2BK/v1SrueO+X3YJi97eC7rJnn4y5KuaWK9tb+z/5P0kqQHJO1WmTZJ0r/k/fqKpImShjVTx7r9+FVJV1aGZ0v6VWV4rqSN8utd8/E7X9JUSdt1ss5hpBPGIyPimohYEBGLImJaRHyuMt9lks7JZX8V2FHSqpIuqfw9nCBJef4zJF1QWX4jSW9VhqfmfXJPLuPVkobmaSvm7b2Q9+edkt7X0/3VHQdaezsCuDj/7CVpTYCImEpq5exRmfdQ4JL8+jhgP2BXYG3gReCndeveFdgY2CsP3wCMAtYA7s3brPlp3t5apGCthuuKpA/uS/KyhwD/oUogNvAUcD4wrot5urIWsBwwAvh+XtfhwDakFu33Ja1bmf/TwJXAqrmc10h6j6RlgP8FHsjr+ghwvKS96pa9CliFt++Tmn8HhgLrkvbpEcAXIuJm3n5GPraXdf0c8HHgg8AWNAjVbpb9LjAMeAO4g/TeDiPV6Sd18x9GOh7WAzbIy/bWWnk7I0jHy3mSNuxuIUnL5nJMrYzu6njeJf9eJe/nO4DJwG6V6U/kZWvDk7tbr6QRwHWkFs6qwDeBqyWtXinXocAXSMf9snmenppcK1s+Zt+q1SmHcETEo5LWIB2rZ5BasT8Drq8FRp0Pk3p1rm9i+4eTwm9l4O683veQjrePAl/N9WzWEaT3bwRpn/w4j/8yMDiPHwYcA/ytB+ttTkT4pw1/SAflm8CwPPwo8E+V6acCv8ivVyYFzsg8/Ajwkcq8w/O6BgMdpIN93S62vUqeZygwKC+7Yd22b8+vDwJuq1v+58APOln3JNLBvTowH9iU1A0yszJPAOtXhi8ETs2vdwNeBwZV6h7ADpX57wH2y6/HAVMr05YBniEF3w7An+vK9x3gl5Vlp3SxnwaRgmKTyrijgUmVss5p4r1uOB8wEzi8MvyvwM/y67G196DRfsv77PzKtGOBRyrDmwMv1W3rK5XhvYE/dVHm7t6jt4AVK9OvAL7XxTHxGvAS6UNuPm8/fps5ngdXpn8JmFBZ9svAZXl4FjC6ifV+G/h1XTlvAsZUyvzdyrR/BG7s5fs8l9QiHUvqvZie6/VV4Io8z5H1xyJwH3Bwg/V9mcrfUx53b96/rwPb53GXAedV5hkCLKTy2QB8vVYvUpheUJm2EfBWZXgqMK4yPBp4tbJ/JgObdff3sCQ/bqG1rzHAxIiYl4cvodIyysMHKN0scgBwb0TMytNGAuNz0/4l0h/uQlJfes3s2gtJg3J3wp+UutZm5knDSMEzuDp/3euRwA61beXtHUY6Q+9URDwPnAuc0tV8nfhLRCzMr1/Pv5+rTH8dqF6f+Ht5I2IRMId0Rj6S1CVYLftJdLKfGhhGOgudVRk3i3QW2leqd0e+xtvr1Z36fdLVPoK313UWaR9R6bZc0GzXIfBiRLzaaH2dOC4iViG1vPcFrpK0RZ7WzPFcNRnYWdJapJOOy4GdlK5JDwXub2K9I4HP1h0bHyaFXs2SvDdVU0hhV2s9TiK12nZlcWtybd5+nEHnx9pfgDVrXYUAETE6799XeXvPXPU9XytP+3MT2+hM/TG0Qm5F/hepLlfl7ujTtRSu2bXlnUHvdpKWJ3UXDdLi272HAKtI2jIiHoiIhyXNInVrVbsbIR1UX4yI3zdYd0d+Wf2ahUNJXWt7ksJsKKn7RcDzpLPtdYDH8/zvryw7G5gcER/tRVXPJHUH3VU3/jVghcrwWqQQ6q3qHZXLkOryNKleT0bEqC6W7errKOaRzuhHAg/ncR8gdakuba9S2Uf5w3tJVd/XD5D2ERHxiQbzdvcevU/SipVQ+wDwYHcFyCcct0n6I/AxUmulq+N5ZIN1/FHSa6QuxSkR8Ur+OzqK1KpdlGftar2zSS20I7srcx+odZFuTWoZAuxDCrhT8/DTpC7xqs6OtdtJf7t7k7pNu1I9vp8FFuX1PtFgG6/m4ZpGx1z9MfRaRMzPw99n8eWAm4CHaNyN32tuobWn/UhnipsAW+WfjYHbSH3UNZeQ/mh3IV0jqvkZcFrtj13S6pI+3cX2ViZ1nf2F9CF1em1Cbgn9BhgnaYV8gbpahmuBDZRuEX5P/tlO0sbdVTIiXiL1sZ9QN+l+4NDccvw4i69/9NY2kg5QukvxeFJdp5KC9GVJ35a0fN7eZp1dbG9Q/oWkrrTTJK2c9/c/A/3xf28PAJtK2krp5o1xfbDOr0laR+nGiJNILZvONPMe/VDSspJ2JrW6rmwwzztI2pF07D+UR3V1PD9P+hBet241k0nXaWotnEl1w92t9yLgk5L2ynVcTulml3WaqUMPTSZdJ/1b7rmYDHyG1Pqv7YMJwNaSDpQ0WNIRpMC4sX5leR2nA+cr3TC1ktINMKNJLeCGIuINYDxwer6JYz1Sl2PteL4f2F3SiHxDx7cbrGaspA0krUQ6Ji+Hv/9rzib5hPJl0snkwgbLLxEHWnsaQ7qO8+eIeLb2Q+qiO0yLbx+/lHRmd2ulaxLgbNIfwERJr5A+vHfoYnu/InUPPEVqaUytm34MqdX2LOl26ktJoUBEvEI6kz6YdBb5LOl24Wb/b+5s3nlgfx34JKnP/zDgmibX1Zn/IV3rexH4PHBARLyZA+mTpBOGJ0ktrgtIdW3WsaQz1ydIZ8aXAL9YwvJ2KyIeJ3XX3gz8IW97SV0CTCTV5QkWtw4a6e49epa0v58mnYV/JSIe7WJ952rx/2b9mnR96oY8rdPjOSJeA04Dfp+7Bj+Ul5lMOlGb0slwd+udTeq1OIkUmrOBb7F0PjNnkFr6U/K25+Xt3Rb5AlREPEf6152TSSeexwD75pPCd4iIU3LZv0e6Rvcs6YaX44BpXZTl6Px7FukO6AtY3Iq6jnQCW/uMaPR3Wft8eIp0ovGNPH4E6e/wFVJL/XrSyWCfUt5fZk2T9CNgrYgY0+3M9q6jdHv7RRGxNFoz1qYkTQXOjYiWPZnHLTTrltL/m2yhZHvSXWTjW10uM7Mq3xRizViZ1I2wNqn74sek7gMzs7bhLkczMyuCuxzNzKwI7nLsR8OGDYuOjo5WF8PMbEC555575kXE6t3N50DrRx0dHUyb1tUds2ZmVi8/RKJb7nI0M7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuB/rO5HM56aT8eJ3X2BrFk5Zp6xT6uLYO8ibqGZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVo+0CTtJakyyT9SdLDkq6XtEE/bHdBD+cfJ+mbS6s8ZmbWtbYONEkCxgOTImK9iNgEOAlYs7UlMzOzdtPWgQbsDrwZET+rjYiI+4H7JN0i6V5JMyR9GkBSh6RHJJ0v6SFJEyUtn6etL+lmSQ/k5dbL478l6W5J0yX9sFEhOptH0smSHpN0M7DhUtwPZmbWjXYPtM2AexqM/yuwf0SMJoXej3NrDmAU8NOI2BR4CfhMHn9xHr8l8A/AM5I+luffHtgK2EbSLtUNdTaPpG2Ag4GtgQOA7RpVQNJRkqZJmrbwtfm92glmZta9wa0uQC8JOD2HzyJgBIu7IZ/MrThIYdghaWVgRESMB4iIv8Lfw+pjwH15/pVI4TWlsq3O5lkZGB8Rr+V1TWhU0Ig4DzgPYMjwUbEEdTYzsy60e6A9BBzYYPxhwOrANhHxpqSZwHJ52huV+RYCy5MCsBEB/y8ift5FGRrOI+l4wAFlZtYm2r3L8VZgiKQjayMkbQeMBObmMNs9D3cqIl4G5kjaL69jiKQVgJuAL0paKY8fIWmNusU7m2cKsL+k5XML8JN9UWEzM+udtm6hRURI2h84S9KJpGtnM4FxwDmSpgH3A482sbrPAz+XdArwJvDZiJgoaWPgjnwJbgFwODC3UoaG80TEvZIuz9ufBdzWB1U2M7NeUoR7zfrLkOGjYviYs1pdDLN+M/OMfVpdBCuApHsiYtvu5mv3LkczM7OmONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIg1tdgHeTzUcMZdoZ+7S6GGZmRXILzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIowuNUFeDeZ8dR8Ok68rtXFMDPrVzPP2KdftuMWmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWhG4DTdLJkh6SNF3S/ZJ2yOOPl7RCZb7rJa2SXy/IvzskPdhgnctIOkfSg5JmSLpb0ge7KceFkg7saQW7WeenJJ3YF+tfGuUzM7PmDe5qoqQdgX2B0RHxhqRhwLJ58vHARcBrABGxdw+2exCwNrBFRCyStA7wak8LvyQkDY6ICcCE/tyumZktHd210IYD8yLiDYCImBcRT0s6jhRIv5P0OwBJM3PgNWM48ExELMrrnRMRL+b1LKjNJOlASRdWlttT0m2SHpe0b55nU0l35dbjdEmj8vgj8vADkn6dx10o6Se5zD+SNFbSud2sf5CkM3Mrcrqko/N4STpX0sOSrgPWaLLuZma2FHTZQgMmAt+X9DhwM3B5REyOiHMk/TOwe0TM68V2rwBul7QzcAtwUUTc18RyHcCuwHqkMF0f+ApwdkRcLGlZYJCkTYGTgZ0iYp6kVSvr2ADYMyIWShrbxPqPAOZHxHaShgC/lzQR2BrYENgcWBN4GPhFfYElHQUcBTDovas3UUUzM+uNLltoEbEA2Ib0gfw8cHmDEOixiJhDCoPvAIuAWyR9pIlFr4iIRRHxB+AJYCPgDuAkSd8GRkbE68AewFW1sI2IFyrruDIiFvZg/R8DjpB0P3AnsBowCtgFuDQiFkbE08CtndT1vIjYNiK2HbTC0CaqaGZmvdFdC4384T8JmCRpBjAGuHBJN5y7MW8AbpD0HLAfqbUWldmWq1/snauJSyTdCewD3CTpy4AazFvT1bW6d6w/r+vYiLipOkHS3l1sw8zM+lmXLTRJG9auSWVbAbPy61eAlXuzUUmjJa2dXy8DbFFZ73OSNs7j969b9LP5Dsn1gHWBxyStCzwREeeQbvDYghSMn5O0Wt7GqjTnHesHbgK+Kuk9eV0bSFoRmAIcnK+xDQd2782+MDOzvtFdC20l4N/z7fhvAX8kXw8CziO1rp6JiJ5+mK8BnJ+vSQHcBdRuzjgRuBaYDTyYy1DzGDCZdM3qKxHxV0kHAYdLehN4FjglIl6QdBowWdJC4D5gbBPlarT+C0jX1u6VJFLX637AeFLX5gzg8bycmZm1iCLca9ZfhgwfFcPHnNXqYpiZ9auZZ+yzRMtLuicitu1uPj8pxMzMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIgxudQHeTTYfMZRpZ+zT6mKYmRXJLTQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrgiKi1WV415D0CvBYq8vRR4YB81pdiD7iurSfUuoBrktfGBkRq3c3kx991b8ei4htW12IviBpmuvSfkqpSyn1ANelP7nL0czMiuBAMzOzIjjQ+td5rS5AH3Jd2lMpdSmlHuC69BvfFGJmZkVwC83MzIrgQDMzsyI40PqJpI9LekzSHyWd2Ory9ISkX0iaK+nByrhVJf1W0h/y7/e1sozNkPR+Sb+T9IikhyR9PY8fiHVZTtJdkh7IdflhHv9BSXfmulwuadlWl7UZkgZJuk/StXl4oNZjpqQZku6XNC2PG3DHF4CkVSRdJenR/DezY7vXxYHWDyQNAn4KfALYBDhE0iatLVWPXAh8vG7cicAtETEKuCUPt7u3gG9ExMbAh4Cv5fdhINblDWCPiNgS2Ar4uKQPAT8C/i3X5UXgSy0sY098HXikMjxQ6wGwe0RsVfl/rYF4fAGcDdwYERsBW5Len/auS0T4Zyn/ADsCN1WGvwN8p9Xl6mEdOoAHK8OPAcPz6+GkfxpveTl7WKf/AT460OsCrADcC+xAeorD4Dz+bcddu/4A65A+HPcArgU0EOuRyzoTGFY3bsAdX8B7gSfJNw4OlLq4hdY/RgCzK8Nz8riBbM2IeAYg/16jxeXpEUkdwNbAnQzQuuRuuvuBucBvgT8BL0XEW3mWgXKcnQWcACzKw6sxMOsBEMBESfdIOiqPG4jH17rA88Avc1fwBZJWpM3r4kDrH2owzv8v0SKSVgKuBo6PiJdbXZ7eioiFEbEVqYWzPbBxo9n6t1Q9I2lfYG5E3FMd3WDWtq5HxU4RMZp0eeFrknZpdYF6aTAwGvjPiNgaeJV2615swIHWP+YA768MrwM83aKy9JXnJA0HyL/ntrg8TZH0HlKYXRwRv8mjB2RdaiLiJWAS6brgKpJqz2gdCMfZTsCnJM0ELiN1O57FwKsHABHxdP49FxhPOtEYiMfXHGBORNyZh68iBVxb18WB1j/uBkblO7eWBQ4GJrS4TEtqAjAmvx5Duh7V1iQJ+C/gkYj4SWXSQKzL6pJWya+XB/YkXbT/HXBgnq3t6xIR34mIdSKig/R3cWtEHMYAqweApBUlrVx7DXwMeJABeHxFxLPAbEkb5lEfAR6mzeviJ4X0E0l7k848BwG/iIjTWlykpkm6FNiN9NURzwE/AK4BrgA+APwZ+GxEvNCqMjZD0oeB24AZLL5ecxLpOtpAq8sWwH+TjqdlgCsi4hRJ65JaOqsC9wGHR8QbrStp8yTtBnwzIvYdiPXIZR6fBwcDl0TEaZJWY4AdXwCStgIuAJYFngC+QD7WaNO6ONDMzKwI7nI0M7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40Kx4kvaXFJI2anVZuiNpt1zWT1bGXZtvae+L9c+UNKwv1tXNds7M3wJwZmXcF/JT6O+X9LfKU+nPkDRO0jeXQjk6qt8SYWVzoNm7wSHA7aR/3F1i+dsTlqY5wMlLeRs9VnlyRzOOBkZHxLdqIyLil5GeQr8V6ckftafSt/0jlWxgcKBZ0fJzG3ciff3IwZXxl+d/dq8NXyjpM/mBv2dKulvSdElH5+m7KX2X2iWkf8xG0jX5IbQPVR5Ei6QvSXpc0iRJ50s6N49fXdLVed13S9qpk2I/AMyX9NEG9fl7C0vStpIm5dfjJP23pIl5ngMk/WtuBd2YH/lV8y2l71K7S9L6XZUtr/c8SROBX9WVRXlfPZi3c1AePwFYEbizNq5Jm+R99oSk4/K63tbCkvRNSePy60mS/k3SFKXv69pO0m+Uvqvr1Mp6B+d9M13p+71WaLBfd1P+LrY8fK6ksZV9/qP6fWbtx4FmpduP9J1OjwMvSBqdx18G1D6AlyU92ud6UvDNj4jtgO2AIyV9MC+zPXByRNS+y+6LEbENsC1wnKTVJK0NfI/0XMWPAtVuzrNJ3/G1HfAZ0lMYOnMq8N0e1nU9YB/g08BFwO8iYnPg9Ty+5uWI2B44l/T0mu7Ktg3w6Yg4tG57B5C+i21L0qO3zpQ0PCI+BbyeW1+X96D8GwF7kfbzD+pCuDN/i4hdgJ+RHsP0NWAzYGx+QgfAhsB5EbEF8DLwjz0oU02jfWZtxoFmpTuEFF7k34fk1zcAe0gaQnoy+pSIeJ30/L0jlL6W5U7SV5mMysvcFRFPVtZ9nKQHgKmkh0+PIn0YT46IFyLiTeDKyvx7AufmdU8A3qv87L96EXEbgKSde1DXG/I2Z5AeiXVjHj+D9H12NZdWfu/YRNkm5H1T78PApfmp/88Bk0knAb11XUS8ERHzSA+9XbOJZWrPRJ0BPBQRz+RHZD3B4geCz46I3+fXF+Vy91SjfWZtpid94mYDSj5D3wPYTFKQPuRD0gkR8dfcXbcXqaVW+8AScGxE3FS3rt1IX6FRHd4T2DEiXsvrWo7GX31Ss0yev1E4NHIa6VraW5Vxb7H4RHS5uvnfAIiIRZLejMXPtVvE2//Wo8HrhmWTBJV61+mqrr1RfVbjQlKZq/WFTupMqmN1+Wqd65/vF5J2AH6eh78PvNDNdhrtM2szbqFZyQ4EfhURIyOiIyLeT/oW3toZ+mWkB67uDNQC7Cbgq7XuLkkbKD05vd5Q4MUcZhuRuhgB7gJ2lfS+fBPFZyrLTASOqQ0oPfy1UxExEXgfqUuvZiapC5C6dffEQZXfd/SmbNkU4KB83XF1YBdS/fvSc8AauTt3CLBvL9bxAUm1VtUhwO0RcWftBpWImADMIl3DGyJpKKkLuqrRPrM240Czkh3C4qef11wN1K4FTSR9CN8cEX/L4y4gfU3GvflmhJ/TuCfjRtLNBtOBfyF1OxIRTwGnk7orb87rmp+XOQ7YNt+c8DDwlSbqcBrp+8BqfgicLek2UiumN4ZIuhP4OvBPS1C28cB00k0stwIn5K8d6TO5C/UU0v68Fni0F6t5BBiT36tVgf9ssJ3ZpKfITwcuJj3hv6rRPrM246ftm/UxSStFxILcQhtP+rqg+mC1AULpy0e3zdf2rI25hWbW98blmyseJHVxXtPi8pi9K7iFZmZmRXALzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK8P8BTJeW5FAzZf8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7ff4703cf780>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "importances = [62.8941,37.18]\n",
    "churn_d = ['Still Subscribed', 'Canceled']\n",
    "y_pos = np.arange(len(churn_d))\n",
    " \n",
    "plt.barh(y_pos, importances, align='center')\n",
    "plt.yticks(y_pos, churn_d)\n",
    "plt.xlabel('Average Number of Thumb-up')\n",
    "plt.title('Average Number of Thumb-up Between Two Groups')\n",
    "plt.savefig('Average Number of Thumb-up Between Two Groups', dpi=300)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbQAAAEWCAYAAAAO4GKjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XncVWW9/vHPBSg4kOYYoPHkkPOMWpmmieVUoFlqWpieyk6l1jEz7RT1s36U1XHqpGaJHscsKU5mkgOD5QQOgKjkgIozmijOwvf8cd9bFttnnjbP7fV+vZ4Xew17re9ae611rXWvzdqKCMzMzPq6fo0uwMzMrDs40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA4069MkjZV0UQPnf4qkBZKe7MR750ka2RN1dfe8JI2XdEp31mTW3RxoDSZpsqR/SRrY6Fq6Q16eVyWtX+k3UtK8BpbVI/Iy/geweUS8p27YYZIW5b9XJC2pdC9qTMV9Vw7U1/P6e1HSDEkf6cD7Q9JGPVljZ0natbJtvJRrXVT5e28PzntfSVPzOn1W0u2Sjpe0Yk/Nsyc50BpIUhOwKxDAJ3toHgN6YrpteAn4zwbMt0s6sa6GA89GxNP1AyLi4ohYNSJWBfYBHq91537WcT/N62414FfAlZL6N7imLouIaZXtYovce/XK9vJIT8xX0uHAJcD5wPoRsSZwOLAh8J4W3tOI40m7OdAa6/PAzcB4YEytp6QPSHqyurNKOkDSzPy6n6QTJT2Qz6p+J2mNPKwpn+EdJekR4Prc/4o8zYX5jGyLyrTXlPS/kl6QdFtuRruxMnxTSX+T9Jyk+yR9po3lOgM4tKUz4vqz5WpzlqTdJc2XdIKkpyU9IWl0PpOcm2s4qW6SgyRdns8yb5e0TWXaQyX9QdIzkh6SdExl2FhJv5d0kaQXgCOaqXU1SRfm9z8s6bt5/Y8E/gYMzWfR49tYJy3ZVtLM/LlcLmlQnu8R1c+gfr3ldfbfkq7O8/+7pPdIOk3piv9eSdvVzWtHSXPy8PNr82qOpO3yunxR0uXAoLrhX5R0f/48Jkoamvv/QNKZ+fUK+Yrjp7l7JaWr93dXttMxkh5RarY9uT0rLCKWkA7EawDrVmo6UtI9efmukTQ895+aR7krr6uDJU2R9Kk8/MO5ln1z90hJd7Y13TysxX0jf0a/lHRVXo+3SNqwPctYt673kXRbpfvGyjIhabqkvfPrrSRNk/R83q72aWGaA4CfASdHxPkR8Xxet3Mi4iu1EJU0TtIltf0LOCR/jr/M++Z8SadKWiGPf7SkayvzGZTX7Xq5+zJJZ0q6Ia+T6yQNy8P6Szor72sLJd0laZMOrayI8F+D/oD7gX8HdgDeANatDHsA2KvSfQVwYn59HCkI1wMGAucAl+ZhTaQrvguBVYCVcv8jgcF5/NOAOyvTviz/rQxsDjwK3JiHrZK7vwAMALYHFgBbtLBMk4F/A34BXJT7jQTmVcYJYKNK93jglPx6d+BN4HvACsAXgWdIB7DBpDPYV4EN8vhj87o7KI9/PPBQft0PmJGntSKwAfAg8PG6947O467UzPJcCPwpz7sJmAscVal1fjs+52bHA+YBtwJDSQfne4Cj87Ajap9Bc+str7MFpG1nEOnE5SHSSVJ/4BTghrp5zQbWz/P6e22dN1PXisDDwDfyejwor6faZ/TRPO/tSdvTmcDUyrBZ+fWHSNvxLZVhd9Vtp78GVgK2AV4DNmuhpuo20h84On+W/XO/0aT9aTPSdvpd4B+tbHM/BM7Mr0/Kdf6kMuz0tqZLG/tGrvk5YKc8/GLgsja2ldp6GVDpNzivm3flz/oJ4Mn8ejBpfxicux8mNYOvAHwcWAS8r5n5bJvn85426hmX570veR8BfgpMA9YinVDcRgpG8udybeX9g/J81qsca54HPpiHnV0bHxgF3JSXsx9pX1+nQ8fU7jo4+69jf8CHSQeJtXL3vcA3KsNPAX5b2aBfAobn7nuAPSvjDsnTGlDZITZoZd6r53FWIx0c3gA2qZt3LdAOBqbVvf8c4PstTHsyKdDWBhbmjbKjgfYKSw9Ug/P4O1fGnwGMzq/HAjdXhvXLO/yuwM7AI3X1fQc4v/Leqa2sp/55Z9680u/LwORKrV0NtMMr3T8Fzs6vj6DtQPt1ZdjXgXsq3VsBz9fN6+hK977AAy3UuxvwOKBKv39UPqPfkJr/asNWzdtQE+mA9yqwJnAiKSzm53F+AJyR39NE5UCX+90KHNJCTePzdJ/P/74KHFYZfjX5RKOyHbzM0n2mfpvbE5iZX/+VtM3enLunAAe2NV3a2DdyzefVrfN729hWautlQF3/2/L7dwcm5r/dSc3Zt+Zx9iIFWvVzm0A+Ea6b3khgCdCv0u+Pef2+DHw69xsHTKp772PARyvdo2rLRfsCbXxl+Bp5+Np5+e4mnQCotfXU0p+bHBtnDGlDWZC7L6HS7Ji7D1T6ssiBwO0R8XAeNhyYkJsVnicF3GIqzS+kM0fgrUv5cUpNlC+QDm6QzrDWJgXho829N89r59q88vwOo4U29pqIeAY4i3S221HPRsTi/PqV/O9TleGvkA6Qb6s3UnPUfNJVz3BSk2C19pNoYT01Yy2WXq3UPAwM68CytKX67ciXWXa52lK/TlpbR7Dssj5MWkdUmi0XSTos938s8hGnMn7N0Gp3RCwCngWGRcQrwHTgI6RgnEIKw11yvyl1NXVk+X8WEauTQnMEcGqlSW04cHrlc34OEC1/VjcB75e0Lulq5UJgfUlrkQ6otSa91qbbnn2jK59v1RRSgNXW6WTS+qyu06GkE7j6z625dfBsXo639oWIGJ3X7xzSyVxN9Vgi0vJ1ZZ+o7q/Pka4ih5JOHn5DOil4SqlJvUPry4HWAJJWAj4DfETpvtaTpOadbZTv/0TEHNKGsg/wWVLA1TwK7BMRq1f+BkXEY5Vxqhv1Z0lnUSNJV2VNtVJIzXlvkpova9avvH4UmFI3r1Uj4ivtWNRTgT1IzWJVL5OaN2taDcd2qH6jsh9pWR4n1f5QXe2DI2LfynuDli0gXXkMr/R7L+kMtae9RGUdSerqOoJlP9f3ktYREbFPLP0CwsWkK9xh+eBVHb/mcSrrRNIqpCuy2nqZQmpe3I50ZTGF1PxVDYpOi2Q2qdl0v9z7UeDLdZ/1ShHxjxam8TLpSv9YYHZEvE4K3m+SrlwXtGO6Xdk3Oqo+0Kbw9kB7nGU/J2h5e50FPE06WW7LW/tIDssnaXmfWGa7pfl9u7q/rkEK+Sfy5/qLiNgO2JrUDH1sO+p7iwOtMUaTrqg2J50dbktqo59GugdScwlwDGkjvqLS/2zgR5Wb3mtLGtXK/Gpt8M+SNrYf1wbkK6ErgbGSVpa0aV0NfyadyX5O6Sb/CpJ2lLRZWwsZ6Ubzz4ET6gbdCXw2XznuTdopu2IHSQfmG93HkZb1ZlIT1guSvp1vZPeXtKWkHdsz0bxufkda14Pz+v4m0Bv/7+0uYAtJ2yp9eWNsN0zzq5LWyweRk4DLWxjvJtJJzjGSBkg6kBRGNZcAX8i1DSRtT7dExLw8fAppG5qTg2IyqUnvoXzl3mV5O/0wqYkK0j7xHeUvOyl9mefTlbc8RbqHWjUF+BpLA2FyXXdb0+30vtEJ00gH+C2BO/LfZqSThhsr4/STdFz+3PYCPsayxw4AIuJN0n75I0lfkLS6kk1JLROtuRT4vtKXydYBTmbpPnEnsJ2kLSStTLp/XW+UpJ3ztlO71/u00pfhRuT9+CXgddJxst0caI0xhnQf55GIeLL2R2qiO0xLvxp7Kems7PrKGSPA6aQ29En5m0c3k+4XteRC0tXeY6TmhJvrhn+NdOX2JPA/eb6vAUTEi6Sd4hDSGeCTwE9IXwZoj9N5+0Z5LPAJUnv9YaS2+674E+l+xr+Az5Huf7yRA+kTpBOGh0hXXOeRlrW9vk7auR4kHTguAX7bxXrbFBFzSc211wL/ZOlBqysuASaRluVB0sGkuXm/TjpzP4K0Tg8mnfTUhl9H+m8ZfyBdzW1I2j5q/kFqFqxdjc0h3fPq6tXZCblZ9KW8HOeTmqeIiAmk7fKy3Kw+m9S6UTMWuCA3Dda+iTiFdLI3tYXuVqfbDftGu+WTwznAHRGxODetzyDdN619Q/FVYH/Sl3ieJX0x6+CIeKCFaV5A2l+OJDXTLyBtI7XjS0u+l2u5mxRgfyfd/yUiZrH0SyP3kk4S6l1Euje3gBTKtVstq5PuOz5P2j4fJn1jut20bHOrGUj6CenbT2PaHNnMrJ0kXUZq4u2Rp874Cs1q/5dm69zksBNwFOnbUWZmfcZy/b++rdcMJjUzDiXdKP45qRnPzKzPcJOjmZkVwU2OZmZWBDc59qK11lormpqaGl2GmVmfMmPGjAURsXZb4znQelFTUxPTp09vdBlmZn2KpIfbHstNjmZmVggHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkR/B+re9GsxxbSdOJVjS7DzHrQvHH7tT2S9QhfoZmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRVjuA03SeyRdJukBSXMk/UXS+3thvos6OP5YScf3VD1mZta65TrQJAmYAEyOiA0jYnPgJGDdxlZmZmbLm+U60IA9gDci4uxaj4i4E7hD0nWSbpc0S9IoAElNku6R9GtJd0uaJGmlPGwjSddKuiu/b8Pc/1uSbpM0U9IPmiuipXEknSzpPknXApv04HowM7M2LO+BtiUwo5n+rwIHRMT2pND7eb6aA9gY+GVEbAE8D3wq9784998G+BDwhKSP5fF3ArYFdpC0W3VGLY0jaQfgEGA74EBgx+YWQNKXJE2XNH3xyws7tRLMzKxtAxpdQCcJ+HEOnyXAMJY2Qz6Ur+IghWGTpMHAsIiYABARr8JbYfUx4I48/qqk8JpamVdL4wwGJkTEy3laE5srNCLOBc4FGDhk4+jCMpuZWSuW90C7Gziomf6HAWsDO0TEG5LmAYPysNcq4y0GViIFYHME/P+IOKeVGpodR9JxgAPKzGw5sbw3OV4PDJT0xVoPSTsCw4Gnc5jtkbtbFBEvAPMljc7TGChpZeAa4EhJq+b+wyStU/f2lsaZChwgaaV8BfiJ7lhgMzPrnOX6Ci0iQtIBwGmSTiTdO5sHjAXOkDQduBO4tx2T+xxwjqQfAm8An46ISZI2A27Kt+AWAYcDT1dqaHaciLhd0uV5/g8D07phkc3MrJMU4Vaz3jJwyMYxZMxpjS7DzHrQvHH7NbqE4kiaEREj2hpveW9yNDMzaxcHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFWFAowt4J9lq2GpMH7dfo8swMyuSr9DMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCAMaXcA7yazHFtJ04lWNLsPMrFfNG7dfr8zHV2hmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRXCgmZlZERxoZmZWBAeamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWRHaDDRJJ0u6W9JMSXdK2jn3P07SypXx/iJp9fx6Uf63SdLsZqbZT9IZkmZLmiXpNknva6OO8ZIO6ugCtjHNT0o6sTum3xP1mZlZ+w1obaCkDwL7A9tHxGuS1gJWzIOPAy4CXgaIiH07MN+DgaHA1hGxRNJ6wEsdLb4rJA2IiInAxN6cr5mZ9Yy2rtCGAAsi4jWAiFgQEY9LOoYUSDdIugFA0rwceO0xBHgiIpbk6c6PiH/l6SyqjSTpIEnjK+8bKWmapLmS9s/jbCHp1nz1OFPSxrn/53P3XZL+J/cbL+kXueafSDpC0lltTL+/pFPzVeRMSV/O/SXpLElzJF0FrNPOZTczsx7Q6hUaMAn4nqS5wLXA5RExJSLOkPRNYI+IWNCJ+f4OuFHSrsB1wEURcUc73tcEfATYkBSmGwFHA6dHxMWSVgT6S9oCOBnYJSIWSFqjMo33AyMjYrGkI9ox/c8DCyNiR0kDgb9LmgRsB2wCbAWsC8wBfltfsKQvAV8C6P+utduxiGZm1hmtXqFFxCJgB9IB+Rng8mZCoMMiYj4pDL4DLAGuk7RnO976u4hYEhH/BB4ENgVuAk6S9G1geES8AnwU+H0tbCPiuco0roiIxR2Y/seAz0u6E7gFWBPYGNgNuDQiFkfE48D1LSzruRExIiJG9F95tXYsopmZdUZbV2jkg/9kYLKkWcAYYHxXZ5ybMa8Grpb0FDCadLUWldEG1b/t7ZOJSyTdAuwHXCPp3wA1M25Na/fq3jb9PK2vR8Q11QGS9m1lHmZm1stavUKTtEntnlS2LfBwfv0iMLgzM5W0vaSh+XU/YOvKdJ+StFnuf0DdWz+dvyG5IbABcJ+kDYAHI+IM0hc8tiYF42ckrZnnsQbt87bpA9cAX5G0Qp7W+yWtAkwFDsn32IYAe3RmXZiZWfdo6wptVeDM/HX8N4H7yfeDgHNJV1dPRERHD+brAL/O96QAbgVqX844Efgz8CgwO9dQcx8whXTP6uiIeFXSwcDhkt4AngR+GBHPSfoRMEXSYuAO4Ih21NXc9M8j3Vu7XZJITa+jgQmkps1ZwNz8PjMzaxBFuNWstwwcsnEMGXNao8swM+tV88bt16X3S5oRESPaGs9PCjEzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgONDMzK4IDzczMiuBAMzOzIjjQzMysCA40MzMrggPNzMyK4EAzM7MiONDMzKwIDjQzMyuCA83MzIrgQDMzsyI40MzMrAgDGl3AO8lWw1Zj+rj9Gl2GmVmRfIVmZmZFcKCZmVkRHGhmZlYEB5qZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZmZmRVBENLqGdwxJLwL3NbqOLloLWNDoIrrA9TdWX68f+v4y9MX6h0fE2m2N5Edf9a77ImJEo4voCknT+/IyuP7G6uv1Q99fhr5ef2vc5GhmZkVwoJmZWREcaL3r3EYX0A36+jK4/sbq6/VD31+Gvl5/i/ylEDMzK4Kv0MzMrAgONDMzK4IDrZdI2lvSfZLul3Rio+vpCEnrS7pB0j2S7pZ0bKNr6gxJ/SXdIenPja6lMyStLun3ku7Nn8UHG11TR0j6Rt5+Zku6VNKgRtfUGkm/lfS0pNmVfmtI+pukf+Z/393IGtvSwjKcmrehmZImSFq9kTV2JwdaL5DUH/glsA+wOXCopM0bW1WHvAn8R0RsBnwA+Gofq7/mWOCeRhfRBacDf42ITYFt6EPLImkYcAwwIiK2BPoDhzS2qjaNB/au63cicF1EbAxcl7uXZ+N5+zL8DdgyIrYG5gLf6e2ieooDrXfsBNwfEQ9GxOvAZcCoBtfUbhHxRETcnl+/SDqQDmtsVR0jaT1gP+C8RtfSGZLeBewG/AYgIl6PiOcbW1WHDQBWkjQAWBl4vMH1tCoipgLP1fUeBVyQX18AjO7VojqouWWIiEkR8WbuvBlYr9cL6yEOtN4xDHi00j2fPhYINZKagO2AWxpbSYedBpwALGl0IZ20AfAMcH5uNj1P0iqNLqq9IuIx4GfAI8ATwMKImNTYqjpl3Yh4AtKJHrBOg+vpqiOBqxtdRHdxoPUONdOvz/1/CUmrAn8AjouIFxpdT3tJ2h94OiJmNLqWLhgAbA/8KiK2A15i+W/ueku+1zQKeB8wFFhF0uGNreqdTdLJpNsJFze6lu7iQOsd84H1K93rsZw3t9STtAIpzC6OiCsbXU8H7QJ8UtI8UnPvRyVd1NiSOmw+MD8ialfGvycFXF8xEngoIp6JiDeAK4EPNbimznhK0hCA/O/TDa6nUySNAfYHDouC/jOyA6133AZsLOl9klYk3Qyf2OCa2k2SSPdu7omIXzS6no6KiO9ExHoR0URa99dHRJ+6OoiIJ4FHJW2Se+0JzGlgSR31CPABSSvn7WlP+tCXWiomAmPy6zHAnxpYS6dI2hv4NvDJiHi50fV0JwdaL8g3YL8GXEPaiX8XEXc3tqoO2QX4HOnK5s78t2+ji3oH+jpwsaSZwLbAjxtcT7vlK8vfA7cDs0jHnuX6EUySLgVuAjaRNF/SUcA4YC9J/wT2yt3LrRaW4SxgMPC3vC+f3dAiu5EffWVmZkXwFZqZmRXBgWZmZkVwoJmZWREcaGZmVgQHmpmZFcGBZkWQdICkkLRpo2tpi6Tdc62fqPT7s6Tdu2n68ySt1R3TamM+p+an559a6feFyn/teF3SrPx6nKSxko7vgTqaqk+T78D7xks6qLvrscZxoFkpDgVupJue4J5/IaEnzQdO7uF5dFh+cHB7fRnYPiK+VesREedHxLYRsS3paTh75O4+85gu67scaNbn5WdM7gIcRSXQJF1e/Q/g+Yz8U/l30U6VdFv+Tagv5+G75999u4T0n3+R9EdJM/KVyJcq0zpK0lxJkyX9WtJZuf/akv6Qp32bpF1aKPsuYKGkvZpZnreusCSNkDQ5vx4r6QJJk/I4B0r6ab4K+mt+PFnNtyTdmv82aq22PN1zJU0CLqyrRXldzc7zOTj3nwisAtxS69dOm+d19qCkY/K0lrnCknS8pLH59WRJ/yVpqtJvwO0o6Uql3yM7pTLdAXndzFT6zbiVm1mvknSWpDmSrqLyYGFJeyo99HmW0m+IDZS0k6Qr8/BRkl6RtKKkQZIerNT3k7ye50ratQPrwrqZA81KMJr0O2Fzgeck1Z5xeBlQOwCvSHrc0l9IwbcwInYEdgS+KOl9+T07ASdHRO333o6MiB2AEcAxktaUNBT4T9Jvw+0FVJs5Twf+K0/7U7T+czWnAN/t4LJuSPoZnFHARcANEbEV8EruX/NCROxEeirEae2obQdgVER8tm5+B5KeSrIN6XmMp0oaEhGfBF7JV1+Xd6D+TYGPk9bz9+tCuCWvR8RuwNmkR019FdgSOELSmnmcTYBz8298vQD8ezPTOSCPtxXwRfKzJJV+aHQ8cHBelwOAr5CearJdfu+uwGzS9rIzy/7axIC8ro8Dvt+O5bEe4kCzEhxKCi/yv4fm11eTHtc1kPTjqlMj4hXgY8DnJd1JOjCtCWyc33NrRDxUmfYxku4i/W7U+nm8nYApEfFcftDuFZXxRwJn5WlPBN4laXBzRUfENIAOntVfnec5i/QjmX/N/WcBTZXxLq38W/tl69Zqm5jXTb0PA5dGxOKIeAqYQjqod9ZVEfFaRCwgPdh33Xa8p/bc01nA3fn3+V4DHmTpQ78fjYi/59cX5brr7cbSZXkcuD7334T04OS5ufsCYLf8yLr7JW1G+sx/kaexKzCtMt3aw7pnsOxnYL2sI+3lZsudfIb+UWBLSUE6yIekEyLi1dxc93HSlVrtIC/g6xFxTd20dif9LEu1eyTwwYh4OU9rEM3/HFBNvzx+c+HQnB+R7qW9Wen3JktPNgfVjf8aQEQskfRG5UnpS1h2f45mXjdbmySoLHed1pa1M16rvF5Mqrm6vNDCMpOWsfr+6jLXP8MvJO0MnJO7v9fCeND6Mk4jnQy9AVxLupLrD1S/3FKrqbY81iC+QrO+7iDgwogYHhFNEbE+8BBLz9AvA75AOquuBdg1wFdqzV2S3q/mfyxzNeBfOcw2JTUxAtwKfETSu5W+RPGpynsmkR5ETZ72tq0Vn3/k8t2kJr2aeaQmQOqm3REHV/69qTO1ZVOBg5XuO65NukK5tZM1teQpYJ3cnDuQ9LMmHfVeSbUr0UOBGyPiltoXVCJiImlZDsnLMgTYI49/L9BUu9dIehD3lPx6Kqkp8aaIeIZ0Nb8p0JceLv6O4UCzvu5QYEJdvz8AtXtBk0gH4Wsj4vXc7zzST6/cnr+McA7Nn1n/lfRlg5nA/yM1O9Z+ffnHpObKa/O0Fub3HAOMyF9OmAMc3Y5l+BHpN/JqfgCcLmka6ay/MwZKugU4FvhGF2qbAMwkfYnleuCE/FM23SY3of6QtD7/TAqYjroHGJM/qzWAXzUzzgTgn6Smy1+RQysiXiWd9FwhaRbpyq/2BPpbSM2iU3P3TGBmSb8hVhI/bd+sEyStGhGL8hXaBOC3EVEfrGbWi3yFZtY5Y/NSHUvyAAAAN0lEQVSXK2aTmjj/2OB6zN7xfIVmZmZF8BWamZkVwYFmZmZFcKCZmVkRHGhmZlYEB5qZmRXh/wAld6rGXLyV0gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7ff46c2de668>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "importances = [13.2258,10.3333]\n",
    "churn_d = ['Still Subscribed', 'Canceled']\n",
    "y_pos = np.arange(len(churn_d))\n",
    " \n",
    "plt.barh(y_pos, importances, align='center')\n",
    "plt.yticks(y_pos, churn_d)\n",
    "plt.xlabel('Average Number of Thumb-down')\n",
    "plt.title('Average Number of Thumb-down Between Two Groups')\n",
    "plt.savefig('Average Number of Thumb-down Between Two Groups.png', dpi=300)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Engineering\n",
    "Once you've familiarized yourself with the data, build out the features you find promising to train your model on. To work with the full dataset, you can follow the following steps.\n",
    "- Write a script to extract the necessary features from the smaller subset of data\n",
    "- Ensure that your script is scalable, using the best practices discussed in Lesson 3\n",
    "- Try your script on the full data set, debugging your script if necessary\n",
    "\n",
    "If you are working in the classroom workspace, you can just extract features based on the small subset of data contained here. Be sure to transfer over this work to the larger dataset when you work on your Spark cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Total time spent \n",
    "d1 = data_log_new.select('userID','length').groupBy('userID').agg({'length':'sum'}).withColumnRenamed('sum(length)', 'listen_time')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of songs listened to \n",
    "d2 = data_log_new.select('userID','song').groupBy('userID').agg({'song':'count'}).withColumnRenamed('count(song)', 'num_song')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of thumbs-up, thumbs-down\n",
    "d3 = data_log_new .select('userID','page').where(data_log_new.page == 'Thumbs Up').groupBy('userID').agg({'page':'count'}).withColumnRenamed('count(page)', 'num_thumb_up')\n",
    "d4 = data_log_new .select('userID','page').where(data_log_new.page == 'Thumbs Down').groupBy('userID').agg({'page':'count'}).withColumnRenamed('count(page)', 'num_thumb_down')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of adds to playlist\n",
    "d5 = data_log_new.select('userID','page').where(data_log_new.page == 'Add to Playlist').groupBy('userID').agg({'page':'count'}).withColumnRenamed('count(page)', 'add_to_playlist')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of lifetime\n",
    "d6 = data_log_new.select('userID','registration','ts').withColumn('lifetime',(data_log_new.ts-data_log_new.registration)).groupBy('userID').agg({'lifetime':'max'}).withColumnRenamed('max(lifetime)','lt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of add to friend\n",
    "d7 = data_log_new.select('userID','page').where(data_log_new.page == 'Add Friend').groupBy('userID').agg({'page':'count'}).withColumnRenamed('count(page)', 'add_friend') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of help/rolladvert\n",
    "d8 = data_log_new.select('userID','page').where(data_log_new.page == 'Help').groupBy('userID').agg({'page':'count'}).withColumnRenamed('count(page)', 'help') \n",
    "d9 = data_log_new.select('userID','page').where(data_log_new.page == 'Roll Advert').groupBy('userID').agg({'page':'count'}).withColumnRenamed('count(page)', 'rolladvert')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-------+\n",
      "|userID|churn_d|\n",
      "+------+-------+\n",
      "|100010|      0|\n",
      "|200002|      0|\n",
      "|   125|      1|\n",
      "|    51|      1|\n",
      "|   124|      0|\n",
      "|     7|      0|\n",
      "|    54|      1|\n",
      "|    15|      0|\n",
      "|   155|      0|\n",
      "|   132|      0|\n",
      "|   154|      0|\n",
      "|100014|      1|\n",
      "|   101|      1|\n",
      "|    11|      0|\n",
      "|   138|      0|\n",
      "|300017|      0|\n",
      "|    29|      1|\n",
      "|    69|      0|\n",
      "|100021|      1|\n",
      "|    42|      0|\n",
      "+------+-------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#create a table indicating for each user, whether he/she churned or not?\n",
    "user_churn = spark.sql(\"SELECT userID, SUM(churn) AS churn_d FROM df_churn_log GROUP BY userID\")\n",
    "user_churn.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# join table together\n",
    "dt = d1.join(d2,'userID','outer') \\\n",
    "    .join(d3,'userID','outer') \\\n",
    "    .join(d4,'userID','outer') \\\n",
    "    .join(d5,'userID','outer') \\\n",
    "    .join(d6,'userID','outer') \\\n",
    "    .join(d7,'userID','outer') \\\n",
    "    .join(d8,'userID','outer') \\\n",
    "    .join(d9,'userID','outer') \\\n",
    "    .join(user_churn,'userID','outer') \\\n",
    "    .drop('userID') \\\n",
    "    .fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+--------+------------+--------------+---------------+-----------+----------+----+----------+-------+\n",
      "|       listen_time|num_song|num_thumb_up|num_thumb_down|add_to_playlist|         lt|add_friend|help|rolladvert|churn_d|\n",
      "+------------------+--------+------------+--------------+---------------+-----------+----------+----+----------+-------+\n",
      "| 66940.89735000003|     275|          17|             5|              7| 4807612000|         4|   2|        52|      0|\n",
      "| 94008.87593999993|     387|          21|             6|              8| 6054448000|         4|   2|         7|      0|\n",
      "|2089.1131000000005|       8|           0|             0|              0| 6161779000|         0|   0|         1|      1|\n",
      "|1012312.0927899999|    4079|         171|            41|            118|11366431000|        74|  23|         4|      0|\n",
      "| 523275.8428000004|    2111|         100|            21|             52| 1680985000|        28|  12|         0|      1|\n",
      "| 38034.08710000002|     150|           7|             1|              5| 6288035000|         1|   1|        16|      0|\n",
      "|477307.60581000015|    1914|          81|            14|             59| 4882773000|        31|   8|         1|      0|\n",
      "| 711344.9195400011|    2841|         163|            29|             72| 9568946000|        33|  17|        47|      1|\n",
      "|       198779.2919|     820|          58|             3|             24| 2035240000|        11|   9|         8|      0|\n",
      "| 67703.47208000004|     257|          17|             3|              7| 7351206000|         6|   2|         2|      1|\n",
      "| 483118.9038399997|    1928|          96|            17|             38| 5779219000|        41|  16|         2|      0|\n",
      "|20660.023910000007|      84|          11|             0|              1| 2062544000|         3|   1|        10|      0|\n",
      "| 447464.0146699989|    1797|          86|            16|             61| 4662657000|        29|  12|         8|      1|\n",
      "|159669.96303999983|     647|          40|             9|             20|10754921000|         6|   3|        39|      0|\n",
      "| 512449.8827599989|    2070|          95|            24|             67| 5756546000|        41|  13|        17|      0|\n",
      "| 897406.9802100015|    3632|         303|            28|            113| 6424576000|        63|  27|        11|      0|\n",
      "| 57633.17563999999|     230|          11|             5|              7| 5593438000|         7|   0|        30|      1|\n",
      "| 754517.5625700009|    3028|         154|            22|             89| 5192990000|        47|  28|        22|      1|\n",
      "| 286064.0256399999|    1125|          72|             9|             33| 6171072000|        12|   7|         3|      0|\n",
      "| 53062.74659000002|     215|           9|             3|              7| 7556771000|         7|   1|        21|      0|\n",
      "+------------------+--------+------------+--------------+---------------+-----------+----------+----+----------+-------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "dt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "225"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "incol = [\"listen_time\", \"num_song\", \"num_thumb_down\", \\\n",
    "          'num_thumb_up','add_to_playlist','lt','add_friend','help','rolladvert']\n",
    "assembler = VectorAssembler(inputCols=incol, outputCol=\"NumFeatures\")\n",
    "dt = assembler.transform(dt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler2 = StandardScaler(inputCol=\"NumFeatures\", outputCol=\"ScaledNumFeatures2\", withStd=True)\n",
    "scalerModel = scaler2.fit(dt)\n",
    "dt = scalerModel.transform(dt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(listen_time=66940.89735000003, num_song=275, num_thumb_up=17, num_thumb_down=5, add_to_playlist=7, lt=4807612000, add_friend=4, help=2, rolladvert=52, churn_d=0, NumFeatures=DenseVector([66940.8974, 275.0, 5.0, 17.0, 7.0, 4807612000.0, 4.0, 2.0, 52.0]), ScaledNumFeatures2=DenseVector([0.2431, 0.2489, 0.3823, 0.2596, 0.214, 1.4775, 0.1943, 0.2761, 2.413])),\n",
       " Row(listen_time=94008.87593999993, num_song=387, num_thumb_up=21, num_thumb_down=6, add_to_playlist=8, lt=6054448000, add_friend=4, help=2, rolladvert=7, churn_d=0, NumFeatures=DenseVector([94008.8759, 387.0, 6.0, 21.0, 8.0, 6054448000.0, 4.0, 2.0, 7.0]), ScaledNumFeatures2=DenseVector([0.3414, 0.3502, 0.4588, 0.3207, 0.2445, 1.8606, 0.1943, 0.2761, 0.3248]))]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt.take(2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modeling\n",
    "Split the full dataset into train, test, and validation sets. Test out several of the machine learning methods you learned. Evaluate the accuracy of the various models, tuning parameters as necessary. Determine your winning model based on test accuracy and report results on the validation set. Since the churned users are a fairly small subset, I suggest using F1 score as the metric to optimize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(label=0, features=DenseVector([0.2431, 0.2489, 0.3823, 0.2596, 0.214, 1.4775, 0.1943, 0.2761, 2.413]))]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt2 = dt.select(col('churn_d').alias('label'), col('ScaledNumFeatures2').alias('features'))\n",
    "dt2.take(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "rest,validation = dt2.randomSplit([0.8, 0.2], seed=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Models to train: logistic regression, svm, gradient boosting tree\n",
    "logistic_reg = LogisticRegression(maxIter=10, regParam =0.0)\n",
    "gbt = GBTClassifier(maxDepth = 5, maxIter = 10, seed = 42)\n",
    "svm = LinearSVC (maxIter = 10,  regParam = 0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.7570234861391317, 0.6766748159004269, 0.6668220687467296]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Logistic Regression\n",
    "e1 = MulticlassClassificationEvaluator(metricName='f1')\n",
    "paramGrid = ParamGridBuilder() \\\n",
    "    .addGrid(logistic_reg.regParam,[0.0, 0.05, 0.1]) \\\n",
    "    .build()\n",
    "crossval = CrossValidator(estimator=logistic_reg,\n",
    "                          estimatorParamMaps=paramGrid,\n",
    "                          evaluator=e1,\n",
    "                          numFolds=3)\n",
    "cvModel_q1 = crossval.fit(rest)\n",
    "cvModel_q1.avgMetrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.6700167079862496, 0.6700167079862496, 0.6700167079862496]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Support Vector Machine \n",
    "e1 = MulticlassClassificationEvaluator(metricName='f1')\n",
    "paramGrid = ParamGridBuilder() \\\n",
    "    .addGrid(svm.regParam,[0.01, 0.05, 0.5]) \\\n",
    "    .build()\n",
    "crossval = CrossValidator(estimator=svm,\n",
    "                          estimatorParamMaps=paramGrid,\n",
    "                          evaluator=e1,\n",
    "                          numFolds=3)\n",
    "cvModel_q2 = crossval.fit(rest)\n",
    "cvModel_q2.avgMetrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.702863203740007, 0.6896282778027504]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Gradient boosted tree\n",
    "e1 = MulticlassClassificationEvaluator(metricName='f1')\n",
    "paramGrid = ParamGridBuilder() \\\n",
    "    .addGrid(gbt.maxDepth,[5, 10]) \\\n",
    "    .build()\n",
    "crossval = CrossValidator(estimator=gbt,\n",
    "                          estimatorParamMaps=paramGrid,\n",
    "                          evaluator=e1,\n",
    "                          numFolds=3)\n",
    "cvModel_q3 = crossval.fit(rest)\n",
    "cvModel_q3.avgMetrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "gbt_tuned = GBTClassifier(maxDepth=5,maxIter=10,seed=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "gbt_model = gbt_tuned.fit(rest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = gbt_model.transform(validation)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7567567567567568\n"
     ]
    }
   ],
   "source": [
    "evaluator = MulticlassClassificationEvaluator(predictionCol=\"prediction\")\n",
    "print(evaluator.evaluate(results, {evaluator.metricName: \"accuracy\"}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.76094659252554\n"
     ]
    }
   ],
   "source": [
    "print(evaluator.evaluate(results, {evaluator.metricName: \"f1\"}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SparseVector(9, {0: 0.134, 1: 0.0408, 2: 0.1519, 3: 0.0639, 4: 0.0946, 5: 0.2436, 6: 0.0988, 7: 0.0301, 8: 0.1421})"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gbt_model.featureImportances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcMAAAEWCAYAAAAadfxCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XuYFNWd//H3RzSAghjFddGoowaDIgZlxPUafOKaq1FXXV0vkZgNuWjMjWRJ3CQkbhJd/W0k0WiIv8jGqMFbjIaNolHU4I3hIiMRLwFco24ULyMIosB3/6gza9HpmemZ6Zmenvq8nqefqa46p863zjR855yqrlJEYGZmVmSb1ToAMzOzWnMyNDOzwnMyNDOzwnMyNDOzwnMyNDOzwnMyNDOzwnMyNDOzwnMyNCtD0smSHpL0uqQX0vLnJCltnyHpTUmrJa2SNF/S+9K2b6T1qyW9IWlD7v2SMm01SIpcmdWSHqnCMUyV9Mvu7qeTba6QdGRvttkWSXMk/XOt47D64GRoVkLSV4BpwIXA3wI7AJ8BDgHekSv67xExBBgGXAbcJGlARHw/IoakbZ8BHmh9HxGj22l6m1y59/bEsXWGpM1rHUNXKOP/26xT/IExy5E0DPgu8LmIuCEiVkVmYUScGhHrSutExEbgGmBbssRZ7ZjOlPSYpFck3S5p19y2aZKekfRaGp0eltZ/EPgGcFJ+pFk6csuPHnMj1E9K+m/grrT+7yTdL+lVSY9ImlBh3BMlzZX0w1R3maSD0/pn0oj7jFz5GZIul3RHGm3fU3KsB0uaJ6kl/Tw4t22OpO9JmgusAa4CDgMuScd/SXv9leuL6yT9IrW/RFJjbvvOkm6S9KKkl1r32d7vKCXmH6ZjbZG0WNI+lfSf9S4nQ7NNHQQMBH5TaQVJA4CPA8uBv1QzGEnHkiW1fwC2B+4Drs0VmQeMJUvE1wDXSxoUEbcB3wdmdmGk+T5gL+ADknYCZgH/ltqYDNwoafsK93UgsBjYLsX3K+AA4N3AaWTJakiu/KnAecBwYBFwdeqHbVMcP0r7+g9glqTtcnVPByYBQ4GJZH11djr+s1OZsv2V28fHUozbALcArUl0APBb4GmgAdgplevod3QUcDiwZ9rnScBLFfad9SInQ7NNDQdWRsT61hW5UdFaSYfnyk6W9CrwOnAx8M2I2NCNtlemdl6VNDmt+zTwg4h4LMX0fWBs68gjIn4ZES9FxPqI+H9kifw93YgBYGpEvB4Ra8kS1n9FxH9FxMaIuANoAj5c4b6WR8SVqV9mAjsD342IdRExG3iTLDG2mhUR96YR+LnAQZJ2Bj4CPBkRV6VjvRZYChydqzsjIpak7W+VC6aC/vpDOtYNZKPL1j8ixgM7Al9NffNGRPwhbWvvd/QWWXIeBSiVeb7CvrNe5GRotqmXgOH582URcXBEbJO25f/NXJTWDwYagQslfagbbQ+PiG3S66K0bldgWmuSBF4GRDYyQdJX0vRcS9o+jCyhd8czueVdgRNzSfpV4FBgRIX7yo+U1wJEROm6/Mjw/9qOiNVkx7tjej1dsu+nSf1QJu6yKuiv/8ktrwEGpc/CzsDT+T+Sctr8HUXEXWSjy0uBv0iaLmnrjuK03udkaLapB4B1wDGVVkjnFB8F5pKNYKrpGeDTuSS5TUQMjoj70/mufwH+EXhnSswtZP8RA5R7JM3rwJa5939bpky+3jPAVSXtbxUR53f7yMrbuXUhTZ9uCzyXXruWlN0FeLaNuP/qfQX91Z5ngF1U/qKiNn9HABHxo4gYB4wmmy79agXtWS9zMjTLiYhXge8AP5F0gqQhkjaTNBbYqq16kkaRjZj+6qsT3XQ58HVJo1M7wySdmLYNBdYDLwKbS/oWkB91/AVo0KZXVi4CTpa0Rbo45IQO2v8lcLSkD0gaIGmQpAmS3lWFYyvnw5IOlfQOsnOHD0XEM8B/AXtKOkXS5pJOAvYmO4/Xlr8Au+fed9Rf7XkYeB44X9JWqR8OSdva/B1JOkDSgZK2IPtD5A2gO1Pp1kOcDM1KRMS/A18Gvga8QPaf6k/JRhX354p+LV2p+DowG7gylatmLL8GLgB+Jek14FGgdSr2duB3wBNkU4ZvsOlU4fXp50uSFqTlbwJ7AK+QJf1rOmj/GbJR8jfIksgzZCObnvq/4xrg22RTjePILqghIl4CPgp8hWy6+mvARyNiZTv7mgackK7w/BEd91eb0jnEo8nOb/438Geyi2E6+h1tDfyMrL+fTrFfhPU58sN9zawvkDQD+HNE/GutY7Hi8cjQzMwKz8nQzMwKz9OkZmZWeB4ZmplZ4dXljXiLaPjw4dHQ0FDrMMzM6sr8+fNXRkSHtw90MqwTDQ0NNDU11ToMM7O6Iqn0zkVleZrUzMwKz8nQzMwKz8nQzMwKz8nQzMwKz8nQzMwKz8nQzMwKz8nQzMwKz8nQzMwKz1+6rxPNz7bQMGVWrcMw61Erzv9IrUOwgvLI0MzMCs/J0MzMCs/J0MzMCs/J0MzMCs/J0MzMCq+QyVDSCknD0/LqTtadKmlyleI4VtLe1diXmZl1Xb9Nhsr02eOTtDlwLOBkaGZWY302WXSFpAZJj0n6CbAAOF1Ss6RHJV3QQd0hkn4vaUGqc0xu27mSHpd0J/CetG4vSQ+XtL04LY+TdI+k+ZJulzQirZ8j6fuS7gH+BfgYcKGkRZL2qHZ/mJlZZfrjl+7fA3wC+DfgQWAc8AowW9KxEXFzG/XeAI6LiNfSFOqDkm4B9gdOBvYj668FwPyIeEzSOyTtHhHLgJOA6yRtAfwYOCYiXpR0EvA94MzUzjYR8T4ASSOB30bEDeUCkjQJmAQwYOvtu9MnZmbWjn41MkyejogHgQOAORHxYkSsB64GDm+nnoDvp9HdncBOwA7AYcCvI2JNRLwG3JKrcx3wj2n5JGAmWTLeB7hD0iLgX4F35erMrPRAImJ6RDRGROOALYdVWs3MzDqpP44MX08/1cl6pwLbA+Mi4i1JK4BBaVu0UWcmcL2km4CIiCcljQGWRMRBHcRnZmZ9RH8cGbZ6CHifpOGSBgD/BNzTTvlhwAspER4B7JrW3wscJ2mwpKHA0a0VIuJPwAbgm7w94nsc2F7SQQCStpA0uo02VwFDu3Z4ZmZWLf02GUbE88DXgbuBR4AFEfGbdqpcDTRKaiIbJS5N+1lAlugWATcC95XUmwmcRjZlSkS8CZwAXCDpkVTv4Dba/BXwVUkLfQGNmVntKKKtGUDrSwaOGBkjzri41mGY9Sg/tcKqTdL8iGjsqFy/HRmamZlVysnQzMwKz8nQzMwKrz9+taJfGrPTMJp8PsXMrEd4ZGhmZoXnZGhmZoXnZGhmZoXnZGhmZoXnC2jqRPOzLTRMmVXrMMryF6XNrN55ZGhmZoXnZGhmZoXnZGhmZoXnZGhmZoXnZGhmZoXnZNhFkhokPdqJ8lMlTe7JmMzMrGucDM3MrPCcDLtngKSfSVoiabakwZL2kHSbpPmS7pM0qrSSpDmSLpZ0v6RHJY2vRfBmZpZxMuyekcClETEaeBU4HpgOfD4ixgGTgZ+0UXeriDgY+Bzw83IFJE2S1CSpacOalupHb2ZmgO9A013LI2JRWp4PNAAHA9dLai0zsI261wJExL2Stpa0TUS8mi8QEdPJkisDR4yMKsduZmaJk2H3rMstbwB2AF6NiLEV1C1Nbk52ZmY14mnS6noNWC7pRABl3ttG2ZNSmUOBlojwPKiZWY04GVbfqcAnJT0CLAGOaaPcK5LuBy4HPtlbwZmZ2V/zNGkXRcQKYJ/c+4tymz9YpvzUklU3RsTXeyQ4MzPrFI8Mzcys8DwyrIGImFDrGMzM7G0eGZqZWeF5ZFgnxuw0jCY/Ud7MrEd4ZGhmZoXnZGhmZoXnZGhmZoXnc4Z1ovnZFhqmzKp1GFZDK3zO2KzHeGRoZmaF52RoZmaF52RoZmaF52RoZmaF52RoZmaF52RoZmaF16+ToaSJki5pY9vqDupeKGmJpAvLbPuYpClVinGqpMnV2JeZmXWNv2fYtk8D20fEuvxKSZtHxC3ALbUJy8zMqq2uk6Gkm4GdgUHAtIiYLukTwNeB54EngHWp7G7ANWTHfFsH+70F2Ap4SNIPgA8BLwP7AQskNQONEXG2pO3Jnla/S6r+xYiYK2lqWrd7+nlxRPwo7f9c4OPAM8CLwPw24pgETAIYsPX2nescMzOrWF0nQ+DMiHhZ0mBgnqRZwHeAcUALcDewMJWdBlwWEb+QdFZ7O42Ij0laHRFjASR9CNgTODIiNkiamCs+DfhhRPxB0i7A7cBeadso4AhgKPC4pMuAfYGTyRLr5sAC2kiGETEdmA4wcMTIqLRTzMysc+o9GZ4j6bi0vDNwOjAnIl4EkDSTLIkBHAIcn5avAi7oZFvXR8SGMuuPBPaW1Pp+a0lD0/KsNM26TtILwA7AYcCvI2JNitHTrWZmNVa3yVDSBLJEdFBErJE0B1jK26Oycrozunq9jfWbpRjWlsQHaYo22cDb/e1RnplZH1LPV5MOA15JiXAU8HfAYGCCpO0kbQGcmCs/l2x6EuDUKsYxGzi79Y2ksR2Uvxc4TtLgNII8uoqxmJlZF9RzMrwN2FzSYuA84EGyi2amAg8Ad5Kdj2v1BeAsSfPIEmm1nAM0Slos6Y/AZ9orHBELgJnAIuBG4L4qxmJmZl2gCM/Y1YOBI0bGiDMurnUYVkN+hJNZ50maHxGNHZWr55GhmZlZVdTtBTTVIGkM2ZWleesi4sBaxGNmZrXhadI60djYGE1NTbUOw8ysrnia1MzMrEJOhmZmVnhOhmZmVnhOhmZmVniFvpq0njQ/20LDlFm1DsPMrFf11vdrPTI0M7PCczI0M7PCczI0M7PCczI0M7PCczKsAUmr088GSafUOh4zs6JzMqytBsDJ0MysxpwMa+t84DBJiyR9qdbBmJkVlb9nWFtTgMkR8dFaB2JmVmQeGfZhkiZJapLUtGFNS63DMTPrt5wM+7CImB4RjRHROGDLYbUOx8ys33IyrK1VwNBaB2FmVnROhrW1GFgv6RFfQGNmVju+gKYGImJI+vkW8P4ah2NmVngeGZqZWeE5GZqZWeE5GZqZWeE5GZqZWeH5Apo6MWanYTT10hOfzcyKxiNDMzMrPCdDMzMrPCdDMzMrPJ8zrBPNz7bQMGVWrcOwXrLC54fNepVHhmZmVnhOhmZmVnhOhmZmVnhOhmZmVnhOhmZmVnhOhmZmVnh9JhlKmijpkja2rW6nXoOkU3ognjbb7KBeg6RH03KjpB91ULbqsZuZWef0mWTYDQ1An0woEdEUEee0U6SBPhq7mVmR9FoylHSzpPmSlkialNZ9QtITku4BDsmV3U3SA5LmSTqvg12fDxwmaZGkL0kaJOlKSc2SFko6op2YJkr6jaTbJD0u6dtlygyR9HtJC9I+j0nrz5P0hVy570k6p6TuBEm/TcvvSzEuSnENLY29TNuTJDVJatqwpqWDbjAzs67qzTvQnBkRL0saDMyTNAv4DjAOaAHuBhamstOAyyLiF5LO6mC/U4DJEfFRAElfAYiIMZJGAbMl7RkRb7RRfzywD7CmNa6IaMptfwM4LiJekzQceFDSLcD/B24CpknaDDg57WtoG+1MBs6KiLmShqT9bhJ7qYiYDkwHGDhiZHTQD2Zm1kW9OU16jqRHgAeBnYHTgTkR8WJEvAnMzJU9BLg2LV/VyXYOba0TEUuBp4E92yl/R0S8FBFryZLboSXbBXxf0mLgTmAnYIeIWAG8JGk/4ChgYUS81E47c4H/SKPHbSJifSePy8zMekivjAwlTQCOBA6KiDWS5gBLgb3aqdbVkZA6Wb60ndL3pwLbA+Mi4i1JK4BBadsVwETgb4Gft9tIxPlpNPxhstHlkZ2M08zMekhvjQyHAa+kRDgK+DtgMDBB0naStgBOzJWfSzbtCFkyas8qNp2avLe1jqQ9gV2Ax9up//eStk3Tt8emtktjfyElwiOAXXPbfg18EDgAuL29ICXtERHNEXEB0ASMKhO7mZnVQG8lw9uAzdNU43lkU6XPA1OBB8imHxfkyn8BOEvSPLJk1J7FwHpJj6SLUH4CDJDUTDb1OjEi1rVT/w9k06qLgBtLzhcCXA00SmoiS7JLWzek6d27gesiYkMHcX5R0qNpqngt8LsysZuZWQ0oorjXZUiaCDRGxNldrL8ZWRI/MSKerGZspQaOGBkjzri4J5uwPsSPcDKrDknzI6Kxo3L94XuGNSFpb+Ap4Pc9nQjNzKxn1c3DfSWN4a+vLF0XEQdWUPcDwAUlq5dHxHHAjK7EExF/BHbvSl0zM+tbCj1NWk8aGxujqan0dKaZmbXH06RmZmYVcjI0M7PCczI0M7PCczI0M7PCq5urSYuu+dkWGqbMqnUY/Ya/x2dmeR4ZmplZ4TkZmplZ4TkZmplZ4TkZmplZ4TkZmplZ4fWLZChpoqQdc+9XSBreA+1MlTS52vs1M7Pa6hfJkOxp8zt2VMjMzKycHkuGkhokPSbpZ5KWSJotabCkOZIaU5nhklak5YmSbpZ0q6Tlks6W9GVJCyU9KGnbNto5AWgErpa0KD2xHuDzkhZIapY0KpXdZGSXHrbbkF5LJV2R1l0t6UhJcyU9KWl8rsn3Srorrf9UO8c/QdJvc+8vSc9PbB25XiDp4fR6d1f62MzMqqOnR4YjgUsjYjTwKnB8B+X3AU4BxgPfA9ZExH7AA8DHy1WIiBuAJuDUiBgbEWvTppURsT9wGVDJ1Oa7gWnAvsCoFMehqe43cuX2BT4CHAR8Kz8920mvRcR44BKg7FN7JU2S1CSpacOali42Y2ZmHenpZLg8Ihal5flAQwfl746IVRHxItAC3JrWN1dQt9RNnWgXslibI2IjsITsob1Rpu3fRMTaiFgJ3E2WuLvi2tzPg8oViIjpEdEYEY0DthzWxWbMzKwjPZ0M1+WWN5Dd/m19rt1B7ZTfmHu/kc7fOq61bmu7lLRd2n6lbZc+ALKtB0K211ZpPT9U0syshmpxAc0KYFxaPqFK+1wFDK2w7f0BJO0P7NaFto6RNEjSdsAEYF4b5Z4G9pY0UNIw4P0l20/K/XygC3GYmVmV1OJG3RcB10k6HbirSvucAVwuaS1tTDkmNwIfl7SILIk90YW2HgZmAbsA50XEc+UKRcQzkq4DFgNPAgtLigyU9BDZHyT/1IU4zMysSpSdFrPelK6gbUznHSsycMTIGHFG2etsrAv81AqzYpA0PyIaOyrXX75naGZm1mV19TxDSZcCh5SsnhYRV9YiHgBJY4CrSlavi4gD26oTEQ09GpSZmXVKXSXDiDir1jGUiohmYGyt4zAzs66rq2RYZGN2GkaTz3OZmfUInzM0M7PCczI0M7PCczI0M7PC8znDOtH8bAsNU2bVOgyzPsffGbVq8MjQzMwKz8nQzMwKz8nQzMwKz8nQzMwKz8nQzMwKz8nQzMwKr08lQ0kTJe2Ye79C0vAeaGeqpMldrDtDUrUeSmxmZn1An0qGwERgx44KmZmZVVOHyVBSg6THJP1M0hJJsyUNljRHUmMqMzw9sLZ1dHezpFslLZd0tqQvS1oo6UFJ27bRzglAI3C1pEWSBqdNn5e0QFKzpFGp7CYjO0mPpjgbJC2VdEVad7WkIyXNlfSkpPG5Jt8r6a60/lPtHL8kXSLpj5JmAX+T2/b+dFzNkn4uaaCk8ZJuStuPkbRW0jskDZK0LK2fI+kCSQ9LekLSYW20PUlSk6SmDWtaOvpVmZlZF1U6MhwJXBoRo4FXgeM7KL8PcAowHvgesCYi9gMeAD5erkJE3AA0AadGxNiIWJs2rYyI/YHLgEqmNt8NTAP2BUalOA5Ndb+RK7cv8BHgIOBb+enZEscB7wHGAJ8CDgaQNAiYAZwUEWPI7ubzWWABsF+qexjwKHAAcCDwUG6/m0fEeOCLwLfLNRwR0yOiMSIaB2w5rIJDNzOzrqg0GS6PiEVpeT7Q0EH5uyNiVUS8CLQAt6b1zRXULXVTJ9qFLNbmiNgILAF+HxFRpu3fRMTaiFgJ3E2WuMs5HLg2IjZExHPAXWn9e1JbT6T3/wkcHhHrgack7ZX2+R9pH4cB93XjuMzMrIdUmgzX5ZY3kI2C1ufqD2qn/Mbc+410/n6orXVb26Wk7dL2K207Stopfd/RNrVT/j7gQ8BbwJ1kI9NDgXvLxJk/LjMzq4HuXECzAhiXlqt1deUqYGiFbe8PIGl/YLcutHVMOo+3HTABmNdGuXuBkyUNkDQCOCKtXwo0SHp3en86cE+uzheBB9LoeDuyKdslXYjTzMx6WHeS4UXAZyXdD1Tr6w8zgMtLLqAp50ZgW0mLyM7TPdFO2bY8DMwCHgTOS1Og5fwaeJJsmvUyUsKLiDeATwDXS2omG3lenuo8BOzA2yPBxcDiNF1rZmZ9jPz/c30YOGJkjDjj4lqHYdbn+BFO1h5J8yOisaNyfe17hmZmZr2uJhduSLoUOKRk9bSIuLIW8QBIGgNcVbJ6XUQcWIt4zMys93iatE40NjZGU1NTrcMwM6srniY1MzOrkJOhmZkVnpOhmZkVnu98Uiean22hYcqsWodRVb4k3sz6Co8Mzcys8JwMzcys8JwMzcys8JwMzcys8JwMzcys8JwMzcys8JwMzcys8JwMzcys8Oo+GUpqkPSYpJ9JWiJptqTBkuZIakxlhktakZYnSrpZ0q2Slks6W9KXJS2U9KCkbdtp6xxJf5S0WNKv0rpt0/4Wp/r7pvVTJf08xbFM0jm5/XxT0lJJd0i6VtLkNtqbJKlJUtOGNS1V7DUzM8ur+2SYjAQujYjRwKvA8R2U3wc4BRgPfA9YExH7AQ8AH2+n3hRgv4jYF/hMWvcdYGFa9w3gF7nyo4APpHa+LWmLlKCPB/YD/gFo827qETE9IhojonHAlsM6OCQzM+uq/pIMl0fEorQ8H2jooPzdEbEqIl4EWoBb0/rmDuouBq6WdBqwPq07lPQcxIi4C9hOUmvmmhUR6yJiJfACsEMq/5uIWBsRq3Jtm5lZjfSXZLgut7yB7J6r63n7+Aa1U35j7v1G2r9f60eAS4FxwHxJmwMqU671IZHl4ipX3szMaqi/JMNyVpAlLYATurszSZsBO0fE3cDXgG2AIcC9wKmpzARgZUS81s6u/gAcLWmQpCFkCdbMzGqoPz+14iLgOkmnA3dVYX8DgF+mKVABP4yIVyVNBa6UtBhYA5zR3k4iYp6kW4BHgKeBJrKpWjMzqxFFRMelrKokDYmI1ZK2JBtZToqIBe3VGThiZIw44+LeCbCX+BFOZtbTJM2PiDYvVGzVn0eGfdl0SXuTncv8z44SoZmZ9SwnwzIkXQocUrJ6WkRcWY39R8Qp1diPmZlVh6dJ60RjY2M0NTXVOgwzs7pS6TRpf76a1MzMrCJOhmZmVnhOhmZmVnhOhmZmVni+mrROND/bQsOUWbUOw6zT/H1SqwceGZqZWeE5GZqZWeE5GZqZWeE5GZqZWeE5GZqZWeE5GZqZWeHVZTKUtDr93FHSDe2U20bS53qg/YmSdsy9vyI9hcLMzOpQXSbDVhHxXES09xT7bYCqJ0NgIvB/yTAi/jki/tgD7ZiZWS+o62QoqUHSo2l5tKSHJS2StFjSSOB8YI+07sJU7quS5qUy38nt5zFJP5O0RNJsSYPbaPMEoBG4Ou13sKQ5khrT9tWSLpA0X9Kdksan7cskfSyVGSDpwlwcn26jrUmSmiQ1bVjTUu3uMzOzpK6TYYnPkD1zcCxZsvozMAX4U0SMjYivSjoKGAmMB8YC4yQdnuqPBC6NiNHAq8Dx5RqJiBuAJuDUtN+1JUW2AuZExDhgFfBvwN8DxwHfTWU+CbRExAHAAcCnJO1Wpq3pEdEYEY0DthzWlT4xM7MK9KfbsT0AnCvpXcBNEfGkpNIyR6XXwvR+CFkS/G9geUQsSuvnAw1djONN4La03Aysi4i3JDXn9nkUsG8aZQIMS3Es72KbZmbWDf0mGUbENZIeAj4C3C7pn4FlJcUE/CAifrrJSqkBWJdbtQEoO01agbfi7Scmb2zdb0RslNTa3wI+HxG3d7ENMzOron4zTSppd2BZRPwIuAXYl2yacmiu2O3AmZKGpDo7SfqbLjRXut/Ouh34rKQtUhx7StqqG/szM7Nu6DcjQ+Ak4DRJbwH/A3w3Il6WNDddZPO7dN5wL+CBNIW6GjiNbCTYGTOAyyWtBQ7qQqxXkE2ZLlAWyIvAsV3Yj5mZVYHentGzvmzgiJEx4oyLax2GWaf5EU5WS5LmR0RjR+X6zTSpmZlZV/WnadKqk3QpcEjJ6mkRcWUt4jEzs57hadI60djYGE1NTbUOw8ysrnia1MzMrEJOhmZmVnhOhmZmVnhOhmZmVnhOhmZmVnhOhmZmVnhOhmZmVnhOhmZmVnhOhmZmVni+A02dkLQKeLzWcdTYcGBlrYPoA9wP7oNW7oeO+2DXiNi+o5343qT14/FKbinUn0lqKnofgPsB3Aet3A/V6wNPk5qZWeE5GZqZWeE5GdaP6bUOoA9wH2TcD+6DVu6HKvWBL6AxM7PC88jQzMwKz8nQzMwKz8mwD5D0QUmPS3pK0pQy2wdKmpm2PySpIbft62n945I+0JtxV1NX+0BSg6S1khal1+W9HXu1VNAHh0taIGm9pBNKtp0h6cn0OqP3oq6+bvbDhtxn4Zbei7q6KuiDL0v6o6TFkn4vadfctiJ9Ftrrh859FiLCrxq+gAHAn4DdgXcAjwB7l5T5HHB5Wj4ZmJmW907lBwK7pf0MqPUx9XIfNACP1voYeqkPGoB9gV8AJ+TWbwssSz/fmZbfWetj6u1+SNtW1/oYeqkPjgC2TMufzf17KNpnoWw/dOWz4JFh7Y0HnoqIZRHxJvAr4JiSMscA/5mWbwDeL0lp/a8iYl1ELAeeSvurN93pg/6iwz6IiBURsRjYWFL3A8AdEfFyRLwC3AF8sDeC7gHd6Yf+opI+uDsi1qS3DwLvSstF+yy01Q+d5mRYezsBz+Te/zmtK1smItYDLcB2FdatB93pA4DdJC2UdI+kw3o62B7Snd9lf/kcQPePZZCkJkkPSjq2uqH1ms72wSeB33XMtgGvAAAFJ0lEQVSxbl/WnX6ATn4WfDu22is3uin9vktbZSqpWw+60wfPA7tExEuSxgE3SxodEa9VO8ge1p3fZX/5HED3j2WXiHhO0u7AXZKaI+JPVYqtt1TcB5JOAxqB93W2bh3oTj9AJz8LHhnW3p+BnXPv3wU811YZSZsDw4CXK6xbD7rcB2mK+CWAiJhPdo5hzx6PuPq687vsL58D6OaxRMRz6ecyYA6wXzWD6yUV9YGkI4FzgY9FxLrO1K0T3emHzn8Wan2StOgvstH5MrILYFpPEo8uKXMWm148cl1aHs2mF9Asoz4voOlOH2zfesxkJ9qfBbat9TH1RB/kys7gry+gWU52wcQ703Ld9UEV+uGdwMC0PBx4kpILLurhVeG/h/3I/vAbWbK+UJ+Fdvqh05+Fmh+wXwHwYeCJ9Es9N637LtlfOgCDgOvJLpB5GNg9V/fcVO9x4EO1Ppbe7gPgeGBJ+oeyADi61sfSg31wANlfy68DLwFLcnXPTH3zFPCJWh9LLfoBOBhoTp+FZuCTtT6WHuyDO4G/AIvS65aCfhbK9kNXPgu+HZuZmRWezxmamVnhORmamVnhORmamVnhORmamVnhORmamVnhORma9UGSVvdyew2STunNNkvaP1fSkvT0gUWSDqxVLFZMvh2bWcGlO/o0AKcA19Sg/YOAjwL7R8Q6ScPJvmTdnX1uHtk9bM0q4pGhWR8maUK6Afl1kp6QdL6kUyU9LKlZ0h6p3AxJl0u6L5X7aFo/SNKVqexCSUek9RMlXS/pVmA2cD5wWBqVfSmNFO9Lzw1cIOngXDxzJN0gaamkq1ufHiLpAEn3S3okxTdU0gBJF0qal0Z9ny5zmCOAlZFupRURKyPdSquNfVZ6TEj6aq7t7/Tcb8rqnUeGZn3fe4G9yO5Huwy4IiLGS/oC8Hngi6lcA9mNivcA7pb0brLb2BERYySNAmZLar1360HAvhHxsqQJwOSIaE2iWwJ/HxFvSBoJXEt2I2TIboE1muw+kXOBQyQ9DMwEToqIeZK2BtaSPUmgJSIOkDQQmCtpdmSPHGs1G/iWpCfI7igyMyLukfSONvb5hQqP6ShgJNmjgATcIunwiLi3a78G68+cDM36vnkR8TyApD+RRj1kt5k6IlfuuojYCDwpaRkwCjgU+DFARCyV9DRv38j8joh4uY02twAukTQW2MCmNz9/OCL+nOJZRJaEW4DnI2Jeauu1tP0oYF+9/UT6YWQJ6v+SYUSsTk8cOSwdz0xlTzWf38Y+Kz2mo9JrYXo/JLXtZGh/xcnQrO9bl1vemHu/kU3/DZfeW7Gtx3y1er2dbV8iu+fje8lOp7zRRjwbUgwq0z5p/ecj4vZ22iIiNpA9WWCOpGbgDLJ7zba1z7bkj0nADyLip+21bQY+Z2jWn5woabN0HnF3spu33wucCpCmEndJ60utAobm3g8jG5VtBE4HBnTQ9lJgR0kHpLaGpgtzbgc+K2mL1hgkbZWvKOk9aSq21Vjg6Xb2Wekx3Q6cKWlIKruTpL/p4DisoDwyNOs/HgfuAXYAPpPO9/0EuDyNttYDE9MVm6V1FwPrJT1C9miknwA3SjoRuJv2R5FExJuSTgJ+LGkw2bm9I4EryKZRF6QLbV4ESp86PiTV2ybF+BQwqZ19VnRMETFb0l7AA2nbauA04IX2jsWKyU+tMOsHJM0AfhsRN9Q6FrN65GlSMzMrPI8Mzcys8DwyNDOzwnMyNDOzwnMyNDOzwnMyNDOzwnMyNDOzwvtf3FNFz7r5dDQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7ff46c2cd2b0>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "importances  = [ 0.134,0.0408,0.1519,0.0639,0.0946,0.2436,0.0988,0.0301,0.1421]\n",
    "feature = [\"listen_time\", \"num_song\", \"num_thumb_down\", \\\n",
    "          'num_thumb_up','add_to_playlist','lt','add_friend','help','rolladvert']\n",
    "y_pos = np.arange(len(feature))\n",
    " \n",
    "plt.barh(y_pos, importances, align='center')\n",
    "plt.yticks(y_pos, feature)\n",
    "plt.xlabel('Importance Score')\n",
    "plt.title('GBT Feature Importances')\n",
    "plt.savefig('GBT feature Importance.png', dpi=300)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Steps\n",
    "Clean up your code, adding comments and renaming variables to make the code easier to read and maintain. Refer to the Spark Project Overview page and Data Scientist Capstone Project Rubric to make sure you are including all components of the capstone project and meet all expectations. Remember, this includes thorough documentation in a README file in a Github repository, as well as a web app or blog post."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}