9. Advanced pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "#### Over time, pandas has developed depth of features for power users.\n",
    "#### To learn these advanced features will help you deepen your expertise as pandas user."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorical Data\n",
    "#### We can use the pandas categorical data type to achieve better performance and memory use.\n",
    "#### We can also use it in statistics and machine learning applications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Background and Motivation\n",
    "#### Sometimes a column in a table may contain repested instances of smaller set of distinct values.\n",
    "#### We know about methods like 'unique' (allows us to extract distinct values from array) and 'value_counts' (allows us to compute their frequencies). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     apple\n",
       "1    orange\n",
       "2     apple\n",
       "3     apple\n",
       "4     apple\n",
       "5    orange\n",
       "6     apple\n",
       "7     apple\n",
       "dtype: object"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "values = pd.Series(['apple', 'orange', 'apple',\n",
    "                   'apple'] * 2)\n",
    "\n",
    "values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['apple', 'orange'], dtype=object)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.unique(values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "apple     6\n",
       "orange    2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.value_counts(values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Systems (used for data warehousing, statistical computing, etc.) have developed specialized approaches to represent data with repeated values.\n",
    "#### In Data Warehousing, a best practice is to use 'dimension tables' having distinct values. Primary observations are stored as integer keys referncing teh dimension table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    1\n",
       "2    0\n",
       "3    0\n",
       "4    0\n",
       "5    1\n",
       "6    0\n",
       "7    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "values = pd.Series([0,1,0,0] * 2)\n",
    "dim = pd.Series(['apple', 'orange'])\n",
    "\n",
    "values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     apple\n",
       "1    orange\n",
       "dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We can use the 'take' method to restore the original Series of strings in above example.\n",
    "#### This representation as integers is called 'categorical' or 'dictionary-encoded' representation.\n",
    "#### The array of distinct values can be called 'categories', 'dictionary' or 'levels' of data.\n",
    "#### The integer values that refence the categories are called 'category codes' or simply 'codes'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     apple\n",
       "1    orange\n",
       "0     apple\n",
       "0     apple\n",
       "0     apple\n",
       "1    orange\n",
       "0     apple\n",
       "0     apple\n",
       "dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dim.take(values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Categorical representation yields significant performance improvements when doing analytics.\n",
    "#### We can also perform transformations on these categories without modifying original data. eg - renaming categories, appending new category without changing position or order of older category, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Categorical Type in pandas\n",
    "#### pandas has special 'Categorical' type for holding data that uses integer based categorical representation or 'encoding'.\n",
    "#### We can convert an array of any data type into categorical by using \"astype('category')\" on the array. This results in an instance of 'pandas.Categorical'.\n",
    "#### the Categorical object has 'categorical' and 'codes' atributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>basket_id</th>\n",
       "      <th>fruit</th>\n",
       "      <th>count</th>\n",
       "      <th>weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>apple</td>\n",
       "      <td>6</td>\n",
       "      <td>3.771870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>orange</td>\n",
       "      <td>10</td>\n",
       "      <td>0.033614</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>apple</td>\n",
       "      <td>12</td>\n",
       "      <td>2.829285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>apple</td>\n",
       "      <td>3</td>\n",
       "      <td>2.143480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>apple</td>\n",
       "      <td>3</td>\n",
       "      <td>3.191547</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>orange</td>\n",
       "      <td>9</td>\n",
       "      <td>1.303247</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>6</td>\n",
       "      <td>apple</td>\n",
       "      <td>9</td>\n",
       "      <td>0.865479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7</td>\n",
       "      <td>apple</td>\n",
       "      <td>8</td>\n",
       "      <td>0.617000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   basket_id   fruit  count    weight\n",
       "0          0   apple      6  3.771870\n",
       "1          1  orange     10  0.033614\n",
       "2          2   apple     12  2.829285\n",
       "3          3   apple      3  2.143480\n",
       "4          4   apple      3  3.191547\n",
       "5          5  orange      9  1.303247\n",
       "6          6   apple      9  0.865479\n",
       "7          7   apple      8  0.617000"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fruits = ['apple', 'orange', 'apple', 'apple'] * 2\n",
    "N = len(fruits)\n",
    "\n",
    "df = pd.DataFrame({'fruit': fruits,\n",
    "                  'basket_id': np.arange(N),\n",
    "                  'count': np.random.randint(3, 15, size=N),\n",
    "                  'weight': np.random.uniform(0, 4, size=N)},\n",
    "                 columns = ['basket_id', 'fruit', 'count', 'weight'])\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     apple\n",
       "1    orange\n",
       "2     apple\n",
       "3     apple\n",
       "4     apple\n",
       "5    orange\n",
       "6     apple\n",
       "7     apple\n",
       "Name: fruit, dtype: category\n",
       "Categories (2, object): [apple, orange]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fruit_cat = df['fruit'].astype('category')\n",
    "\n",
    "fruit_cat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.arrays.categorical.Categorical"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = fruit_cat.values\n",
    "\n",
    "type(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['apple', 'orange'], dtype='object')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 0, 0, 0, 1, 0, 0], dtype=int8)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.codes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### You can convert a DataFramecolumn to categorical by assigning it the converted result.\n",
    "#### You can convert other types of Python Sequences into categorical data type as well.\n",
    "#### In case you have received the categorical encoded data (i.e. category and code) from another source, you can use 'from_codes' to create a categorical data out of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     apple\n",
       "1    orange\n",
       "2     apple\n",
       "3     apple\n",
       "4     apple\n",
       "5    orange\n",
       "6     apple\n",
       "7     apple\n",
       "Name: fruit, dtype: category\n",
       "Categories (2, object): [apple, orange]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['fruit'] = df['fruit'].astype('category')\n",
    "\n",
    "df.fruit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[foo, bar, baz, foo, bar]\n",
       "Categories (3, object): [bar, baz, foo]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating category out of Python list.\n",
    "my_categories = pd.Categorical(['foo', 'bar', 'baz', 'foo', 'bar'])\n",
    "\n",
    "my_categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[foo, bar, baz, foo, foo, bar]\n",
       "Categories (3, object): [foo, bar, baz]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categories = ['foo', 'bar', 'baz']\n",
    "codes = [0, 1, 2, 0, 0, 1]\n",
    "\n",
    "my_cats_2 = pd.Categorical.from_codes(codes, categories)\n",
    "\n",
    "my_cats_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Categorical data have no specific ordering unless specifically defined.\n",
    "#### So the 'categories' array may be in different order based on ordering of input data.\n",
    "#### When using 'from_codes' you have the option to indeicate that there is a meaningful order in the categories.\n",
    "#### When looking at the data in output, you can also see the ordering of the categories.\n",
    "#### An unordered categorical instance can be made ordered with 'as_ordered'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[foo, bar, baz, foo, foo, bar]\n",
       "Categories (3, object): [foo < bar < baz]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ordered_cat = pd.Categorical.from_codes(codes, categories,\n",
    "                                       ordered=True)\n",
    "# Shows that 'foo' precedes 'bar' in ordering and so on.\n",
    "ordered_cat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[foo, bar, baz, foo, foo, bar]\n",
       "Categories (3, object): [foo < bar < baz]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_cats_2.as_ordered()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computations with Categoricals\n",
    "#### Using Categorical data instead of the original data (eg - array of strings) behaves the same way.\n",
    "#### Some parts of pandas (eg - groupby) work better with categoricals. Some other functions utilize the ordered flag as well.\n",
    "#### Some functions (eg - pandas.qcut binning function) returns categorical data. The labels are the quartiles itself. We can change the labels by passing 'labels' argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.20470766,  0.47894334, -0.51943872, -0.5557303 ,  1.96578057])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(12345)\n",
    "draws = np.random.randn(1000)\n",
    "\n",
    "draws[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(-0.684, -0.0101], (-0.0101, 0.63], (-0.684, -0.0101], (-0.684, -0.0101], (0.63, 3.928], ..., (-0.0101, 0.63], (-0.684, -0.0101], (-2.9499999999999997, -0.684], (-0.0101, 0.63], (0.63, 3.928]]\n",
       "Length: 1000\n",
       "Categories (4, interval[float64]): [(-2.9499999999999997, -0.684] < (-0.684, -0.0101] < (-0.0101, 0.63] < (0.63, 3.928]]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bins = pd.qcut(draws, 4)\n",
    "\n",
    "bins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Q2, Q3, Q2, Q2, Q4, ..., Q3, Q2, Q1, Q3, Q4]\n",
       "Length: 1000\n",
       "Categories (4, object): [Q1 < Q2 < Q3 < Q4]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using labels to classify the quartiles\n",
    "bins = pd.qcut(draws, 4, labels=['Q1', 'Q2', 'Q3', 'Q4'])\n",
    "\n",
    "bins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2, 1, 1, 3, 3, 2, 2, 3, 3], dtype=int8)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bins.codes[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The labelled categorical does not contain the any information about the bin edges in the data. We can use 'groupby' to extract the summary statistics.\n",
    "#### The 'quartile' column in the example 'results' retains original categorical information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>quartile</th>\n",
       "      <th>count</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Q1</td>\n",
       "      <td>250</td>\n",
       "      <td>-2.949343</td>\n",
       "      <td>-0.685484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Q2</td>\n",
       "      <td>250</td>\n",
       "      <td>-0.683066</td>\n",
       "      <td>-0.010115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Q3</td>\n",
       "      <td>250</td>\n",
       "      <td>-0.010032</td>\n",
       "      <td>0.628894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Q4</td>\n",
       "      <td>250</td>\n",
       "      <td>0.634238</td>\n",
       "      <td>3.927528</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  quartile  count       min       max\n",
       "0       Q1    250 -2.949343 -0.685484\n",
       "1       Q2    250 -0.683066 -0.010115\n",
       "2       Q3    250 -0.010032  0.628894\n",
       "3       Q4    250  0.634238  3.927528"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bins = pd.Series(bins, name='quartile')\n",
    "\n",
    "results = (pd.Series(draws)\n",
    "          .groupby(bins)\n",
    "          .agg(['count', 'min', 'max'])\n",
    "          .reset_index())\n",
    "\n",
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    Q1\n",
       "1    Q2\n",
       "2    Q3\n",
       "3    Q4\n",
       "Name: quartile, dtype: category\n",
       "Categories (4, object): [Q1 < Q2 < Q3 < Q4]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results['quartile']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Better Performance with Categories\n",
    "#### Converting to categorical data can yield substantial eprformance gains in analytical processes.\n",
    "#### Categorical dsata uses significantly less memory as compared to arrays of strings.\n",
    "#### Obviously the conversion is not free, but is just a one-time cost.\n",
    "#### Different operations (like GroupBy) can be significantly faster with categoricals because underlying algorithms use integer-based codes instead of string arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "N = 10000000\n",
    "draws = pd.Series(np.random.randn(N))\n",
    "labels = pd.Series(['foo', 'bar', 'baz', 'qux'] * (N // 4))\n",
    "\n",
    "categories = labels.astype('category')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "80000080"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels.memory_usage()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000272"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categories.memory_usage()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 368 ms\n"
     ]
    }
   ],
   "source": [
    "%time _ = labels.astype('category')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Categorical Methods\n",
    "#### Series having categorical data has several special methods. It provides convinient access to categories and codes.\n",
    "#### The special attribute 'cat' provides access to categorical methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    a\n",
       "1    b\n",
       "2    c\n",
       "3    d\n",
       "4    a\n",
       "5    b\n",
       "6    c\n",
       "7    d\n",
       "dtype: category\n",
       "Categories (4, object): [a, b, c, d]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series(['a', 'b', 'c', 'd'] * 2)\n",
    "\n",
    "cat_s = s.astype('category')\n",
    "cat_s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    1\n",
       "2    2\n",
       "3    3\n",
       "4    0\n",
       "5    1\n",
       "6    2\n",
       "7    3\n",
       "dtype: int8"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_s.cat.codes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['a', 'b', 'c', 'd'], dtype='object')"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_s.cat.categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We can use 'set_categories' if we know that there are more categories than we currently see in the data and we want to include other categories as well.\n",
    "#### Even if there is no data that corresponds to the extra categories, these categories will be reflected in operations that use them. eg - value_counts.\n",
    "#### For large datasets, categoricals is very convinient for memory savings and better performance.\n",
    "#### After filtering a large dataset, you may not have all the categories having any data. To trim such categories, use 'remove_unused_categories'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    a\n",
       "1    b\n",
       "2    c\n",
       "3    d\n",
       "4    a\n",
       "5    b\n",
       "6    c\n",
       "7    d\n",
       "dtype: category\n",
       "Categories (5, object): [a, b, c, d, e]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "actual_categories = ['a', 'b', 'c', 'd', 'e']\n",
    "\n",
    "cat_s2 = cat_s.cat.set_categories(actual_categories)\n",
    "cat_s2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "d    2\n",
       "c    2\n",
       "b    2\n",
       "a    2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_s.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "d    2\n",
       "c    2\n",
       "b    2\n",
       "a    2\n",
       "e    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_s2.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    a\n",
       "1    b\n",
       "4    a\n",
       "5    b\n",
       "dtype: category\n",
       "Categories (4, object): [a, b, c, d]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_s3 = cat_s[cat_s.isin(['a', 'b'])]\n",
    "\n",
    "cat_s3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    a\n",
       "1    b\n",
       "4    a\n",
       "5    b\n",
       "dtype: category\n",
       "Categories (2, object): [a, b]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_s3.cat.remove_unused_categories()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating Dummy Variables for Modeling\n",
    "#### In statistics and machine learning, to transform categorical data into 'dummy variables' is called 'one-hot encoding'.\n",
    "#### This involves creating DataFrame with a column for each distinct category. The column contains 1s for occurences of a category and 0s otherwise.\n",
    "#### The function 'pandas.get_dummies' converts 1-dimensional categorical data into a DataFrame having dummy variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    a\n",
       "1    b\n",
       "2    c\n",
       "3    d\n",
       "4    a\n",
       "5    b\n",
       "6    c\n",
       "7    d\n",
       "dtype: category\n",
       "Categories (4, object): [a, b, c, d]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_s = pd.Series(['a', 'b', 'c', 'd'] * 2, dtype='category')\n",
    "\n",
    "cat_s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   a  b  c  d\n",
       "0  1  0  0  0\n",
       "1  0  1  0  0\n",
       "2  0  0  1  0\n",
       "3  0  0  0  1\n",
       "4  1  0  0  0\n",
       "5  0  1  0  0\n",
       "6  0  0  1  0\n",
       "7  0  0  0  1"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.get_dummies(cat_s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced GroupBy Use\n",
    "#### There are a few advanced techniques in groupby that can be useful in data analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Group Transforms and \"Unwrapped\" GroupBys\n",
    "#### Just like the 'apply' method for grouped operations, we also have 'transform', which is similar but imposes more constraints on kind of function you can use.\n",
    "#### The restrictions on the function are:\n",
    "####     1. It can produce scalar value to be broadcast to the shape of the group.\n",
    "####     2. It can produce an object of the same shape as input group.\n",
    "####     3. It must not mutate its input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>c</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>a</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>b</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>c</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>a</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>b</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>c</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>a</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>b</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>c</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   key  value\n",
       "0    a    0.0\n",
       "1    b    1.0\n",
       "2    c    2.0\n",
       "3    a    3.0\n",
       "4    b    4.0\n",
       "5    c    5.0\n",
       "6    a    6.0\n",
       "7    b    7.0\n",
       "8    c    8.0\n",
       "9    a    9.0\n",
       "10   b   10.0\n",
       "11   c   11.0"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({'key': ['a', 'b', 'c'] * 4,\n",
    "                  'value': np.arange(12.)})\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's group the above DataFrame by the key using the mean of each key as value.\n",
    "#### Now if we want to produce a Series with same length as the above Dataframe's 'value' column, but with values replaced by the average grouped by key.\n",
    "#### To achieve this we can pass the function 'lambda x: x.mean()' to 'transform'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "key\n",
       "a    4.5\n",
       "b    5.5\n",
       "c    6.5\n",
       "Name: value, dtype: float64"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "g = df.groupby('key').value\n",
    "\n",
    "g.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     4.5\n",
       "1     5.5\n",
       "2     6.5\n",
       "3     4.5\n",
       "4     5.5\n",
       "5     6.5\n",
       "6     4.5\n",
       "7     5.5\n",
       "8     6.5\n",
       "9     4.5\n",
       "10    5.5\n",
       "11    6.5\n",
       "Name: value, dtype: float64"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Replaced value with mean of the key group\n",
    "g.transform(lambda x: x.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For built-in aggregation methods, we can use a string alias instead of lambda. eg - 'mean' for getting mean in the values.\n",
    "#### Transform works with functions that return Series but the result should be of same size as the input.\n",
    "#### We can perform more complicated operations by passing a more complicated lambda function to transform. eg - ranking in descending order for each group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     4.5\n",
       "1     5.5\n",
       "2     6.5\n",
       "3     4.5\n",
       "4     5.5\n",
       "5     6.5\n",
       "6     4.5\n",
       "7     5.5\n",
       "8     6.5\n",
       "9     4.5\n",
       "10    5.5\n",
       "11    6.5\n",
       "Name: value, dtype: float64"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "g.transform('mean')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      0.0\n",
       "1      2.0\n",
       "2      4.0\n",
       "3      6.0\n",
       "4      8.0\n",
       "5     10.0\n",
       "6     12.0\n",
       "7     14.0\n",
       "8     16.0\n",
       "9     18.0\n",
       "10    20.0\n",
       "11    22.0\n",
       "Name: value, dtype: float64"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# As long as result is of same length as input, transform works\n",
    "g.transform(lambda x: x * 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     4.0\n",
       "1     4.0\n",
       "2     4.0\n",
       "3     3.0\n",
       "4     3.0\n",
       "5     3.0\n",
       "6     2.0\n",
       "7     2.0\n",
       "8     2.0\n",
       "9     1.0\n",
       "10    1.0\n",
       "11    1.0\n",
       "Name: value, dtype: float64"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "g.transform(lambda x: x.rank(ascending=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For a group transformation composing simple aggregations, 'transform' and 'apply' lead to equivalent results.\n",
    "#### Built-in aggregation functions (eg - mean, sum, etc) are much faster than a general 'apply' function.\n",
    "#### They also have a 'fast past' when used with 'transform', which allows us to perform 'unwrapped' group operation.\n",
    "#### An unwrapped group operation involves multiple group aggregations, but there is overall benefit of vectorized operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalize(x):\n",
    "    return (x - x.mean()) / x.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    -1.161895\n",
       "1    -1.161895\n",
       "2    -1.161895\n",
       "3    -0.387298\n",
       "4    -0.387298\n",
       "5    -0.387298\n",
       "6     0.387298\n",
       "7     0.387298\n",
       "8     0.387298\n",
       "9     1.161895\n",
       "10    1.161895\n",
       "11    1.161895\n",
       "Name: value, dtype: float64"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Equivalent result for both transform and apply\n",
    "g.transform(normalize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    -1.161895\n",
       "1    -1.161895\n",
       "2    -1.161895\n",
       "3    -0.387298\n",
       "4    -0.387298\n",
       "5    -0.387298\n",
       "6     0.387298\n",
       "7     0.387298\n",
       "8     0.387298\n",
       "9     1.161895\n",
       "10    1.161895\n",
       "11    1.161895\n",
       "Name: value, dtype: float64"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "g.apply(normalize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    -1.161895\n",
       "1    -1.161895\n",
       "2    -1.161895\n",
       "3    -0.387298\n",
       "4    -0.387298\n",
       "5    -0.387298\n",
       "6     0.387298\n",
       "7     0.387298\n",
       "8     0.387298\n",
       "9     1.161895\n",
       "10    1.161895\n",
       "11    1.161895\n",
       "Name: value, dtype: float64"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Unwrapped group aggregations are faster because of inbuilt aggregations\n",
    "\n",
    "normalized = (df['value'] - g.transform('mean')) / g.transform('std')\n",
    "normalized"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grouped Time Resampling\n",
    "#### For time series, 'resample' is semantically a group operation based on time intervalization.\n",
    "#### We usually use the time as index and then resample, grouping based on index (time)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2017-05-20 00:00:00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2017-05-20 00:01:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2017-05-20 00:02:00</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2017-05-20 00:03:00</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017-05-20 00:04:00</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2017-05-20 00:05:00</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2017-05-20 00:06:00</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2017-05-20 00:07:00</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2017-05-20 00:08:00</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2017-05-20 00:09:00</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2017-05-20 00:10:00</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2017-05-20 00:11:00</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2017-05-20 00:12:00</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2017-05-20 00:13:00</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2017-05-20 00:14:00</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  time  value\n",
       "0  2017-05-20 00:00:00      0\n",
       "1  2017-05-20 00:01:00      1\n",
       "2  2017-05-20 00:02:00      2\n",
       "3  2017-05-20 00:03:00      3\n",
       "4  2017-05-20 00:04:00      4\n",
       "5  2017-05-20 00:05:00      5\n",
       "6  2017-05-20 00:06:00      6\n",
       "7  2017-05-20 00:07:00      7\n",
       "8  2017-05-20 00:08:00      8\n",
       "9  2017-05-20 00:09:00      9\n",
       "10 2017-05-20 00:10:00     10\n",
       "11 2017-05-20 00:11:00     11\n",
       "12 2017-05-20 00:12:00     12\n",
       "13 2017-05-20 00:13:00     13\n",
       "14 2017-05-20 00:14:00     14"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "N = 15\n",
    "\n",
    "times = pd.date_range('2017-05-20 00:00', freq='1min', periods=N)\n",
    "df = pd.DataFrame({'time': times,\n",
    "                  'value': np.arange(N)})\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2017-05-20 00:00:00</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-05-20 00:05:00</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-05-20 00:10:00</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     value\n",
       "time                      \n",
       "2017-05-20 00:00:00      5\n",
       "2017-05-20 00:05:00      5\n",
       "2017-05-20 00:10:00      5"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.set_index('time').resample('5min').count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### But if a DataFrame has a time series and a key column, then grouping cannot be done based on time alone.\n",
    "#### To do a resample for each value of the 'key', we use the 'pandas.TimeGrouper' (now 'pandas.Grouper') object.\n",
    "#### We set the time as index, then group by key and time and then aggregate.\n",
    "#### One constraint using TimeGrouper is that the time must be the index of the Series or DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>key</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2017-05-20 00:00:00</td>\n",
       "      <td>a</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2017-05-20 00:00:00</td>\n",
       "      <td>b</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2017-05-20 00:00:00</td>\n",
       "      <td>c</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2017-05-20 00:01:00</td>\n",
       "      <td>a</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017-05-20 00:01:00</td>\n",
       "      <td>b</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2017-05-20 00:01:00</td>\n",
       "      <td>c</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2017-05-20 00:02:00</td>\n",
       "      <td>a</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 time key  value\n",
       "0 2017-05-20 00:00:00   a    0.0\n",
       "1 2017-05-20 00:00:00   b    1.0\n",
       "2 2017-05-20 00:00:00   c    2.0\n",
       "3 2017-05-20 00:01:00   a    3.0\n",
       "4 2017-05-20 00:01:00   b    4.0\n",
       "5 2017-05-20 00:01:00   c    5.0\n",
       "6 2017-05-20 00:02:00   a    6.0"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = pd.DataFrame({'time': times.repeat(3),\n",
    "                   'key': np.tile(['a', 'b', 'c'], N),\n",
    "                   'value': np.arange(N * 3.)})\n",
    "\n",
    "df2[:7]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\adity\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:1: FutureWarning: pd.TimeGrouper is deprecated and will be removed; Please use pd.Grouper(freq=...)\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    }
   ],
   "source": [
    "time_key = pd.TimeGrouper('5min')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>key</th>\n",
       "      <th>time</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">a</th>\n",
       "      <th>2017-05-20 00:00:00</th>\n",
       "      <td>30.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-05-20 00:05:00</th>\n",
       "      <td>105.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-05-20 00:10:00</th>\n",
       "      <td>180.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">b</th>\n",
       "      <th>2017-05-20 00:00:00</th>\n",
       "      <td>35.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-05-20 00:05:00</th>\n",
       "      <td>110.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-05-20 00:10:00</th>\n",
       "      <td>185.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">c</th>\n",
       "      <th>2017-05-20 00:00:00</th>\n",
       "      <td>40.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-05-20 00:05:00</th>\n",
       "      <td>115.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-05-20 00:10:00</th>\n",
       "      <td>190.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         value\n",
       "key time                      \n",
       "a   2017-05-20 00:00:00   30.0\n",
       "    2017-05-20 00:05:00  105.0\n",
       "    2017-05-20 00:10:00  180.0\n",
       "b   2017-05-20 00:00:00   35.0\n",
       "    2017-05-20 00:05:00  110.0\n",
       "    2017-05-20 00:10:00  185.0\n",
       "c   2017-05-20 00:00:00   40.0\n",
       "    2017-05-20 00:05:00  115.0\n",
       "    2017-05-20 00:10:00  190.0"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resampled = (df2.set_index('time')\n",
    "            .groupby(['key', time_key])\n",
    "            .sum())\n",
    "\n",
    "resampled"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Techniques for Method Chaining\n",
    "#### When applying a sequence of transformations to a dataset, you may create many temporary variables which you will never use in your analysis.\n",
    "#### There are ways of avoiding this. The 'DataFrame.assign' method is a 'functional' alternative to column assignments. eg - df[k] = v.\n",
    "#### Rather than modifying object in-place, it returns a new DataFrame with indicated modifications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "N = 15\n",
    "df = pd.DataFrame({'time': times.repeat(3),\n",
    "                   'key': np.tile(['a', 'b', 'c'], N),\n",
    "                   'value': np.arange(N * 3.)})\n",
    "df2 = df[df['value'] < 0]\n",
    "df2['value'] = df2['value'] - df2['value'].mean()\n",
    "result = df2.groupby('key').std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Non-functional way\n",
    "df2 = df.copy()\n",
    "df2['key'] = 'x'\n",
    "\n",
    "# Functional way\n",
    "df2 = df.assign(k='x')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Assigning in-place may execute faster, but 'assign' enables easier method chaining.\n",
    "#### While performing 'method chaining', keep in mind that you may need to refer to temporary objects. To help, 'assign' and many other pandas functions accept function-like arguments called 'callables'.\n",
    "#### The function we need to pass is kept in [], which is then bounded to the object at that stage.\n",
    "#### The use of method chaining and callables allows us to write multiple functions in a single line of code.\n",
    "#### But doing so is a matter of taste. Splitting up expression into multiple steps is more readable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Method chaining with assign\n",
    "result = (df2.assign(col1_demeaned = df2.value - df2.value.mean())\n",
    "         .groupby('key')\n",
    "         .col1_demeaned.std())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using Callable in first example\n",
    "df = (pd.DataFrame({'time': times.repeat(3),\n",
    "                   'key': np.tile(['a', 'b', 'c'], N),\n",
    "                   'value': np.arange(N * 3.)})\n",
    "     [lambda x: x['value'] < 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Writing single line expression with callable and method chaining\n",
    "result = (pd.DataFrame({'time': times.repeat(3),\n",
    "                   'key': np.tile(['a', 'b', 'c'], N),\n",
    "                   'value': np.arange(N * 3.)})\n",
    "     [lambda x: x['value'] < 0]\n",
    "         .assign(col1_demeaned = df2.value - df2.value.mean())\n",
    "          .groupby('key')\n",
    "         .col1_demeaned.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The pipe Method\n",
    "#### You can achieve a lot with pandas built-in functions with method chaining and callables.\n",
    "#### But sometimes you need to use your own functions or those from third-party libraries. This is where 'pipe' method comes in.\n",
    "#### When using functions that accept and return Series and DataFrame from previous operation, you can rewrite in a single line using calls to 'pipe'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\na = f(df, arg1=v1)\\nb = g(a, v2, arg3=v3)\\nc = h(b, arg4=v4)\\n'"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Series of functions using output from previous function as one of the input\n",
    "'''\n",
    "a = f(df, arg1=v1)\n",
    "b = g(a, v2, arg3=v3)\n",
    "c = h(b, arg4=v4)\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nresult = (df.pipe(f, arg1=v1)\\n         .pipe(g, v2, arg3=v3)\\n         .pipe(h, arg4=v4))\\n'"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Same operation using 'pipe'\n",
    "'''\n",
    "result = (df.pipe(f, arg1=v1)\n",
    "         .pipe(g, v2, arg3=v3)\n",
    "         .pipe(h, arg4=v4))\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Although the statements 'f(df)' and 'df.pipe(f)' are equivalent, pipe makes chained invocation easier.\n",
    "#### A potentially useful pattern for pipe is to generalize a sequence of operations into resuable functions. eg - demeaning multiple columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\ng = df.groupby(['key1', 'key2'])\\ndf['col1'] = df['col1'] - g.transform('mean')\\n\""
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A simple demeaning (subtracting mean from a value) operation\n",
    "'''\n",
    "g = df.groupby(['key1', 'key2'])\n",
    "df['col1'] = df['col1'] - g.transform('mean')\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\ndef group_demean(df, by, cols):\\n    result = df.copy()\\n    g = df.groupby(by)\\n    for c in cols:\\n        result[c] = df[c] - g[c].transform('mean')\\n    return result\\n\""
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Demeaning function\n",
    "'''\n",
    "def group_demean(df, by, cols):\n",
    "    result = df.copy()\n",
    "    g = df.groupby(by)\n",
    "    for c in cols:\n",
    "        result[c] = df[c] - g[c].transform('mean')\n",
    "    return result\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\nresult = (df[df.col1 < 0]\\n         .pipe(group_demean, ['key1', 'key2'], ['col1']))\\n\""
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using demean function with pipe\n",
    "'''\n",
    "result = (df[df.col1 < 0]\n",
    "         .pipe(group_demean, ['key1', 'key2'], ['col1']))\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}