jupyter/2018-12-17-comparison-of-pivot-building-methods.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparison of pivot building methods\n",
    "\n",
    "Methods include the following:\n",
    "\n",
    " * `make_pivots` used in the original diffusion frequencies module; this function uses `np.linspace`  which requires the number of data points to create ahead of time instead of spacing between data points. Notably, `linspace` includes the last value in the range by default. Does not allow specification of start and end dates to override the given date observations.\n",
    " * `get_pivots` used in the new modular augur interface (but not provided in the frequency estimators module); this function uses `np.arange` which requires the step size between points defined as a floating point value. `arange` does not include the final value in the range unless there is a floating point overflow. When the step size is not an integer, [this function is known to produce inconsistent results due to floating point overflows](https://docs.scipy.org/doc/numpy/reference/generated/numpy.arange.html#numpy.arange). For example, the final value in the output range can be greater than the given stop value. `linspace` is recommended instead. Allows either a start or an end date to override the given date observations.\n",
    " * `calculate_pivots` used in the KDE frequencies class (but only available as a class method); this function uses `pd.date_range` which requires either the frequency of points in the requested range or the number of points, combining the functionality of `arange` and `linspace` into one date-aware function. `date_range` also includes the final value by default and supports closed intervals on the left or right. The `calculate_pivots` method requires both a start and an end date to override the given date observations\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "np.random.seed(1)\n",
    "\n",
    "def float_to_datestring(time):\n",
    "    \"\"\"Convert a floating point date to a date string\n",
    "\n",
    "    >>> float_to_datestring(2010.75)\n",
    "    '2010-10-01'\n",
    "    >>> float_to_datestring(2011.25)\n",
    "    '2011-04-01'\n",
    "    >>> float\n",
    "    \"\"\"\n",
    "    year = int(time)\n",
    "    month = int(np.rint(((time - year) * 12) + 1))\n",
    "    day = 1\n",
    "    return \"%s-%02d-%02d\" % (year, month, day)\n",
    "\n",
    "def timestamp_to_float(time):\n",
    "    \"\"\"Convert a pandas timestamp to a floating point date.\n",
    "\n",
    "    >>> import datetime\n",
    "    >>> time = datetime.date(2010, 10, 1)\n",
    "    >>> timestamp_to_float(time)\n",
    "    2010.75\n",
    "    >>> time = datetime.date(2011, 4, 1)\n",
    "    >>> timestamp_to_float(time)\n",
    "    2011.25\n",
    "    \"\"\"\n",
    "    return time.year + ((time.month - 1) / 12.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate some example date observations in floating point space like augur would provide."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2010.21428914,  2010.32525726,  2010.3449102 ,  2010.43651063,\n",
       "        2010.67904424,  2011.01164996,  2011.36769014,  2011.40976011,\n",
       "        2011.44511106,  2011.44933273])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_samples = 15\n",
    "observations = np.array(sorted(np.random.choice([2010.0, 2011.0], size=n_samples) +\n",
    "                               np.random.random(n_samples) * 0.75))\n",
    "observations[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare calculation of monthly pivots with different methods described above.\n",
    "Start with `linspace` approach from `make_pivots`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2010.19938122,  2010.27541159,  2010.35144195,  2010.42747232,\n",
       "        2010.50350268,  2010.57953305,  2010.65556341,  2010.73159378,\n",
       "        2010.80762415,  2010.88365451,  2010.95968488,  2011.03571524,\n",
       "        2011.11174561,  2011.18777597,  2011.26380634,  2011.3398367 ,\n",
       "        2011.41586707,  2011.49189743,  2011.5679278 ,  2011.64395816,\n",
       "        2011.71998853])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Determine the range of the given observations.\n",
    "dt = np.max(observations) - np.min(observations)\n",
    "\n",
    "# Define number of pivots to match the arange and date_range examples below.\n",
    "n_pivots = 21\n",
    "\n",
    "# Calculate pivots with linear spacing between start and end with 1% of the total range\n",
    "# added to each side of the range.\n",
    "np.linspace(np.min(observations) - 0.01 * dt, np.max(observations) + 0.01 * dt, n_pivots)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, use `arange` approach from `get_pivots` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2010.16666667,  2010.25      ,  2010.33333333,  2010.41666667,\n",
       "        2010.5       ,  2010.58333333,  2010.66666667,  2010.75      ,\n",
       "        2010.83333333,  2010.91666667,  2011.        ,  2011.08333333,\n",
       "        2011.16666667,  2011.25      ,  2011.33333333,  2011.41666667,\n",
       "        2011.5       ,  2011.58333333,  2011.66666667,  2011.75      ])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min_date = None\n",
    "max_date = None\n",
    "pivots_per_year = 12\n",
    "dt = 1.0 / pivots_per_year\n",
    "\n",
    "first_pivot = min_date if min_date else np.floor(np.min(observations) / dt) * dt\n",
    "last_pivot = max_date if max_date else np.ceil(np.max(observations) / dt) * dt\n",
    "np.arange(first_pivot, last_pivot, dt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08333333333333333"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Floating point step size leads to overflow in arange call above.\n",
    "dt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2010.1666666666665"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_pivot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2011.75"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_pivot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on which observations are included above, the last pivot may or may not be included in the output from `arange` because the step size of `1 / 12.0` leads to a buffer flow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2010.  ,  2010.25,  2010.5 ,  2010.75,  2011.  ,  2011.25,  2011.5 ])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min_date = None\n",
    "max_date = None\n",
    "pivots_per_year = 4\n",
    "dt = 1.0 / pivots_per_year\n",
    "\n",
    "first_pivot = min_date if min_date else np.floor(np.min(observations) / dt) * dt\n",
    "last_pivot = max_date if max_date else np.ceil(np.max(observations) / dt) * dt\n",
    "np.arange(first_pivot, last_pivot, dt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2011.75"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_pivot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the last pivot value is not included in the output from `arange` because the step size of `1 / 4.0` does not lead to a buffer overflow.\n",
    "Next, try to replicate the above examples with the pandas `date_range` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatetimeIndex(['2010-03-01', '2010-04-01', '2010-05-01', '2010-06-01',\n",
       "               '2010-07-01', '2010-08-01', '2010-09-01', '2010-10-01',\n",
       "               '2010-11-01', '2010-12-01', '2011-01-01', '2011-02-01',\n",
       "               '2011-03-01', '2011-04-01', '2011-05-01', '2011-06-01',\n",
       "               '2011-07-01', '2011-08-01', '2011-09-01', '2011-10-01'],\n",
       "              dtype='datetime64[ns]', freq='MS')"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use same step size as first arange example above of 1 month between pivots.\n",
    "pivot_frequency = 1\n",
    "dt = 1.0 / 12\n",
    "\n",
    "first_pivot = float_to_datestring(min_date if min_date else np.floor(np.min(observations) / dt) * dt)\n",
    "last_pivot = float_to_datestring(max_date if max_date else np.ceil(np.max(observations) / dt) * dt)\n",
    "pd.date_range(first_pivot, last_pivot, freq=\"%sMS\" % pivot_frequency)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2010-03-01'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_pivot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2011-10-01'"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_pivot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2010.16666667,  2010.25      ,  2010.33333333,  2010.41666667,\n",
       "        2010.5       ,  2010.58333333,  2010.66666667,  2010.75      ,\n",
       "        2010.83333333,  2010.91666667,  2011.        ,  2011.08333333,\n",
       "        2011.16666667,  2011.25      ,  2011.33333333,  2011.41666667,\n",
       "        2011.5       ,  2011.58333333,  2011.66666667,  2011.75      ])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert datetime values to floats\n",
    "np.array([\n",
    "    timestamp_to_float(timestamp)\n",
    "    for timestamp in pd.date_range(first_pivot, last_pivot, freq=\"%sMS\" % pivot_frequency)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatetimeIndex(['2010-01-01', '2010-04-01', '2010-07-01', '2010-10-01',\n",
       "               '2011-01-01', '2011-04-01', '2011-07-01', '2011-10-01'],\n",
       "              dtype='datetime64[ns]', freq='3MS')"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use same step size as second arange example above of 3 months between pivots (4 pivots per year).\n",
    "pivot_frequency = 3\n",
    "dt = 1.0 / 4\n",
    "\n",
    "first_pivot = float_to_datestring(min_date if min_date else np.floor(np.min(observations) / dt) * dt)\n",
    "last_pivot = float_to_datestring(max_date if max_date else np.ceil(np.max(observations) / dt) * dt)\n",
    "pd.date_range(first_pivot, last_pivot, freq=\"%sMS\" % pivot_frequency)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2010.  ,  2010.25,  2010.5 ,  2010.75,  2011.  ,  2011.25,\n",
       "        2011.5 ,  2011.75])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert datetime values to floats\n",
    "np.array([\n",
    "    timestamp_to_float(timestamp)\n",
    "    for timestamp in pd.date_range(first_pivot, last_pivot, freq=\"%sMS\" % pivot_frequency)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the pivots from `date_range` always include the last pivot value regardless of the step size between pivots. Values in the given range are [calculated by applying the \"frequency\" as an integer offset from the start value up to the end date](https://github.com/pandas-dev/pandas/blob/v0.23.4/pandas/tseries/offsets.py#L2380-L2452)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:janus_python2]",
   "language": "python",
   "name": "conda-env-janus_python2-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}