-
Notifications
You must be signed in to change notification settings - Fork 3
/
introduction.Rmd
120 lines (63 loc) · 7.28 KB
/
introduction.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
```{r, message=FALSE, echo=FALSE, cache=FALSE}
source("./customization/knitr_options.R")
```
# (PART) Introduction {-}
# Statistics {#stat_overview}
## History
The practice of statistics has been around for hundreds of years. The early application of statistics was centered around collecting demographic and economic data for governments. The terms "statistics" was coined in the mid-1700s, which is derived from its relationship to state or government data.
Early methodological ideas introduced in statistics involved procedures such as calculating the mean or median, representing data graphically, and trying to model the distribution of "errors" between observed data and a model.
In the 1800s, probability started to become integrated into statistical thinking, which lead to the era of inferential statistics; it gave rise to deep results on how we design and analyze studies to use finite amounts of data that have been collected through a probabilistic mechanism. During the early to mid 1900s, inferential statistics became a well developed and understood component to statistics. Its ties to probability were deepened, especially with the formal axiomatization and rigorous development of probability.
The latter half of the 1900s resulted in another major leap forward for the field of statistics, with the introduction of modern computing. This had a massive impact on how we collect and analyze data. Statistics became less reliant on mathematical models and more immersed in computational approaches.
In the 2000s we have witnessed yet another major leap forward. Data collection has never been faster, cheaper, or larger than today. We are able to collect and analyze massive amounts of data in most areas of science and industry. This has led to a sea-change in statistics, both in its scope and in its importance. The field of statistics has never been more challenging or impactful than it is today.
## Definition
The modern definition of **Statistics** is the study of how to extract information from data, including how to collect, organize, analyze, and present information in data.
**Applied Statistics** is concerned with the practical considerations and implementations needed to carry out a statistical analysis.
In Chapter \@ref(challenges) below, we make this definition concrete by discussing the various problems that statistics tackles.
## Relationship to Machine Learning
From <https://en.wikipedia.org/wiki/Machine_learning>:
> **Machine learning** explores the study and construction of algorithms that can learn from and make predictions on data. Machine learning is closely related to and often overlaps with computational statistics; a discipline which also focuses in prediction-making through the use of computers.
## Relationship to Data Science
From <https://en.wikipedia.org/wiki/Data_science>:
> **Data Science** is an interdisciplinary field about processes and systems to extract knowledge or insights from data in various forms, either structured or unstructured, which is a continuation of some of the data analysis fields such as statistics, data mining, and predictive analytics.
## Some History On Data Science
### John Tukey {-}
John Tukey pioneered a field called "exploratory data analysis" (EDA)
From [The Future of Data Analysis](https://projecteuclid.org/euclid.aoms/1177704711) (1962) *Annals of Mathematical Statistics* ...
> For a long time I have thought I was a statistician, interested in inferences from the particular to the general. But as I have watched mathematical statistics evolve, I have had cause to wonder and to doubt.
> All in all, I have come to feel that my central interest is in data analysis, which I take to include, among other things: procedures for analyzing data, techniques for interpreting the results of such procedures, ways of planning the gathering of data to make its analysis easier, more precise or more accurate, and all the machinery and results of (mathematical) statistics which apply to analyzing data.
> Data analysis is a larger and more varied field than inference, or incisive procedures, or allocation.
### Jeff Wu {-}
From <https://en.wikipedia.org/wiki/Data_science>:
> In November 1997, C.F. Jeff Wu gave the inaugural lecture entitled "Statistics = Data Science?". In this lecture, he characterized statistical work as a trilogy of data collection, data modeling and analysis, and decision making. In his conclusion, he initiated the modern, non-computer science, usage of the term "data science" and advocated that statistics be renamed data science and statisticians data scientists.
### William Cleveland {-}
From <https://en.wikipedia.org/wiki/Data_science>:
> In 2001, William Cleveland introduced data science as an independent discipline, extending the field of statistics to incorporate "advances in computing with data" in his article [Data Science: An Action Plan for Expanding the Technical Areas of the Field of Statistics](http://onlinelibrary.wiley.com/doi/10.1111/j.1751-5823.2001.tb00477.x/abstract) in *International Statistical Review*
>
> Cleveland establishes six technical areas which he believed to encompass the field of data science:
>
> 1. multidisciplinary investigations
> 2. models and methods for data
> 3. computing with data
> 4. pedagogy
> 5. tool evaluation
> 6. theory
### Industry {-}
Individuals working in industry began to call themselves "data scientists" in the late 2000's, but this was far after statisticians had introduced the field.
# Components of Applied Statistics {#challenges}
Let's first cosnider the central challenges of applied statistics, shown in the following schematic I'll call the "central dogma of applied statistics", shown in Figure \@ref(fig:central-dogma-statistics)
```{r central-dogma-statistics, echo=FALSE, out.width='100%', fig.cap='Central Dogma of Statistics'}
knitr::include_graphics("./images/central_dogma_statistics.png")
```
## Study Design
An applied statistics project is usually preceded by a scientific question that involves the collection and analysis of data. The design of the data should involve careful application of statistical principles to design which data are to be collected and how the data will be measured. The study design should also be driven by the questions that will be answered and the type of applied statistical analysis techniques will be employed.
Study design is an area that is almost solely studied by statisticians and it is one of the core strengths of the field of statistics. We will be considering study design throughout this book.
## Data Wrangling {#data-wrangling-introduction-chapter}
Data wrangling is a new terms that refers to the process of convertng raw data, which is often very messy, into data that can be readily analyzed. The importance of this activity has grown substantially in recent years as data sets have becoem larger and more comoplex. Data wrangling
## Data Analysis
### Exploratory Data Analysis
### Modeling
### Inference
### Prediciton
## Communication
# Data Sets Used in this Book
I will eventually include several data sets that will be used throughout the book. These data sets will be described here. As of now, all of the R examples are completed with a variety of smaller data sets. Threfore, the R code I demonstrate will not change much, but there will be a more consistent use of data sets that I find interesting.