Ilmu Informatika    
   
Daftar Isi ☛
(Sebelumnya) ExplorationExploreZip (Berikutnya)

Exploratory data analysis

In statistics, exploratory data analysis (EDA) is an approach to analyzing data sets to summarize their main characteristics in easy-to-understand form, often with visual graphs, without using a statistical model or having formulated a hypothesis. Exploratory data analysis was promoted by John Tukey to encourage statisticians visually to examine their data sets, to formulate hypotheses that could be tested on new data-sets.

Tukey's championing of EDA encouraged the development of statistical computing packages, especially S at Bell Labs. The S programming language inspired the systems 'S'-PLUS and R. This family of statistical-computing environments featured vastly improved dynamic visualization capabilities, which allowed statisticians to identify outliers, trends and patterns in data that merited further study.

Tukey's EDA was related to two other developments in statistical theory: Robust statistics and nonparametric statistics, both of which tried to reduce the sensitivity of statistical inferences to errors in formulating statistical models. Tukey promoted the use of five number summary of numerical data—the two extremes (maximum and minimum), the median, and the quartiles—because these median and quartiles, being functions of the empirical distribution are defined for all distributions, unlike the mean and standard deviation; moreover, the quartiles and median are more robust to skewed or heavy-tailed distributions than traditional summaries (the mean and standard deviation). The packages S, S-PLUS, and R included routines using resampling statistics, such as Quenouille and Tukey's jackknife and Efron's bootstrap, which are nonparametric and robust (for many problems).

Exploratory data analysis, robust statistics, nonparametric statistics, and the development of statistical programming languages facilitated statisticians' work on scientific and engineering problems. Such problems included the fabrication of semiconductors and the understanding of communications networks, which concerned Bell Labs. These statistical developments, all championed by Tukey, were designed to complement the analytic theory of testing statistical hypotheses, particularly the Laplacian tradition's emphasis on exponential families.[1]

Contents

EDA development

Tukey held that too much emphasis in statistics was placed on statistical hypothesis testing (confirmatory data analysis); more emphasis needed to be placed on using data to suggest hypotheses to test. In particular, he held that confusing the two types of analyses and employing them on the same set of data can lead to systematic bias owing to the issues inherent in testing hypotheses suggested by the data.

The objectives of EDA are to:

  • Suggest hypotheses about the causes of observed phenomena
  • Assess assumptions on which statistical inference will be based
  • Support the selection of appropriate statistical tools and techniques
  • Provide a basis for further data collection through surveys or experiments

Many EDA techniques have been adopted into data mining and are being taught to young students as a way to introduce them to statistical thinking.[2]

Techniques

There are a number of tools that are useful for EDA, but EDA is characterized more by the attitude taken than by particular techniques.[3]

Typical graphical techniques used in EDA are:

Typical quantitative techniques are:

  • Median polish
  • the Trimean
  • Ordination

History

Many EDA ideas can be traced back to earlier authors, for example:

  • Francis Galton emphasized order statistics and quantiles.
  • Arthur Lyon Bowley used precursors of the stemplot and five-number summary (Bowley actually used a "seven-figure summary", including the extremes, deciles and quartiles, along with the median - see his Elementary Manual of Statistics (3rd edn., 1920), p. 62 – he defines "the maximum and minimum, median, quartiles and two deciles" as the "seven positions").
  • Andrew Ehrenberg articulated a philosophy of data reduction (see his book of the same name).

The Open University course Statistics in Society (MDST 242), took the above ideas and merged them with Gottfried Noether's work, which introduced statistical inference via coin-tossing and the median test.

Software

  • R is an open source programming language and software environment for statistical computing and graphics
  • GGobi is a free software for interactive data visualization
  • OpenSHAPA (modern open source successor to MacSHAPA), permits analysis of various media files (e.g. video, sound).
  • CMU-DAP (Carnegie-Mellon University Data Analysis Package, FORTRAN source for EDA tools with English-style command syntax, 1977).
  • Data Applied, a comprehensive web-based data visualization and data mining environment.
  • Fathom (for high-school and intro college courses).
  • JMP, an EDA package from SAS Institute.
  • KNIME Konstanz Information Miner – Open-Source data exploration platform based on Eclipse.
  • Orange, an open-source data mining software suite.
  • PanXpan, a platform on online data analysis modules.
  • SAS Visual Analytics, also from the SAS Institute, includes a web-based EDA application called SAS Visual Analytics Explorer (VAE).
  • SOCR provides a large number of free Internet-accessible.
  • TinkerPlots (for upper elementary and middle school students).
  • VisuMap for interactive exploration of high dimensional multivariate data.
  • Weka an open source data mining package that includes visualisation and EDA tools such as targeted projection pursuit

See also

References

  1. ^ "Conversation with John W. Tukey and Elizabeth Tukey, Luisa T. Fernholz and Stephan Morgenthaler, Statistical Science, Volume 15, Number 1 (2000), 79–94.
  2. ^ Konold, C. (1999). Statistics goes to school. Contemporary Psychology, 44(1), 81–82.
  3. ^ "We need both exploratory and confirmatory" John W. Tukey The American Statistician, 34(1), (Feb., 1980), pp. 23–25.

Bibliography

  • Andrienko, N & Andrienko, G (2005) Exploratory Analysis of Spatial and Temporal Data. A Systematic Approach. Springer. ISBN 3-540-25994-5
  • Hoaglin, D C; Mosteller, F & Tukey, John Wilder (Eds) (1985). Exploring Data Tables, Trends and Shapes. ISBN 0-471-09776-4. 
  • Hoaglin, D C; Mosteller, F & Tukey, John Wilder (Eds) (1983). Understanding Robust and Exploratory Data Analysis. ISBN 0-471-09777-2. 
  • Leinhardt, G., Leinhardt, S., Exploratory Data Analysis: New Tools for the Analysis of Empirical Data, Review of Research in Education, Vol. 8, 1980 (1980), pp. 85–157.
  • Martinez, W. L., Martinez, A. R., and Solka, J. (2010). Exploratory Data Analysis with MATLAB, second edition. Chapman & Hall/CRC. ISBN 9781439812204. 
  • Theus, M., Urbanek, S. (2008), Interactive Graphics for Data Analysis: Principles and Examples, CRC Press, Boca Raton, FL, ISBN 978-1-58488-594-8
  • Tucker, L; MacCallum, R. (1993). Exploratory Factor Analysis. 
  • Tukey, John Wilder (1977). Exploratory Data Analysis. Addison-Wesley. ISBN 0-201-07616-0. 
  • Velleman, P F & Hoaglin, D C (1981) Applications, Basics and Computing of Exploratory Data Analysis ISBN 0-87150-409-X
  • Young, F. W. Valero-Mora, P. and Friendly M. (2006) Visual Statistics: Seeing your data with Dynamic Interactive Graphics. Wiley ISBN 978-0-471-68160-1

External links

(Sebelumnya) ExplorationExploreZip (Berikutnya)