2020

Data Engineering for Data Analytics: A Classification of the Issues, and Case Studies

Nazabal, Alfredo, Williams, Christopher K. I., Colavizza, Giovanni et al.

Understand

Consider the situation where a data analyst wishes to carry out an analysis on a given dataset.

It is widely recognized that most of the analyst's time will be taken up with \emph{data engineering} tasks such as acquiring, understanding, cleaning and preparing the data.
In this paper we provide a description and classification of such tasks into high-levels groups, namely data organization, data quality and feature engineering.
We also make available four datasets and example analyses that exhibit a wide variety of these problems, to help encourage the development of tools and techniques to help reduce this burden and push forward research towards the automation or semi-automation of the data engineering process.

Built on

I. P. Fellegi and A. B. Sunter, “A theory for record linkage,” Journal of the American Statistical Association , vol. 64, no. 328, pp. 1183–1210, 1969
1969
Earlier work this paper cites.
R. J. A. Little and D. B. Rubin, Statistical Analysis with Missing Data . John Wiley & Sons, 1987
1987
Earlier work this paper cites.
M. Basseville, “Detecting changes in signals and systems—a survey,” Automatica , vol. 24, no. 3, pp. 309–326, 1988
1988
Earlier work this paper cites.
E. Rahm and H. H. Do, “Data cleaning: Problems and current approaches,” IEEE Data Eng. Bull. , vol. 23, no. 4, pp. 3–13, 2000
2000
Earlier work this paper cites.
P. Chapman, J. Clinton, R. Kerber, T. Khabaza, T. Reinartz, C. Shearer, and R. Wirth, “CRISP-DM 1.0 Step-by-step data mining guide,” 2000
2000
Earlier work this paper cites.
Y. Chen, X. S. Zhou, and T. S. Huang, “One-class SVM for learning in image retrieval,” in Proc. International Conf. on Image Processing (ICIP) , 2001, pp. 34–37
2001
Earlier work this paper cites.
L. Breiman, “Random forests,” Machine Learning , vol. 45, no. 1, pp. 5–32, 2001
2001
Earlier work this paper cites.
J. L. Schafer and J. W. Graham, “Missing data: our view of the state of the art.” Psychological Methods , vol. 7, no. 2, p. 147, 2002
2002
Earlier work this paper cites.
E. Eskin, A. Arnold, M. Prerau, L. Portnoy, and S. Stolfo, “A geometric framework for unsupervised anomaly detection,” in Applications of Data Mining in Computer Security . Springer, 2002, pp. 77–101
2002
Earlier work this paper cites.
T. Dasu and T. Johnson, Exploratory Data Mining and Data Cleaning . John Wiley & Sons, 2003, vol. 479
2003
Earlier work this paper cites.
W. Kim, B.-J. Choi, E.-K. Hong, S.-K. Kim, and D. Lee, “A taxonomy of dirty data,” Data Mining and Knowledge Discovery , vol. 7, no. 1, pp. 81–99, 2003
2003
Earlier work this paper cites.
H. Zou and T. Hastie, “Regularization and variable selection via the elastic net,” Journal of the Royal Statistical Society: Series B (Statistical Methodology) , vol. 67, no. 2, pp. 301–320, 2005
2005
Earlier work this paper cites.
A. K. Elmagarmid, P. G. Ipeirotis, and V. S. Verykios, “Duplicate record detection: A survey,” IEEE Transactions on Knowledge and Data Engineering , vol. 19, no. 1, pp. 1–16, 2006
2006
Earlier work this paper cites.
R. K. Pearson, “The problem of disguised missing data,” ACM SIGKDD Explorations Newsletter , vol. 8, no. 1, pp. 83–92, 2006
2006
Earlier work this paper cites.
D. Nadeau and S. Sekine, “A survey of named entity recognition and classification,” Lingvisticae Investigationes , vol. 30, no. 1, pp. 3–26, 2007
2007
Earlier work this paper cites.
A. Poggi, D. Lembo, D. Calvanese, G. De Giacomo, M. Lenzerini, and R. Rosati, “Linking data to ontologies,” J. Data Semantics , vol. 10, pp. 133–173, 2008
2008
Earlier work this paper cites.
F. T. Liu, K. M. Ting, and Z. H. Zhou, “Isolation forest,” in 2008 Eighth IEEE International Conference on Data Mining . IEEE, 2008, pp. 413–422
2008
Earlier work this paper cites.
P. Vassiliadis, “A survey of extract–transform–load technology,” International Journal of Data Warehousing and Mining (IJDWM) , vol. 5, no. 3, pp. 1–27, 2009
2009
Earlier work this paper cites.
V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey,” ACM Computing Surveys (CSUR) , vol. 41, no. 3, p. 15, 2009
2009
Earlier work this paper cites.

Similar

J. Quinonero-Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence, Dataset Shift in Machine Learning . The MIT Press, 2009
2009
Cited alongside, same era.
T. Johnson, “Data Profiling,” in Encyclopedia of Database Systems , L. Liu and M. T. Özsu, Eds. Springer, 2009, pp. 604–608
2009
Cited alongside, same era.
P. Christen, “A survey of indexing techniques for scalable record linkage and deduplication,” IEEE Transactions on Knowledge and Data Engineering , vol. 24, no. 9, pp. 1537–1555, 2011
2011
Cited alongside, same era.
S. Gulwani, “Automating string processing in spreadsheets using input-output examples,” ACM SIGPLAN Notices , vol. 46, no. 1, pp. 317–330, 2011
2011
Cited alongside, same era.
M. J. Azur, E. A. Stuart, C. Frangakis, and P. J. Leaf, “Multiple imputation by chained equations: what is it and how does it work?” International Journal of Methods in Psychiatric Research , vol. 20, no. 1, pp. 40–49, 2011
2011
Cited alongside, same era.
S. Kandel, A. Paepcke, J. Hellerstein, and J. Heer, “Wrangler: Interactive visual specification of data transformation scripts,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . ACM, 2011, pp. 3363–3372
2011
Cited alongside, same era.
N. Görnitz, M. Kloft, K. Rieck, and U. Brefeld, “Toward supervised anomaly detection,” Journal of Artificial Intelligence Research , vol. 46, pp. 235–262, 2013
2013
Cited alongside, same era.
F. Serban, J. Vanschoren, J.-U. Kietz, and A. Bernstein, “A survey of intelligent assistants for data analysis,” ACM Comput. Surv. , vol. 45, no. 3, pp. 1–35, Jul. 2013
2013
Cited alongside, same era.
H. Wickham, “Tidy data,” Journal of Statistical Software , vol. 59, no. 10, pp. 1–23, 2014
2014
Cited alongside, same era.
V. Le and S. Gulwani, “FlashExtract: a framework for data extraction by examples,” ACM SIGPLAN Notices , vol. 49, no. 6, pp. 542–553, 2014
2014
Cited alongside, same era.
M. Zhong, N. Goddard, and C. Sutton, “Interleaved factorial non-homogeneous hidden Markov models for energy disaggregation,” arXiv preprint arXiv:1406.7665 , 2014
Original
2014
Cited alongside, same era.
Z. Abedjan, L. Golab, and F. Naumann, “Profiling relational data: a survey,” The VLDB Journal—The International Journal on Very Large Data Bases , vol. 24, no. 4, pp. 557–581, 2015
2015
Cited alongside, same era.

Then

N. D. Lawrence, “Data Readiness Levels,” arXiv preprint arXiv:1705.02245 , 2017
Original
2017
Later among the works it cites.
A. D. Bjorkman, I. Myers-Smith et al. , “Plant functional trait change across a warming tundra biome,” Nature , vol. 562, pp. 57–62, 2018
2018
Later among the works it cites.
C. A. Sutton, T. Hobson, J. Geddes, and R. Caruana, “Data Diff: Interpretable, Executable Summaries of Changes in Distributions for Data Wrangling,” in 24th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2018) . ACM, 2018, pp. 2279–2288
2018
Later among the works it cites.
P. Cerda, G. Varoquaux, and B. Kégl, “Similarity encoding for learning with dirty categorical variables,” Machine Learning Journal , vol. 107, pp. 1477–1494, 2018
2018
Later among the works it cites.
J. Yoon, J. Jordon, and M. Schaar, “GAIN: Missing data imputation using generative adversarial nets,” in International Conference on Machine Learning , 2018, pp. 5675–5684
2018
Later among the works it cites.
A. Nazabal, P. M. Olmos, Z. Ghahramani, and I. Valera, “Handling incomplete heterogeneous data using VAEs,” arXiv preprint arXiv:1807.03653 , 2018
Original
2018
Later among the works it cites.
K. Lee, H. Lee, K. Lee, and J. Shin, “Training confidence-calibrated classifiers for detecting out-of-distribution samples,” International Conference on Learning Representations (ICLR) , 2018
2018
Later among the works it cites.
“Trifacta,” 2018, accessed on 27/06/2018. [Online]. Available: https://www.trifacta.com/
2018
Later among the works it cites.
E. Mansour, D. Deng et al. , “Building Data Civilizer Pipelines with an Advanced Workflow Engine,” in Proc. 34th IEEE International Conference on Data Engineering (ICDE) . IEEE, 2018, pp. 1593–1596
2018
Later among the works it cites.
S. Harris, S. Shi, D. Brealey, N. S. MacCallum, S. Denaxas, D. Perez-Suarez, A. Ercole, P. Watkinson, A. Jones, S. Ashworth et al. , “Critical Care Health Informatics Collaborative (CCHIC): Data, tools and methods for reproducible research: A multi-centre UK intensive care database,” International Journal of Medical Informatics , vol. 112, pp. 82–89, 2018
2018
Later among the works it cites.
G. J. J. Van Den Burg, A. Nazabal, and C. Sutton, “Wrangling messy csv files by detecting row and type patterns,” Data Mining and Knowledge Discovery , vol. 33, no. 6, pp. 1799–1820, 2019
2019
Later among the works it cites.
J. Chen, E. Jimenez-Ruiz, I. Horrocks, and C. A. Sutton, “ColNet: Embedding the Semantics of Web Tables for Column Type Prediction,” in Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19) . AAAI, 2019
2019
Later among the works it cites.
T. Ceritli, C. K. I. Williams, and J. Geddes, “ptype: Probabilistic type inference,” arXiv preprint arXiv:1911.10081 , 2019
Original
2019
Later among the works it cites.
S. Eduardo, A. Nazábal, C. K. I. Williams, and C. Sutton, “Robust Variational Autoencoders for Outlier Detection in Mixed-Type Data,” arXiv preprint arXiv:1907.06671 , 2019
Original
2019
Later among the works it cites.
M. P. J. Camilleri and C. K. I. Williams, “The Extended Dawid-Skene Model: Fusing Information from Multiple Data Schemas,” arXiv preprint arXiv:1906.01251 , 2019
Original
2019
Later among the works it cites.
L. A. Castelijns, Y. Maas, and J. Vanschoren, “The ABC of Data: A Classifying Framework for Data Readiness,” ECMLPKDD workshop on automating data science (ADS) , 2019
2019
Later among the works it cites.
“OpenRefine,” 2019, accessed on 04/01/2020. [Online]. Available: https://openrefine.org/
2020
Closest in time.
G. J. van den Burg and C. K. Williams, “An evaluation of change point detection algorithms,” arXiv preprint arXiv:2003.06222 , 2020
Original
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…