Lecture 4 – Data quality

Volume Velocity Variety Veracity

The speed at which data is generated - sensors generate data every seconds - interactions on a website create data every second high velocity –> analysis of streaming data

Scale of data Datasets in Terrabytes or Petabytes –> too big to process by a single processing computer –> new data storage and processing technology

Uncertainty of data Quality of the data (high veracity = valuable to analyze and contributes in a meaningful way, low veracity = not valuable, inaccurate, not contributing)

Different forms of data (sources, formats, structured/unstructured)

Moore’s: Velocity & Variety Koomey’s: Variety Bell’s: Variety & Veracity Zimmerman’s: all of them

Data sources Data volume Data velocity Data variety Data veracity Software Analytics Processing Capabilities Security/Privacy Lifecycle Other

Cleaning the data so it can be used

Volume: with a lot of data, irregularities creep in Velocity: data can be our-of-date quickly Variety: data can be of different formats and types Veracity: the accuracy of consistency of data from different sources or sets

NA = not available, missing data point NaN = not a number, undefined or unrepresentable value (e.g. we divided number by 0)

• omit them from the data (drop whole column, or drop observations) • give them a value (impute)

see how missing values relate to other variables in the table

Simple parametric: use mean/median Simple non-parametric: find the k nearest neighbors and average these multiple imputation: use a statistical distribution and simulate for the missing values