Chapter 3- Data Exploration

Tabular reports that describe the characteristics of each feature in an ABT using standard measures of central tendency and variation

• Histogram for continuous features we can apply Quantitative scales to • Bar plot for each categorical feature we can apply Qualitative scales to

The consist of a set of rectangles/ bars that reflect the counts of frequencies of the classes present in the given data

• Examine the mode, 2nd mode, mode % and 2nd mode % • These tell us the most common levels within the features and will identify if any levels dominate the dataset

• Examine the mean and standard deviation • they will help us get a sense of the central tendency and variation of the values within the dataset for the feature • Examine the minimum and maximum (and other quartiles) • They will help us understand the range that is possible for each feature

• Uniform • Normal (unimodal) • Unimodal (skewed right) • Unimodal (skewed left) • Exponential • Multimodal

• It indicated that a feature is equally likely to take a value in any of the ranges present • Sometimes it shows that a descriptive feature contains an ID rather than a measure of something more interesting

• They have a strong tendency toward a central value and symmetrical variation on either side of the central tendency • It is called unimodal because of the single peak around the central tendency • Naturally occurring phenomena follow normal distribution

Skew right/ positive skew

Skew left/ Negative skew

• Right skewed- Mode < Median • Left skewed- Mode > Median

• The likelihood of certain values occurring is very high but diminishes rapidly for higher or lower values • It is a clear warning sign that outliers are likely

• It has two or more very commonly occurring ranges of values that are clearly separated • Bi-modal distribution can be thought of as two normal distributions pushed together • Occurs when a feature contains a measurement made across a number of distinct groups

• Caution because measures of central tendency and variation tend to break down for multimodal data • Optimism because if we are lucky, the separate peaks in the distribution will be associated with the different target levels we are trying to predict

The distribution moves side to side

Distribution moves up and down

• 68% of observations will be between one standard deviation of mean (mean - sd) • 95% of observations will be within two standard deviations of mean (mean - 2sd) • 99.7% of observations will be within three standard deviations of mean (mean - 3sd)

It is loosely defined as anything unusual about the data in an ABT

• Missing values • Irregular cardinality • Outliers

• Issues due to invalid data (syntax error) • Issues due to valid data (human error)

• Approach 1: Drop any features that have missing values • Approach 2: Apply complete case analysis (delete records) • Approach 3: Derive a missing indicator feature from features with missing value • Approach 4: Impute the missing values

• It can result in massive, and frequently needless loss of data • Only features with 60% excess missing values should be considered for removal • An alternative is to derive a missing indicator feature for them, could be categorical

• We delete instances that are missing one or more feature values • This results in significant amounts of data loss and can introduce a bias to the dataset • This should rarely be used and only when a data instance is missing values for multiple features • It is recommended to remove instances that are missing the value in the target feature

• This could be a categorical feature that flags the missing data as a new label (unknown in marital status) • Or a binary feature that flags whether the value was present or missing (T/F) • When missing indicator features are used the original feature is usually discarded