Data Science Essentials

A random variable assigns a numerical value to each possible outcome of a random experiment.

• Velocity • Veracity • Variability • Volume • Value

Machine Learning is a computing technique that has its origins in artificial intelligence (AI) and statistics. Machine Learning solutions include: • Classification - Predicting a Boolean true/false value for an entity with a given set of features. • Regression - Predicting a real numeric value for an entity with a given set of features. • Clustering - Grouping entities with similar features.

• Min • Max • Q1 • Q2 • Q3

http://chrisalbon.com/python/pandas_join_merge_dataframe.html

Now in general, the first step in machine learning is to figure out how to represent your data as a vector.

Summary statistics generally include the mean, the median and quartiles of the data. This gives you a first quick look at the distribution of data values.

Scatter plot matrix methods quickly produce a single overall view of the relationships in a dataset. The scatter plot matrix allows you to examine the relationships between many variables in one view.

• Data selection. • Preprocessing. • Transformation. • Data Mining. • Interpretation and evaluation.

A discrete random variable has a number of outcomes that you could count.

• replace intuition with data driven analytical decisions. • Transform raw data to valuable asset • Increase pace of action

Data Science is the exploration and quantitative analysis of all available structured and unstructured data to develop understanding, extract knowledge, and formulate actionable results.

A continuous variable is a variable that has an infinite number of possible values. In other words, any value is possible for the variable. A continuous variable is the opposite of a discrete variable, which can only take on a certain number of values.

• Linear Regression • Logistic Regression • Decision Tree • SVM • Naive Bayes • KNN • K-Means • Random Forest • Dimensionality Reduction Algorithms • Gradient Boost & Adaboost

Machine learning is a super powerful set of techniques for prediction. Machine learning allows you to make predictions and detect patterns that otherwise would have gone unnoticed. Now, machine learning started as the subfield of artificial intelligence, and its goal is to allow computers to learn by example.

A discrete variable is a variable whose value is obtained by counting. Examples: number of students present number of red marbles in a jar number of heads when flipping three coins students’ grade level A continuous variable is a variable whose value is obtained by measuring. Examples: height of students in class weight of students in class time it takes to get to school distance traveled between classes

Ingested and joined data from multiple sources. • Deleted unnecessary and redundant columns. • Consolidated the number of categories of a categorical feature. • Treated missing values. • Removed duplicate rows. • Generated a calculated column • Located and treated outliers. • Scaled numeric values.

One of the most important aspects of any data science project is to clean, filter, and otherwise transform data to prepare it for use in a model. Common tasks when preparing data include: • Identifying and handling missing or duplicate values. • Identifying and handling outliers and errors • Scaling numeric values to make them easier to compare.

A conditioned histogram is a histogram of a subset of data conditioned on another variable in the dataset. Often the histogram of a numeric variable is conditioned on a categorical variable. It is also possible to condition a histogram on (generally overlapping) ranges of a numeric variable.

The prediction quality should always be judged out of sample. You should make a judgement based on the results of the Test Data Set and not the Training Data Set.