Big Data Technologies

3 V’s: Volume, Variety, Velocity Data that is not atomic (not at lowest level). Data that needs horizontal scaling to handle it - Scale Out not Scale Up

CAP (Brewers) theorem states: It’s impossible for a web service to provide all 3 of these at same time, can only have two: Consistency Availability Partition Tolerance (failing to achieve consistency within set time causes partition). Either network pipe broken or delay in response

Structured data is data such as that held in relational DBs, with ACID (Atomicity, Consistency, Isolation, Durability) features. ‘Unstructured’ data is messy data, variety of formats. Data is only really unstructured in counterpoint to structured relational data. Can adhere to BASE (Basic availablility - allow temp inconsistency to allow info to be available; soft-state - some data may not persist; Eventual consistency)

Means Not Only SQL, as well as No SQL. From 2010 on, idea that relational DBs could handle Big Data

Key-Value: Key Value Pair e.g. Amazon Dynamo Column Family: Extended Key-Value e.g. Cassandra Document (store JSON) e.g. MongoDB Graph DB e.g. Neo4J

NewSQL systems offer the best of both worlds: the relational data model and ACID transactional consistency of traditional operational databases; the familiarity and interactivity of SQL; and the scalability and speed of NoSQL e.g. VoltDB

With large amounts of processing required, instead of getting bigger servers (scale up), use parallel approach of lots of servers (scale out). Instead of adding bigger machines, add more machines.

Solved Problem 1: With lots of servers there are problems with the connections btw nodes. MR ensure nodes have their own disk, memory, CPU - no issues with message passing btw nodes. Solved Problem 2. Issues with disk access and speed and access. MR spreads the processing out, much faster and resistant to node loss.

1st: Processing framework - MapReduce. Splits a task across processors near the data, then assembles results. 2nd. Storage system. HDFS 3rd. YARN. Scheduler that allows other frameworks like Spark to be available - don;t need MR. Batch processing approach.

3 Commercial vendors. MapR has gone bust. Cloudera and Hortonworks have merged Hadoop deemed slow and now maybe legacy, but still in use.

Pig is a high-level query language on top of MapReduce. Code converted to MR job when run - replacement for ETL stuff. if given a file and told to look at it, Pig is good for exploration.

Designed by FB to abstract from messy MR/Java code. Hive language (HQL Hive Query lang) converted to MR job. Allows a schema to be defined on data in HDFS. Not designed for real-time query response, designed for Big Data queries. Pig used to clean data and prep for Hive.

Spark is a cluster computing framework, compatible with Hadoop (runs via YARN), to process iterative jobs and interactive analytics. Doesn’t use MR. Spark all done in memory => typically faster than running MR jobs. Written in Scala language

Airflow is a Workflow Management Platform developed by AirBnb and now open sourced to Apache. Sort of like a powerful ETL/scheduler, uses DAGs as one of it’s constructs.

A Platform that unifies data science and engineering across the Machine Learning lifecycle from data prep, to experimentation and deployment of ML applications. Created by founders of Apache Spark