Data Engineering

A distributed processing system used for big data workloads

When dealing with big data or when dealing with resource-heavy queries

Data layer on top of an existing data lake that brings both reliability and performance to data lakes

• ACID transactionality • Data versioning • Schema enforcement and evolution • Audit history • Parquet format • Compatible with Spark API • Unifies streaming and batch data processing

A system or repository of data stored in its natural/raw format

A repository for business data. Only highly structured and unified data lives in a data warehouse to support specific business intelligence and analytics needs

A database stores the current data required to power an application. A data lake stores current and historical data for one or more systems in its raw form for the purpose of analyzing the data

Any operation that is treated as a single unit of work, which either completes fully or does not complete at all, and leaves the storage system in a consistent state

• Atomicity • Consistency • Isolation • Durability

Each statement in a transaction (read, write, update, delete) is treated as a single unit. Either the entire statement is executed, or none of it is executed. This property prevents data loss and corruption from occurring

Ensures that transactions only make changes to tables in predefined, predictable ways. Ensures corruption or errors in data do not create unintended consequences for the integrity of the table

When multiple users are reading and writing from the same table all at once, isolation of their transactions ensures that the concurrent transactions don’t interfere with or affect one another

Ensures that changes to your data made by successfully executed transactions will be saved, even in the event of system failure

Online Analytical Processing. A system for performing multidimensional analysis at high speeds on large volumes of data. Typically, this data is from a warehouse.

Online Transactional Processing. It enables the real-time execution of large numbers of database transactions by large numbers of people, such as in ATMs and in reservation systems

A multidimensional array of data

A column-oriented data file format designed for efficient data storage and retrieval. It provides efficient data compression and encoding schemes with enhanced performance to handle complex data in bulk

Hadoop Distributed File System. A distributed file system that provides high-throughput access to application data

It has a master/slave architecture. An HDFS cluster consists of a single NameNode and a number of DataNodes.

A master server that manages the file system namespace and regulates access to files by clients. It executes operations like opening, closing, and renaming files and directories. It also determines the mapping of data blocks to DataNodes

Manages storage attached to the nodes that they run on. They are responsible for serving read and write request from the file system’s clients and also perform data block creation, deletion, and replication upon instruction from the NameNode

Files are divided into blocks and each block is stored on a DataNode. The NameNode distributes replicas of these data blocks across the cluster and instructs the user or application where to locate wanted information

Data warehouse software designed to read, write, and manage large datasets extracted from the HDFS

Hive is a SQL-like query language tool that is used to analyze structured data in the Hadoop ecosystem and Spark is a in-memory data processing engine that can be used for batch/stream processing, ML, and interactive SQL