Hadoop Distributed File System

Hadoop is an open-source processing framework for distributed storage systems specific to managing data processing and data storage for big data applications. Part of the Hadoop ecosystem of technologies is its HDFS. The Hadoop Distributed File System (HDFS) is a scalable data storage system designed to provide high-performance data access within the Hadoop ecosystem

HDFS is specifically designed to scale data storage across multiple physical devices organized into clusters. At the heart of HDFS is MapReduce, a tool that ingests data, breaks it down into separate blocks, and then distributes them accordingly to different nodes in various clusters. HDFS uses a primary NameNode, which tracks where those blocks are placed.

The nodes where these blocks are stored, called DataNodes, are built of inexpensive commodity hardware organized into a cluster. Each cluster usually is a single node. As well, blocks are also replicated across multiple nodes to improve redundancy and resilience using parallel processing.

The conjoined inner workings of the NameNode and DataNodes enables dynamic adaptation in server capacity. In real time, Hadoop can grow or shrink the number of nodes based on demand. This also creates a “self-healing” property within the system—as the NameNode discovers DataNode faults, it can assign tasks to other DataNodes with duplicate blocks.

As with new technology, issues arise when similar terminology is used with seemingly similar technologies. HDFS vs. Cloud Object Storage is one of those cases, but HDFS and Cloud Object Storage are not necessarily comparable.

HDFS is a distributed file system for storing large data, within a file hierarchy. Object Storage, like Amazon’s S3, is a strategy for storing data as objects, and managing those objects with referent metadata, typically to store voluminous unstructured data. While they both deal with storing data in different storage architectures, the functionality of Hadoop makes it far more complex.

Another significant difference is that HDFS follows the premise of moving compute power to where data is, such as the DataNode, where processing can occur in order to reduce network congestion instead of sending data away to be computed. Computation is strictly not included in object storage, which only deals with storing data as objects.

Hadoop applications are those designed to work within the Hadoop ecosystem, using the various services to solve problems using big data. Within the Hadoop ecosystem are four main components:

• Hadoop Common Library — Hadoop's common libraries and utilities that build the foundation for all other components.
• Hadoop Distributed File System — Hadoop’s java-based file distribution system designed to span multiple devices and easily and rapidly scale.
• Yet Another Resource Negotiator (YARN) — Hadoop’s main resource management component. Strictly this is the management component of MapReduce.
• MapReduce — MapReduce is the main processing module for big data sets using parallel, distributed algorithms on clusters. MapReduce is a two stage process, Map filters and sorts data, while Reduce summarizes that data.

To support these main components, several other components have also been included in the Hadoop ecosystem.

• Ambari — Provides functions for managing, configuring, and testing Hadoop services via a web interface.
• Flume — Provides collection, aggregation, and migration to HDFS features for large volumes of streaming data.
• HBase — A NoSQL database that runs on top of Hadoop, helpful for storing sparse data sets during operations.
• Hive — A database style programming language, like SQL, useful for presenting data in tables.
• Oozie — A Hadoop job scheduler.
• Pig — An alternative scripting language to MapReduce for HDFS, which can be compiled into MapReduce programs. MapReduce programs are written in Java.
• Solr — A powerful, fast, and scalable search tool for Hadoop.
• Spark — An open-source cluster computing framework with in-memory analytics that allows for faster processing.
• Sqoop — A migration tool that can translate relational database data into HDFS, and vice versa.
• Zookeeper — A tool for coordinating and synchronizing nodes and distributed processing.