Exploring Yarn
YARN..well the name is quite trivial. It’s “Yet Another Resource Manager”.
It’s like the operating system of the Hadoop ecosystem. It takes care of cluster resource provisioning, scheduling job and node maintenance.
Following are the different components of YARN-
- Resource Manager
The job of “Resource Manager” is to negotiate resources with nodes so that the jobs to be executed in Hadoop cluster can be executed in a fault tolerant manner. It also takes responsibility of scheduling jobs in the Hadoop cluster.In a nutshell, its responsible for the entire maintenance of Hadoop cluster.
Tasks/Applications are submitted to YARN which allocates the tasks to worker nodes based on priority. All the nodes register themselves to Resource Manager and keep sending heartbeats to signal their availability.
2. Application Master
It manages task execution on nodes and negotiates resources with Resource Manager. It also sends heartbeats to Resource Manager. Whenever an application is submitted to the cluster , the Application Master manages the lifecycle of the application. AM keeps track of the container’s task execution status and negotiates resources with Resource Manager.
3. Node Manager
A Node Manager exists for every node. Its responsible for launching new containers as requested by Application Master. NodeManager oversees the execution of containers and terminates them when job is done or if it is utilising more resources than assigned to it. It keeps informing ResourceManager about health of the node.
