Survey on Efficient Resource Utilization using Hadoop Cluster for Big Data Processing

Abstract: Hadoop is a framework which is used to store and process large amount of data. Hadoop cluster is designed to analyze and store a huge amount of data. As day by day amount of data stored and processed is increasing rapidly, so we need such an optimal scheduling algorithm to meet the requirement. Job Scheduling is an important parameter to achieve high performance in Hadoop cluster. Hadoop scheduler is pluggable module used for resource allocation. Majorly used schedulers are FIFO, Fair and Capacity scheduler. In this paper we compare and classify parameter such as Average Response Time, Average waiting time and Fairness of various scheduling algorithm in different environment.

Keywords: Hadoop, MapReduce, Scheduling.