Cloud Based Adaptive Overlapped Data Chained Declustering

Abstract: Distributed file systems (DFS) are key building blocks for cloud computing applications based on the Map Reduce programming paradigm. In such file systems, nodes simultaneously serve computing and storage functions; a file is partitioned into a number of chunks allocated in distinct nodes so that Map Reduce tasks can be performed in parallel over the nodes. However, in a cloud computing environment, failure is the norm, and nodes may be upgraded, replaced, and added in the system. Files can also be dynamically created, deleted, and appended. This results in load imbalance; that is, the file chunks are not distributed as uniformly as possible in the nodes. Although distributed load balancing algorithms exist in the literature to deal with the load imbalance problem, emerging DFS in production systems strongly depend on a central node for chunk reallocation. The performance of the proposal implemented in the Hadoop distributed file system is further investigated in a cluster environment.

Keywords: Load balance, Distributed file systems, Clouds, Map Reduce