Some Workload Scheduling Alternatives in a High Performance Computing Environment

About 15 years ago, clusters of commodity microprocessors largely overtook custom designed systems as the high performance computing (HPC) platform of choice. The design and optimization of workload scheduling systems for these clusters has been an active research area. This paper surveys some representative examples of workload scheduling methods used in contemporary large-scale applications such as Google, Yahoo, Facebook, and Amazon that employ a MapReduce parallel processing framework. It examines a specific MapReduce framework, Hadoop, in some detail. The paper describes a novel dynamic prioritization, self-tuning workload scheduling approach, and provides simulation results that suggest this approach will improve performance compared to the standard Hadoop scheduling algorithm.