University of Houston

Hadoop has been emerging as a popular distributed framework for data intensive computing in clustered environments. Main usage has been in parallel computing problems where interconnected clusters would transfer parts of the data between individual compute nodes to accomplish one job. The clusters are usually connected with shared network infrastructure where other applications also access and transfer on the same bandwidth. Specifically, Hadoop MapReduce jobs suffer when running in parallel with other traffic in the underlying network due to their sensitivity to delay between compute phases. We propose a dynamic priority mechanism realized by OpenFlow protocol on such an infrastructure with a preferred QoS policy over all other traffic. Furthermore, such priority mechanisms can be enhanced if coupled with dynamic network information on traffic issues has been provided for the underlying network. We proposed to use the emerging ALTO (Application Layer Traff ic Optimization) server to provide network traffic information to the OpenFlow controllers. The ALTO server will be based on the industry standard, IF-MAP (Interface for Metadata Access Protocol), to leverage publish/subscribe capabilities and the flexible schema definitions.