The rapid growth of commodity hardware capacity has made it feasible to build powerful unified information systems on top of decentralized, inexpensive off-the-shelf components. Such clustered systems consolidate the resources of component nodes to serve demanding workloads, while the nodes can be heterogeneous in capacity, vendor, hardware, or even software. Clustered systems provide an attractive alternative to traditional centralized parallel systems, owing to their cost-effectiveness and increased scalability.
However, heterogeneity among the nodes in clustered systems pose significant challenges to the effectiveness of traditional load management techniques and endangers the performance as well as scalability of those systems. In this talk, we present a set of new techniques for efficient load management in clustered systems built with heterogeneous components. Our techniques adapt to the heterogeneous infrastructure and non-uniform workloads of clustered systems with no a priori knowledge of either. They also mesh well with the decentralized architecture of clustered systems, because they build on tunable hashing schemes and require little or no shared state among the cluster nodes. Our techniques facilitate the construction of self-managing resource allocation and load balancing systems in order to improve the performance of heterogeneous clustered systems and dramatically reduce their management cost. This allows clustered systems to scale to sizes that were previously unmanageable and allows a clustered system to enlist highly heterogeneous resources automatically.