Big Data Systems:

Oases, a system that implements a new decentralized overlay-based architecture to support social spam detection in online and distributed manner, with fetching online social data from multiple distributed web servers and updating the classifier with the aggregation of new social spam features to catch the latest spam. Experiments with an implementation of Oases have shown an efficient processing with more than 90% accuracy in online spam detection and attractive load balancing in scaling with hundreds of agents and millions of social posts.

See more…


GOVERNOR, a controller that factors the anticipated checkpointing costs into the backpressure mechanism in stream processing systems, and thus reduces the processing delay caused by checkpointing proactively, lowers the risk of system instability and improves overall throughput. Experiments with an implementation of GOVERNOR in Apache Spark Streaming have shown an overall performance improvement of up to 26% for representative streaming operators and real-world workloads, with negligible overhead.

See more…


ELF, a system that implements a new decentralized “many masters/many workers” architecture to support high-speed iterative, batch, and streaming processing jobs all together, and to offer runtime job feedback and cross-job coordination on large clusters. An ELF prototype implemented and evaluated for a larger scale configuration demonstrates high per-node throughput, sub-second job latency, and sub-second ability to adjust the actions of jobs being run.

See more…


Project Hoover, a middleware that provides the elasticity to horizontally scale-up or scale-down the real-time data processing systems according to current rates of input events and desired event processing latencies. Experiments with an implementation of Project Hoover in Apache Flume have shown Project Hoover can dynamically and continuously monitor Apache Flume’s performance, and right-size the number of Flume collectors according to different log production rates.

See more…


Datacenters and Cloud Computing:

RBay, an integrated information plane that enables secure and scalable resource sharing between geographically distributed datacenters, by building a decentralized ‘hierarchical aggregation tree’ to seamlessly aggregate spare resources from distributed datacenters and attach each datacenter ‘admin-customized’ handlers to customize the resource sharing policies. An experimental evaluation on eight real-world geo-distributed sites demonstrates RBay’s rapid response to composite queries, as well as its extensible, scalable, and lightweight nature.

See more…


v-Bundle, a system that enables weighted fair resource sharing across diverse applications belonging to one tenant, by letting tenants control their own scheduling and automatically shuffle free resources using DHTs. Experimental evaluations show that v-Bundle can scale well to thousands of hosts and VMs, provide customers with better bandwidth utilization and improved application’s quality of service (QoS) through borrowing extra bandwidth when needed, at no additional cost in terms of the total resources allocated to the customer.

See more…


System Virtualization:

DocMan, a toolset that adopts a black-box approach to discover container ensembles and collect information about intra-ensemble container interactions. It uses a combination of techniques such as distance identification and hierarchical clustering. The experimental results demonstrate that DocMan enables optimized containers placement to reduce the stress on bi-section bandwidth of the datacenter’s network. The method can detect container ensembles at low cost and with 92% accuracy and significantly improve performance for multi-tier applications under the best of circumstances.

See more…


Net-Cohort, a set of lightweight, “black-box” techniques that detect dependencies between collaborative VMs which jointly provide a service or accomplish a task, so as to enable smart VM placements and migrations. An implementation of Net-Cohort on a Xen-based system with 15 hosts and 225 VMs shows that its methods can detect VM ensembles at low cost and with about 90.0% accuracy.

See more…


Magnet, a multi-ring based scheme for optimizing power usage in virtualized datacenters using live VM migrations. The nodes are organized into a multi-layer ring-based overlay for transferring load from outer rings to inner rings. Experimental measurements show that the new scheme can reduce the power consumption by 74.8% over base at most with certain adjustably acceptable overhead. The effectiveness and performance insights are also analytically verified.

See more…


Middleware in Distributed Systems:

PKTown, a Peer-to-Peer middleware embedded into Multiplayer PVP Online Games (e.g., StarCraft, DOTA) that locates nodes satisfying target latency constraints. PKTown captures all packets generated by Star Craft. We apply application layer multicast (ALM) to transmit the broadcast packets through overlay network. PKTown extends the LAN game experiences to players belong to different LAN. No change is made on binary code of Star Craft.

We are thankful to the following federal agencies and Institutes for supporting our research efforts: