SPX: Collaborative Research: NG4S: A Next-generation Geo-distributed Scalable Stateful Stream Processing System.
This project advances stream processing to enable next-generation streaming applications to store and update state along with computation, therefore processing live data streams in a timely fashion from massive and geo-distributed datasets. Existing systems are mainly designed for stateless stream processing in intra-datacenter settings and do not scale well for running stream applications that contain large distributed states. This project breaks the traditional abstractions of a centralized architecture and hashtable-based stateless operators, redefining them with a new decentralized architecture and new memory-efficient stateful operators, which enables novel approaches to improve overall system performance and scalability.
Open Source Software:
All codes are provided under GNU General Public License (GPL) or as a web-service, which guarantees your freedom to use the software for academic purposes. For more information, help or comments please contact Dr. Hu.
- FP4S: Fragment-based Parallel State Recovery for Stateful Stream Applications [Source Code Download]
FP4S, a novel fragment-based parallel state recovery mechanism that can handle many simultaneous failures for a large number of concurrently running stream applications. The novelty of FP4S is that we organize all the application’s operators into a distributed hash table (DHT) based consistent ring to associate each operator with a unique set of neighbors.
- Pinchao Liu, Liting Hu, Hailu Xu, Dilma Da Silva, Qingyang Wang and Sarker Tanzir Ahmed, “FP4S: Fragment-based Parallel State Recovery for Stateful Stream Applications”, In Proceedings of 34th IEEE International Parallel and Distributed Processing Symposium (IPDPS 2020), May 2020.
- Hailu Xu, Liting Hu, Pinchao Liu, and Boyuan Guan, “Exploiting the Spam Correlations in Scalable Online Social Spam Detection”, In Proceedings of 2019 International Conference on Cloud Computing (CLOUD 2019), June 2019.（Best Student Paper Award）
- Hailu Xu, Liting Hu, Pinchao Liu, Yao Xiao, Wentao Wang, Jai Dayal, Qingyang Wang, and Yuzhe Tang, “Oases: An Online Scalable Spam Detection System for Social Networks”, In Proceedings of the 11th International Conference on Cloud Computing (IEEE Cloud), July 2018.
- Hailu Xu, Boyuan Guan, Pinchao Liu, William Escudero, and Liting Hu, “Harnessing the Nature of Spam in Scalable Online Social Spam Detection”, In The 2018 International Workshop on Big Social Media Data Management and Analysis, in conjunction with IEEE Big Data , December 2018.
- Xin Chen, Ymir Vigfusson, Douglas M. Blough, Fang Zheng, Kun-Lung Wu, and Liting Hu, “GOVERNOR: Smoother Stream Processing Through Smarter Backpressure”, In Proceedings of the 14th IEEE International Conference on Autonomic Computing (ICAC), July 2017.