Real-time Elastic Stream Analytics
Easy to be deployed by multiple normal machines
Fast to process streaming data
Scalable to dig data scenario
Elasticity is highly desirable for stream processing systems to guarantee low latency against workload dynamics,
such as surges in data arrival rate and fluctuations in data distribution. Existing systems achieve elasticity following a resource-centric approach that uses dynamic
key partitioning across the parallel instances, i.e. executors, to balance the workload and scale operators. However,
such operator-level key repartitioning needs global synchronization and prohibits rapid elasticity. To address
this problem, we propose an executor-centric approach, whose core idea is to avoid operator-level key repartitioning
while implementing each executor as the building block of elasticity. Following this new approach, we
design the Elasticutor framework with two level of optimizations: i) a novel implementation of executors, i.e.,
elastic executors, that perform elastic multi-core execution via efficient intra-executor load balancing and executor
scaling and ii) a global model-based scheduler that dynamically allocates CPU cores to executors based on
the instantaneous workloads. We implemented a prototype of Elasticutor and conducted extensive experiments.
Our results show that Elasticutor doubles the throughput and achieves an average processing latency up to 2 orders
of magnitude lower than previous methods, for a dynamic workload of real-world applications.
Publication: "Elasticutor: Rapid Elasticity for Realtime Stateful Stream Processing", under submission.
Massive data streams from sensors in Internet of Things (IoT) and smart devices with Global Positioning System (GPS) are now flooding to database systems for further processing
and analysis. The capability of real-time retrieval from both fresh and historical data turns out to be the key enabler to the real world applications in smart manufacturing and smart city
utilizing these data streams. In this project, we present a simple and effective distributed solution to achieve millions of tuple insertions per second and ad-hoc temporal range query processing
in milliseconds. To this end, we propose a new data partitioning scheme that takes advantage of the workload characteristics and avoids expensive global data merging. Furthermore, to resolve
the throughput bottleneck, we adopt a template-based index method to skip unnecessary index structure adjustments over the relatively stable distribution of incoming tuples. To parallelize
data insertion and query processing, we propose an efficient dispatching mechanism and effective load balancing strategies to fully utilize computational resources in a workload-aware
manner. On both synthetic and real workloads, our solution consistently outperforms state-of-the-art open-source systems by at least an order of magnitude.
Publication: "Waterwheel: Realtime Indexing and Temporal Range Query Processing over Massive Data Streams" , accepted by IEEE International Conference on Data Engineering (ICDE) 2018.
This project aims to design real-time energy optimization solution for chiller plants. We have studied data collected from different chiller plants.
By incorporating domain knowledge, a chiller plant power prediction model is proposed by decomposing and decoupling the system into small components and applying different models to each component.
Additional modules are added to model the relationships among components. With domain knowledge, we are able to achieve high accuracy in modeling with simple models applied.
Based on accurate power prediction models, a real-time optimization approach is proposed and applied on chiller plants by adjusting control parameters of individual components to save total power consumption.
The real-time optimization method is applied online since July and our empirical evaluation shows that we are able to achieve about 7% savings.
The next optimization problem to study is to learn to configure (e.g., turn on/off) components given weather and cooling load information.
We are also investigating applying transfer learning so as to speed up migrating chiller plant optimization algorithms to different sites.
Publication: "Data Driven Chiller Plant Energy Optimization with Domain Knowledge", in Proceedings of ACM International Conference on Information and Knowledge Mangement (CIKM) 2017.
The prosperity of mobile social network and location-based services, e.g., Uber, is backing the explosive growth of spatial temporal streams on the Internet.
It raises new challenges to the underlying data store system, which is supposed to support extremely high-throughput trajectory insertion and low-latency querying with spatial and temporal constraints.
State-of-the-art solutions, e.g., HBase, do not render satisfactory performance, due to the high overhead on index update and the lack of appropriate load balancing.
In this demo, we present our new system prototype, DITIR, tailored to efficiently processing temporal and special queries over historical data as well as latest updates.
Our system provides better performance guarantee, by physically partitioning the incoming data tuples on their arrivals and exploiting a template-based insertion schema, to reach the desired ingestion throughput.
Publication: "DITIR: Distributed Index for High Throughput Trajectory Insertion and Real-time Temporal Range Query", in Proceedings of PVLDB 2017, demo.
Key-based workload partitioning is a common strategy used in parallel stream processing engines.
It is likely to generate poor balancing performance when workload variance occurs on the incoming data stream.
This study presents a new key-based workload partitioning framework, with practical algorithm to support dynamic workload assignment for stateful operators.
To put it simply, in our demonstration, users can configure the balance degree of parallel tasks according the performance of system.
When distribution fluctuations occur on the incoming data stream, system adaptively rebalance the workload with the configured memory consumption and the minimal migration overhead within stateful operator.
We have implemented and integrated our proposed framework with Apache Storm.
Publication: "Distributed Stream Rebalance for Stateful Operator under Workload Variance", in Proceedings of ACM International Symposium on High Performance Parallel and Distributed Computing (HPDC) 2017.
With the growth of spatio-textual messages on social media and registered queries, the computation workload for publish/subscribe systems is quickly increasing, which exceeds the capacity of a single server.
This calls for a distributed solution to building a publish/subscribe system over the spatio-textual data stream.
Moreover, due to the dynamic nature of the social media, the workload of publish/subscribe systems varies dramatically over time.
Motivated by this, we design and present a distributed Publish/Subscribe system for handling subscription queries over a Spatio-textual data stream, and named the system PS2Stream.
Publication: "Distributed Publish/Subscribe Query Processing on the Spatio-Textual Data Stream", in Proceedings of IEEE ICDE 2017.
Vibration sensor is becoming essential to industry sites, used to monitor the running status of crucial devices, such as motor and pipes.
In this work, we attempt to analyze the evolution of vibration pattern, in order to estimate the remaining usefulness lifetime (RUL) of the subject devices.
We have applied the analytical technique on real vacuum pumps in SKT’s FAB factories in South Korea.
Traditional maintenance strategy simply replaces the pumps in every six months, to avoid any potential problem with the vacuum, which generates huge amount of waste on the replacement of pumps in good health conditions.
After using our analytical outcomes, we can improve the lifetime of the pumps by 2 times and therefore save the operational cost of pump replacement by 50%.
Publication: "Vibration Analysis for IoT Enabled Predictive Maintenance", in Proceedings of IEEE ICDE 2017.
We have developed a demo to generate the haze-free video from the hazy video based on the Resa platform. The video on the left side is the input hazy video.
The video on the right is the dehazed result generated by a cluster of PCs with the Resa distributed streaming processing platform.
It can be observed that with the Resa platform, the haze removal algorithm is able to achieve real-time requirement.
Publication: "Component-Based Distributed Framework for Coherent and Real-Time Video Dehazing", in ACCV 2016, demo.
We have developed logo detection demo based on our resa platform.
This is a video demo to show how we re-design and implement the logo detection algorithm with our Resa platform, which is a distributed streaming processing platform.
The comparison between the offline version (single machine in the middle window) and the online version (PC cluster in the right window) shows the efficiency and effectiveness of our Resa platform.
Publication: "Real-Time Logo Recognition from Live Video Streams Using an Elastic Cloud Platform", in WAIM 2016.
We have developed a demo to trace and draw dense trajectories obtained from human actions based on our resa platform.
This demo is accepted by the Video Demo Session of ACM Multimedia Conference, 2015. This is a video demo to trace and draw trajectory of human actions.
The video in the middle part is the result generated by a single PC (standalone algorithm) and the right window shows the results generated by a cluster of PCs with our Resa distributed streaming processing platform. It clearly shows that with the help of our Resa platform, this complicated dense trajectory algorithm can be almost processed and displayed in real time.
Publication: "LiveTraj: Real-Time Trajectory Tracking over Live Video Streams", in ACM Multimedia, video demo, 2015.
This is a video demo to classify human actions in real time based on dense trajectory tracking techniques. The video on the right side is the result generated by a single PC (offline algorithm) and the video in the middle illustrates the results generated by a cluster of PCs with our Resa distributed streaming processing platform. It clearly shows that with the help of Resa platform, this complicated Human Action detection algorithm can be processed and displayed in real time.Watch Demo
This demonstration presents the first ever system implementation of real-time public transportation crowd prediction based on telecommunication data, relying on the analytical power of advanced neural network models and the computation power of parallel streaming analytic engines. By analyzing the feeds of caller detail records (CDR) from mobile users in regions of interest, our system is able to predict the number of metro passengers entering stations, the number of waiting passengers on the platforms and other important metrics on crowd density. New techniques, including geographical spatial data processing, weight-sharing recurrent neural network, and parallel streaming analytical programming, are employed in the system. These new techniques enable accurate and efficient prediction outputs, to meet the real-world business requirements from a public transportation system. This work will be presented at ICDE 2016.
Publication: "Mercury: Metro Density Prediction with Recurrent Neural Network on Streaming CDR Data", in ICDE (demo), 2016.
We have proposed and developed DRS, a novel dynamic resource scheduler for cloud-based data stream process engines. DRS overcomes three fundamental challenges: 1) how to model the relationship between the provisioned resources and query response time
2) where to best place resources to minimize tuple average complete latency
3) how to implement resource scheduling on top of Apache Storm with minimal overhead
In particular, DRS includes an accurate performance model based on the theory of Jackson open queueing networks and is capable of handling arbitrary operator topologies, possibly with loops, splits and joins.
Publication: "DRS: Dynamic Resource Scheduling for Real-Time Analytics over Fast Streams" in ICDCS 2015.
We have built a basic prototype of Resa before the project formally started, which has limited functionalities. In the reporting period, we focused on the performance modelling module of Resa, and devised an accurate model for a simplified version of Resa, based on the theory of Jackson queuing networks. We have validated our model through simulations and a real dataset from Twitter, and obtained encouraging results.
In the Resa framework, the basic programming unit is an operator. Each streaming analytical job consists of operators that collaboratively process streaming data, and continuously update analytics results. As shown in the left figure, an operator consists of a computing function and a local storage component. Note that our model employs a key-binding strategy, which forces the input tuples and the local storage structure to share the same key domain. This feature is important and crucial to the system design on the generic system platform with elasticity.
A major design aspect of Resa is the parallel processing structure of the logic operators, which facilitates load balancing and data locality. The structure of the operator relies on the workload decomposition mechanism, as is illustrated in the right figure. For each logical operator, there are multiplephysical workers (e.g., each running on a different computational node) that collaboratively process the operator’s workload. All workers of an operator share the same processing logic, i.e., the computing function of the operator. Meanwhile, each worker maintains its own internal storage. The workload distribution is based on a hash function from the input tuple’s key domain of the operator to the IDs of its workers. The system is responsible for the maintenance of the hash function in each operator, and continuously updating to the workers of the predecessor operator. Therefore, each worker knows where to route their output tuples, i.e., to the workers of the successor operator.
Publication: "Resa: Realtime Elastic Streaming Analytics in the Cloud", in ACM SIGMOD, 2013, poster.
Before the project formally started in Jan 2014, we did a side project on auction-based service differentiation, which led to our IEEE IC2E 2013 best paper, and the Abacus prototype. Abacus is able to allocate cloud resources, such as processing capacity, storage, bandwidth, etc., to users with different needs, using an auction-based mechanism that ensures that each user’s dominating strategy is to tell the system its exact needs and bids of the cloud resources. Recently, we extended Abacus to support quality-of-service contracts for individual users, and evaluated it on the Amazon EC2 platform, a popular elastic cloud service. We have submitted the extended version to Information System journal, a distinguished journal on information management and system research.
There are two important components in Abacus, Auctioneer and Scheduler. The left figure presents the relationship between these components, in the context of the MapReduce framework. The auctioneer is responsible for the scheduling probability assignment. When jobs are added or removed from the system, the auctioneer recalculates the probability assignment vectors for the scheduler on all types of resources. To support jobs in MapReduce, there are two queues for map nodes and reduce nodes respectively. Given the probabilities derived by the auctioneer, the scheduler selects the next job waiting for certain resources according to the probabilities. This selection procedure runs again when one of the tasks finishes the computation and returns the resource to the scheduler.
Publication: "Abacus: An Auction-Based Approach to Cloud Service Differentiation", in IC2E 2013, best paper award.
Li Wang, Ruichu Cai, Tom Z. J. Fu, Jiong He, Zijie Lu, Marianne Winslett, Zhenjie Zhang,
“Waterwheel: Realtime Indexing and Temporal Range Query Processing over Massive Data Streams”,
accepted by IEEE ICDE, 2018.
Hoang Dung Vu, KoK Soon Chai, Bryan Keating, Nurislam Tursynbek, Boyan Xu, Kaige Yang, Xiaoyan Yang and Zhenjie Zhang, “Data Driven Chiller Plant Energy Optimization with Domain Knowledge”, in Proceedings of ACM CIKM, 2017. [source]
Ruichu Cai, Zijie Lu, Li Wang, Zhenjie Zhang, Tom Z. J. Fu and Marianne Winslett, “DITIR: Distributed Index for High Throughput Trajectory Insertion and Real-time Temporal Range Query”, (Demo) in the demo session of PVLDB, 2017. [source]
Junhua Fang, Rong Zhang, Tom Z. J. Fu, Zhenjie Zhang, Aoying Zhou and Junhua Zhu, “Parallel Stream Processing Against Workload Skewness and Variance”, in Proceedings of ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC), 2017. [source]
Zhida Chen, Gao Cong, Zhenjie Zhang, Tom Z. J. Fu and Lisi Chen, “Distributed Publish/Subscribe Query Processing on the Spatio-Textual Data Stream”, in proceedings of IEEE ICDE 2017. [source]
Deokwoo Jung, Zhenjie Zhang and Marianne Winslett, “Vibration Analysis for IoT Enabled Predictive Maintenance”, in proceedings of IEEE ICDE 2017. [source]
Ning Wang, Xiaokui Xiao, Yin Yang, Zhenjie Zhang, Yu Gu, Ge Yu, “PrivSuper: a Superset-First Approach to Frequent Itemset Mining under Differential Privacy”, in proceedings of IEEE ICDE 2017. [source]
Ganzhao Yuan, Yin Yang, Zhenjie Zhang and Zhifeng Hao, “Semidefinite Optimization for Linear Aggregate Query Processing under Approximate Differential Privacy”, in proceedings of ACM SIGKDD, 2016. [source]
Parijat Mazumdar, Li Wang, Marianne Winslett and Zhenjie Zhang, “An Index Scheme for Fast Data Stream to Distributed Append-Only Store”, in proceedings of International Workshop on Web and Database (WebDB), 2016. [source]
Xiaoyan Yang, Shanshan Ying, Wenzhe Yu, Zhenjie Zhang and Rong Zhang, “Enhancing Topic Modeling on Short Texts with Crowdsourcing”, in proceedings of the 8th Asian Conference on Machine Learning (ACML), 2016. [source]
Junhua Fang, Rong Zhang, Xiaotong Wang, Tom Z. J. Fu, Zhenjie Zhang and Aoying Zhou, “Cost-Effective Stream Join Algorithm on Cloud System”, in proceedings of the 25th ACM International Conference on Information and Knowledge Management (CIKM), 2016. [source]
Meihua Wang, Jiaming Mai, Yun Liang, Tom Z. J. Fu, Zhenjie Zhang and Ruichu Cai, “Component-Based Distributed Framework for Coherent and Real-Time Video Dehazing”, (Demo) in proceedings of Asian Conference on Computer Vision, 2016. [source]
Victor C. Liang, Richard T. B. Ma, Wee Siong Ng, Li Wang, Marianne Winslett, Huayu Wu, Shanshan Ying, Zhenjie Zhang, “Mercury: Metro Density Prediction with Recurrent Neural Network on Streaming CDR Data”, (Demo) in proceedings of the 32nd IEEE International Conference on Data Engineering (ICDE) 2016, Helsinki, Finland. [source]
Jianbing Ding, Hongyang Chao, Mansheng Yang, “Real-Time Logo Recognition from Live Video Streams Using an Elastic Cloud Platform”, in proceedings of 17th International Conference on Web-Age Information Management (WAIM) 2016. [source]
Tom Z. J. Fu, Jianbing Ding, Richard T. B. Ma, Marianne Winslett, Yin Yang, Zhenjie Zhang, Yong Pei, Binbin Ni, “Demonstration of LiveTraj: Real-Time Trajectory Tracking from Live Video Streams”, in Proceedings of ACM Multimedia Conference, video demo, 2015. [source]
Tom Z. J. Fu, Jianbing Ding, Richard T. B. Ma, Marianne Winslett, Yin Yang, Zhenjie Zhang, “DRS: Dynamic Resource Scheduling for Real-Time Analytics over Fast Streams”, in proceedings of the International Conference of Distributed Computing Systems (ICDCS) 2015. [source]
Zhenjie Zhang, Richard T. B. Ma, Jianbing Ding, Yin Yang, “ABACUS: An Auction-Based Approach to Cloud Service Differentiation”, in proceedings of the 1st IEEE International Conference on Cloud Engineering (IC2E), 2013. Received the Best Paper Award. [source]
Tian Tan, Richard T. B. Ma, Marianne Winslett, Yin Yang, Yong Yu, Zhenjie Zhang, “Resa: Realtime Elastic Streaming Analytics in the Cloud”, in proceedings of ACM SIGMOD, 2013, poster. [source]
Zhenjie Zhang, Hu Shu, Zhihong Chong, Hua Lu, and Yin Yang, “C-Cube: Elastic Continuous Clustering in the Cloud”, in Proceedings of 29th IEEE International Conference on Data Engineering (ICDE), 2013. [source]
Ruichu Cai, Zhenjie Zhang, Zhifeng Hao, “SADA: A General Framework to Support Robust Causation Discovery”, in Proceedings of the 30th International Conference on Machine Learning (ICML), 2013. [source]
Yin Yang, Dimitris Papadias, Stavros Papadopoulos, Panos Kalnis, “Authenticated Join Processing in Outsourced Databases”, in Proceedings of the ACM SIGMOD, Providence, RI, USA, June 29-July 2, 2009. [source]
Zhenjie Zhang, Bing Tian Dai, Anthony K. H. Tung, “Estimating Local Optimums of EM Algorithm over Gaussian Mixture Model”, in Proceedings of 25th International Conference on Machine Learning (ICML), 2008. [source]
Yin Yang, Dimitris Papadias, “Just-In-Time Processing of Continuous Queries”, in Proceedings of the IEEE International Conference on Data Engineering (ICDE), Cancún, México, April 7-12, 2008. [source]
Tom. Z. J. Fu, Jianbing Ding, Richard TB Ma, Marianne Winslett, Yin Yang and Zhenjie Zhang,
“DRS: Auto-Scaling for Real-Time Stream Analytics”,
in IEEE/ACM Transactions on Networking (TON), vol: 25(6), Dec 2017.
Ruichu Cai, Zhenjie Zhang, Zhifeng Hao and Marianne Winslett, “Sophisticated Merging over Random Partitions: A Scalable and Robust Causal Discovery Approach”, accepted by IEEE Transactions on Neural Network and Learning Systems (TNNLS). [source]
Tuan-Anh Nguyen Pham, Xutao Li, Gao Cong and Zhenjie Zhang, “A General Recommendation Model for Heterogenous Networks”, in IEEE Transactions on Knowledge and Data Engineering (TKDE), Vol 28(12), 2016. [source]
Jianbing Ding, Zhenjie Zhang, Richard T. B. Ma, Yin Yang, “ABACUS: An Auction-Based Approach to Cloud Service Differentiation”, in Elsevier Journal of Computer Networks, Vol 94(C), 2016. [source]
Ruichu Cai, Zhenjie Zhang, Zhifeng Hao and Marianne Winslett, “Understanding Social Causalities Behind Human Action Sequences”, accepted by IEEE Transactions on Neural Networks and Learning Systems, 2016. [source]
Jia Xu, Bin Lei, Yu Gu, Marianne Winslett, Ge Yu and Zhenjie Zhang, “Efficient Similarity Join Based on Earth Mover's Distance Using MapReduce”, in IEEE Transactions on Knowledge and Data Engineering (TKDE), Vol 27(8), August 2015. [source]
Richard T. B. Ma, Dah Ming Chiu, John C. S. Lui, Vishal Misra and Dan Rubenstein, “Price Differentiation in the Kelly Mechanism”, in ACM Performance Evaluation Reviewer, vol 40 (2), pp. 30-33, September 2012. [source]
Richard T. B. Ma and Vishal Misra, “The Public Option: a Non-regulatory Alternative to Network Neutrality”, in IEEE/ACM Transactions on Networking (TON), Volume 21, Issue 6, Jan 2013. [source]
Richard T. B. Ma, Dah Ming Chiu, John C. S. Lui, Vishal Misra and Dan Rubenstein, “On Cooperative Settlement Between Content, Transit and Eyeball Internet Service Providers”, in IEEE/ACM Transactions on Networking (TON), Volume 19, Issue 3, June 2011. [source]
Zhenjie Zhang, Yin Yang, Anthony K. H. Tung, Dimitris Papadias, “Continuous K-Means Monitoring of Moving Objects”, in IEEE Transactions on Knowledge and Data Engineering (TKDE), 20(9): 1205-1216, 2008. [source]
Yin Yang, Jurgen Krämer, Dimitris Papadias, Bernhard Seeger, “HybMig: A Hybrid Approach to Dynamic Plan Migration for Continuous Queries”, in IEEE Transactions on Knowledge and Data Engineering (TKDE), 19(3): 398-411, 2007. [source]
We are a group of skilled individuals.
Department of Computer Science, University of Illinois at Urbana-Champaign
Senior Research Scientist
Adjunct Research Scientist
Assistent Professor, Division of Information and Communication Technologies,
College of Science, Engineering and Technology, Hamad Bin Khalifa University, Qatar.
Assistant Professor in School of Computing, National University of Singapore
Senior Research Engineer
Senior Research Engineer
Visiting postdoc researcher and Professor,
Faculty of Computer Science, Guangdong University of Technology, China
Visiting postdoc researcher and Associate Professor,
Faculty of Computer Science, Guangdong University of Technology, China
Visiting Postdoc Researcher and Associate Professor
School of Computer Science & Engineering, Southeast University, China
Visiting Postdoc Researcher and Assistant Professor
Guangxi University, China
Li Su is a summer 2015 intern from University of South Denmark.
Francesco Maturi is an intern from University of Trento.
Bryan Keating is an intern Xu is an intern from University of Illinois at Urbana-Champaign.
Nurlan Kanapin is a summer 2014 intern from Nazarbayev University.
Jiaming Mai is an intern from South China Agricultural University.
Jiachen Shi is an intern from Shanghai Jiao Tong University.
Zhaochen She is an intern from Shanghai Jiao Tong University.
Kaige Yang is an intern from University College London (UCL).
Zhenqiang Xu is an intern from University of Science and Technology of China, and now pursuing PhD in Tsinghua University.
For further inquiry about the project, please send email to email@example.com
For other information about ADSC, please visit ADSC’s website.
Advanced Digital Sciences Center
1 Create Way#14-02
Create Tower Singapore 138602
Phone: +65 6591 9090 | Fax: +65 6591 9091
© ADSC. All rights reserved.