Streaming machine learning algorithms with streaming big data systems
DOI:
https://doi.org/10.34117/bjdv10n1-021Keywords:
streaming big data, machine learning algorithms, decision-making, streaming SVMAbstract
As the era of big data unfolds, the need for real-time analytics and decision-making becomes increasingly crucial. Streaming big data systems, designed to process and analyse data in motion, have emerged as a pivotal solution for handling vast streams of continuously arriving information. This research delves into the synergy between streaming big data systems and machine learning algorithms, aiming to harness the power of real-time insights. We explore the challenges and opportunities presented by the dynamic nature of streaming data, emphasizing the importance of adapting traditional machine learning methodologies to suit the evolving requirements of streaming environments.The research begins with an overview of streaming big data systems, laying the foundation for understanding the unique characteristics of data in motion. We then delve into the selection and adaptation of machine learning algorithms that are well-suited for continuous learning and updating. Key aspects of the research include the preprocessing and feature extraction techniques tailored for real-time data streams, ensuring the effective utilization of streaming machine learning algorithms. The paper provides insights into the challenges of model training and updating in a dynamic environment, emphasizing the importance of accuracy and efficiency.
References
P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas, “Apache flink: Stream and batch processing in a single engine,” Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, vol. 36, no. 4, 2015.
S. Kamburugamuve, P. Wickramasinghe, K. Govindarajan, A. Uyar, G. Gunduz, V. Abeykoon, and G. Fox, “Twister: Net-communication library for big data processing in hpc and cloud environments,” in 2018 IEEE 11th International Conference on Cloud Computing (CLOUD). IEEE, 2018, pp. 383–391.
P. Wickramasinghe, S. Kamburugamuve, K. Govindarajan, V. Abeykoon, C. Widanage, N. Perera, A. Uyar, G. Gunduz, S. Akkas, and G. Fox, “Twister2: Tset high-performance iterative dataflow,” in 2019 International Conference on High Performance Big Data and Intelligent Systems (HPBD&IS). IEEE, 2019, pp. 55–60.
S. Kamburugamuve, K. Govindarajan, P. Wickramasinghe, V. Abeykoon, and G. Fox, “Twister2: Design of a big data toolkit,” Concurrency and Computation: Practice and Experience, p. e5189, 2017.
L. Neumeyer, B. Robbins, A. Nair, and A. Kesari, “S4: Distributed stream computing platform,” in 2010 IEEE International Conference on Data Mining Workshops, Dec 2010, pp. 170–177.
Z. Qian, Y. He, C. Su, Z. Wu, H. Zhu, T. Zhang, L. Zhou, Y. Yu, and Z. Zhang, “Timestream: Reliable stream computation in the cloud,” in Proceedings of the 8th ACM European Conference on Computer Systems, ser. EuroSys ’13. New York, NY, USA: ACM, 2013, pp. 1–
D. G. Murray, F. McSherry, R. Isaacs, M. Isard, P. Barham, and M. Abadi, “Naiad: a timely dataflow system,” in Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles. ACM, 2013, pp. 439–455.
W. N. Street and Y. Kim, “A streaming ensemble algorithm (sea) for large-scale classification,” in Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2001, pp. 377–382.
E. Hazan, A. Rakhlin, and P. L. Bartlett, “Adaptive online gradient descent,” in Advances in Neural Information Processing Systems, 2008, pp. 65–72.
S. Zhong, “Efficient online spherical k-means clustering,” in Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., vol. 5. IEEE, 2005, pp. 3180–3185.
S. Chintapalli, D. Dagit, B. Evans, R. Farivar, T. Graves, M. Holderbaugh, Z. Liu, K. Nusbaum, K. Patil, B. J. Peng et al., “Benchmarking streaming computation engines: Storm, flink and spark streaming,” in 2016 IEEE international parallel and distributed processing symposium workshops (IPDPSW). IEEE, 2016, pp. 1789–1792.
J. Gates and Y.-Y. Ahn, “The impact of random models on clustering similarity,” arXiv preprint arXiv:1701.06508, 2017.
Ö. Gözüaçık, A. Büyükçakır, H. Bonab, and F. Can, “Unsupervised concept drift detection with a discriminative classifier,” in Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 2365– 2368, 2019.
M. Carnein, D. Assenmacher, and H. Trautmann, “An empirical comparison of stream clustering algorithms,” in Proceedings of the Computing Frontiers Conference, CF’17, (New York, NY, USA), p. 361–366, Association for Computing Machinery, 2017.
Q. Zhang, Z. Chen, and L. T. Yang, “A nodes scheduling model based on markov chain prediction for big streaming data analysis,” International Journal of Communication Systems, vol. 28, no. 9, pp. 1610–1619, 2015.
T. Kolajo, O. Daramola, and A. Adebiyi, “Big data stream analysis: a systematic literature review,” Journal of Big Data, vol. 6, no. 1, p. 47, 2019.
Al Jallad, Khloud&Desouki, Said &Aljnidi, Mohamad. (2020). Anomaly detection optimization using big data and deep learning to reduce false-positive. Journal of Big Data. 7. 10.1186/s40537-020-00346-1.
Okereke, G.E., Bali, M.C., Okwueze, C.N. et al. K-means clustering of electricity consumers using time-domain features from smart meter data. Journal of Electrical Systems and Inf Technol 10, 2 (2023).
Chatterjee, Piyali & Basu, Subhadip & Kundu, Mahantapas&Nasipuri, Mita &Plewczynski, Dariusz. (2011). PSP-MCSVM: Brainstorming consensus prediction of protein secondary structures using two-stage multiclass support vector machines. Journal of molecular modeling. 17. 2191-201. 10.1007/s00894-011-1102-8.
