Flink-ML: machine learning in Apache Flink
DOI:
https://doi.org/10.38152/bjtv7n4-015Keywords:
Apache Flink, Flink-ML, scalable machine learning, distributed algorithmsAbstract
The emergence of Big Data has spurred the development of various frameworks designed for efficient data storage and processing. Key frameworks include Hadoop, Spark, Flink, Storm, Pig, and Zookeeper. Among these, Apache Flink stands out as a prominent open-source platform known for its powerful stream and batch processing capabilities. It functions as a versatile engine for large-scale processing, incorporating built-in modules for streaming, SQL, machine learning (ML), and visualization tasks.This paper introduces Flink-ML, Flink’s open-source distributed machine learning library, which has been added to the Flink ecosystem in response to the exponential growth of machine learning applications in recent years. Flink-ML addresses the increasing demand for scalable machine learning solutions by offering efficient implementations of a variety of algorithms. As the community around Flink continues to grow, so too does the number of contributors and available algorithms within Flink-ML.Flink-ML is designed to support multiple programming languages and provides a high-level API that leverages Flink’s rich ecosystem. This integration simplifies the development of end-to-end machine learning pipelines, allowing developers to efficiently build and deploy models. Overall, Flink-ML enhances the capabilities of the Flink framework, making it an ideal choice for organizations looking to harness the power of machine learning within their Big Data projects.
References
Ameet Talwalkar. BerkeleyX CS190-1x: Scalable machine learning. https://www.edx.org/course/scalable-machine-learning-uc-berkeleyx-cs190-1x, 2015.
Amin Shokrzade, Mohsen Ramezani , Fardin Akhlaghian, Mahmud Abdulla Mohammad. A novel extreme learning machine based kNN classification method for dealing with big data, Expert Systems with Applications.V 183, 2021, 115293. https://doi.org/10.1016/j.eswa.2021.115293
Apache Hadoop Project. Apache Hadoop. http://hadoop.apache.org. Accessed 14 July 2024.
Apache Mahout Project. Apache Mahout. http://mahout.apache.org. Accessed 14 July 2014.
Biswanath Panda, Joshua S. Herbach, Sugato Basu, and Roberto J. Bayardo. Planet: Massively parallel learning of tree ensembles with mapreduce. In International Conference on Very Large Databases, 2009.
Bondarev A.E., Galaktionov V.A., Shapiro L.Z. Processing and visual analysis of multi-dimensional data / Scientific Visualization. V.9, № 5, с.86-104, 2017, DOI:http://doi.org/10.26583/sv.9.5.08
Burak Yavuz and Xiangrui Meng. Flink 1.1: Flink-ML performance improvements. https://databricks.com/?p=1393, 2014.
Chatzigeorgakidis, Georgios; Karagiorgou, Sophia; Athanasiou, Spiros; Skiadopoulos, Spiros. FML-kNN: scalable machine learning on Big Data using k-nearest neighbor joins. Journal of Big Data (2018-02-06) 5: 1-27 , , 2018.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent dirichlet allocation. Journal of Machine Learning Research, 3, 2003.
Diego García-Gil, Sergio Ramírez-Gallego, Salvador García, Francisco Herrera, "A comparison on scalability for batch big data processing on Apache Spark and Apache Flink," Big Data Analytics, p. 3, 2017.
Dean J, Ghemawat S. Mapreduce: Simplified data processing on large clusters. In: Proceedings of the 6th Conference on Symposium on Operating Systems Design & Implementation - Volume 6. OSDI’04. Berkeley: USENIX Association; 2004. p. 10–10.
Evan R. Flinks, Ameet Talwalkar, Virginia Smith, Jey Kottalam, Xinghao Pan, Joseph E. Gonzalez, Michael J. Franklin, Michael I. Jordan, and Tim Kraska. MLI: An API for Distributed Machine Learning. In International Conference on Data Mining, 2013.
Evan R Flinks, Ameet Talwalkar, Daniel Haas, Michael J. Franklin, Michael I. Jordan, and Tim Kraska. Automating model search for large scale machine learning. Symposium on Cloud Computing, 2015.
Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011.
Fred Lu, Ryan R. Curtin, Edward, Francis, James Holt. Optimizing the Optimal Weighted Average: Efficient Distributed Sparse Classification. arxiv: 2406.01753, 2024.
Flink A. Apache Flink. http://flink.apache.org. Accessed 14 June 2020.
Flink Community. Flink community. https://Flink.apache.org/community.html, 2015.
Flink Packages. Flink packages. https://Flink-packages.org, 2015.
Flink-ML. Flink-ML user guide. https://Flink.apache.org/docs/latest/Flink-ML-guide.html, 2015.
Hamstra M, Karau H, Zaharia M, Konwinski A, Wendell P. Learning Spark: lightning-fast big data analytics. Sebastopol: O’Reilly Media; 2015.
Hamstra M, Karau H, Zaharia M, Konwinski A, Wendell P. Learning Spark: lightning-fast big data analytics. Sebastopol: O’Reilly Media; 2015.
G.M. JACQUEZ . A k nearest neighbour test for space–time interaction
Statistics in Medicine, 15 (1996), pp. 1935-1949.
Jeffrey Dean and Sanjay Ghemawat. Mapreduce: Simplified data processing on large clusters. In USENIX Symposium on Operating Systems Design and Implementation, 2004.
Jeremy Freeman. Introducing streaming k-means in Flink 1.2. https://databricks.com/?p=2382, 2023.
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: Understanding rating dimensions with review text. In ACM Recommender Systems Conference, 2013.
Lars Buitinck et al. API design for machine learning software: experiences from the scikit-learn project. arXiv:1309.0238, 2013.
Mahout. Apache mahout. http://mahout.apache.org/, Accessed 24 July 2024.
Ma, C., Smith, V., Jaggi, M., Jordan, M. I., Richtarik, P., & Takac, M. Adding vs. Averaging in Distributed Primal-Dual Optimization. ICML 2015 - International Conference on Machine Learning.
Matei Zaharia, Tathagata Das, Haoyuan Li, Timothy Hunter, Scott Shenker, and Ion Stoica. Discretized streams: Fault-tolerant streaming computation at scale. In Symposium on Operating Systems Principles, 2013.
Michael Armbrust, Reynold Xin, Cheng Lian, Yin Yuai, Davies Liu, Joseph Bradley, Xian-grui Meng, Tomer Kaftan, Michael Franklin, Ali Ghodsi, and Matei Zaharia. Flink SQL: Relational data processing in Flink. In ACM Special Interest Group on Management of Data, 2015.
MLlib. Machine Learning Library (MLlib) for Spark. http://spark.apache.org/docs/latest/mllib-guide.html. Accessed 04 July 2014.
Numpy. Numpy. http://www.numpy.org/, Accessed 24 July 2024.
Smith, V., Forte, S., Ma, C., Takac, M., Jordan, M. I., Jaggi, M. CoCoA: A General Framework for Communication-Efficient Distributed Optimization. Journal of Machine Learning Research (JMLR), 2018.
Apache Flink Architecture, https://www.cloudduggu.com/flink/architecture. Accessed 14 July 2023.
Y. Song, J. Liang, J. Lu, X. Zhao. An efficient instance selection algorithm for k nearest neighbor regression Neurocomputing, 251 (2017), pp. 26-34
Wehrend S. and Lewis C. “A Problem-Oriented Classification of Visualization Techniques”, Proceedings of the 1st IEEE Conference on Visualization '90, 1990, PP.139-143.
White T. Hadoop: The Definitive Guide. Sebastopol: O’Reilly Media, Inc; 2012.
Xiangrui Meng, Joseph Bradley, Evan Flinks, and Shivaram Venkataraman. Ml pipelines: A new high-level api for Flink-ML. https://databricks.com/?p=2473, 2015.
