Análise exploratória entre arquiteturas de processamento de dados em lotes e tempo real
DOI:
https://doi.org/10.34115/basrv8n1-014Keywords:
processamento de dados, processamento em lotes, processamento em tempo realAbstract
Neste cenário de crescimento exponencial nos dados globais, a habilidade de processamento e análise torna-se vital para empresas e organizações. Este artigo tem por objetivo a análise de duas abordagens fundamentais: processamento em tempo real e processamento em lotes. Ao analisar métricas de comparação, características e vantagens/desvantagens, este estudo busca capacitar organizações na escolha entre tais abordagens, alinhando-as com suas necessidades. O estudo envolveu testes práticos de desempenho e comportamento, incluindo um caso real no Tribunal Regional Eleitoral do Rio Grande do Norte, utilizando ferramentas como Apache Airflow, Apache Spark e Apache Kafka. Os resultados destacam a eficácia do processamento em lotes para análises retrospectivas e relatórios, enquanto o processamento em tempo real é essencial para situações que demandam baixa latência, como detecção de irregularidades, monitoramento de redes sociais e análise de dados de sensores.
References
ALIZADA, M. PhD Thesis [s.l.] UNIVERSITY OF TARTU, 2021.
ANDRADE, H.; GEDIK, B.; TURAGA, D. Fundamentals of Stream Processing: Application Design, Systems, and Analytics. 2014.
ARORA, B. Big Data Analytics: The Underlying Technologies Used by Organizations for Value Generation. Em: CHAHAL, H.; JYOTI, J.; WIRTZ, J. (Eds.). Understanding the Role of Business Analytics: Some Applications. Singapore: Springer Singapore, 2019. p. 9–30.
BELOV, S. et al. BATCH AND STREAM BIG DATA PROCESSING PLATFORM: CASE OF NETWORK TRAFFIC ANALYSIS. 2020. Disponível em: <https://api.semanticscholar.org/CorpusID:231401491>
BENJELLOUN, S. et al. Big Data Processing: Batch-based processing and stream-based processing. 2020 Fourth International Conference On Intelligent Computing in Data Sciences (ICDS). Anais...2020.
DANIEL, F.; POZZI, G. Aiding the Development of Active Applications: A Decoupled Rule Management Solution. p. 250–270, 2010.
FIDELIS, MICHELE B.; ARROYO, JOSÉ E. C.. Meta-heurísticas para o sequenciamento de famílias de tarefas em máquinas paralelas idênticas de processamento em lote: Brazilian Journal of Development, v. 8, n. 9, p. 62006-62027, 2022.
GRABS, T.; LU, M. Measuring Performance of Complex Event Processing Systems. (R. Nambiar, M. Poess, Eds.)Topics in Performance Evaluation, Measurement and Characterization. Anais...Berlin, Heidelberg: Springer Berlin Heidelberg, 2012.
HEIDE, L. Punched - Card Systems and the Early Information Explosion, 1880–1945. Punched-Card Systems and the Early Information Explosion, 1880-1945, p. 1–369, jan. 2009.
IBTISSAME, K.; YASSINE, R.; HABIBA, C. Real time processing technologies in big data: Comparative study. 2017 IEEE International Conference on Power, Control, Signals and Instrumentation Engineering (ICPCSI). Anais...2017.
JANKOV, D. et al. Real-Time High Performance Anomaly Detection over Data Streams: Grand Challenge. Proceedings of the 11th ACM International Conference on Distributed and Event-Based Systems. Anais...: DEBS ’17.New York, NY, USA: Association for Computing Machinery, 2017. Disponível em: <https://doi.org/10.1145/3093742.3095102>
KAR, S. et al. On the Throughput Optimization in Large-scale Batch-processing Systems. Performance Evaluation, v. 144, p. 102142, 2020.
KISTERMANN, F. W. The Invention and Development of the Hollerith Punched Card: In Commemoration of the 130th Anniversary of the Birth of Herman Hollerith and for the 100th Anniversary of Large Scale Data Processing. Annals of the History of Computing, v. 13, n. 3, p. 245–259, 1991.
KOLAJO, T.; DARAMOLA, O.; ADEBIYI, A. Big data stream analysis: a systematic literature review. Journal of Big Data, v. 6, p. 47, jun. 2019.
KUMAR, Y. Lambda Architecture - Realtime Data Processing. PhD Thesis—[s.l: s.n.].
MARTIN, N.; PUFAHL, L.; MANNHARDT, F. Detection of batch activities from event logs. Information Systems, v. 95, p. 101642, 2021.
MARZ, N.; WARREN, J. Big Data: Principles and Best Practices of Scalable Realtime Data Systems. 1st. ed. USA: Manning Publications Co., 2015.
MOGA, A. Load management for streaming analytics. Doctoral Thesis—Zürich: ETH Zurich, 2013.
MARPU, RAMESH; MANJULA, BAIRAM. Streaming machine learning algorithms with streaming big data systems. Brazilian Journal of Development, v. 10, n. 1, p. 322-339, 2024.
OTTINGER, J. B.; GURUZU, S.; MAK, G. Batch Processing and Native SQL. p. 187–195, 2015.
PFANDZELTER, T.; BERMBACH, D. IoT Data Processing in the Fog: Functions, Streams, or Batch Processing? 2019 IEEE International Conference on Fog Computing (ICFC). Anais...2019.
PISHGOO, B.; AZIRANI, A. A.; RAAHEMI, B. A dynamic feature selection and intelligent model serving for hybrid batch-stream processing. Knowledge-Based Systems, v. 256, p. 109749, 2022.
POTKONJAK, M.; SRIVASTAVA, M. B. Behavioral synthesis of high performance, low cost, and low power application specific processors for linear computations. Proceedings of IEEE International Conference on Application Specific Array Processors (ASSAP’94). Anais...1994.
SETH, M. Mulesoft – Salesforce Integration Using Batch Processing. 2018 5th International Conference on Computational Science/ Intelligence and Applied Informatics (CSII), p. 7–14, 2018.
SHAHRIVARI, S. Beyond Batch Processing: Towards Real-Time and Streaming Big Data. Computers, v. 3, n. 4, p. 117–129, 2014.
SHAIKH, E. et al. Apache Spark: A Big Data Processing Engine. 2019 2nd IEEE Middle East and North Africa COMMunications Conference (MENACOMM). Anais...2019.
SMITH, A.; JONES, B. On the Complexity of Computing. Em: SMITH-JONES, A. B. (Ed.). Advances in Computer Science. [s.l.] Publishing Press, 1999. p. 555–566.
SPINNER, S. et al. Evaluating approaches to resource demand estimation. Performance Evaluation, v. 92, p. 51–71, 2015.
TAHER, N. C. et al. An IoT-Cloud Based Solution for Real-Time and Batch Processing of Big Data: Application in Healthcare. 2019 3rd International Conference on Bio-engineering for Smart Technologies (BioSMART). Anais...2019.
WORK, A. People in DP: The 1980s. SIGCPR Comput. Pers., v. 8, n. 3, p. 7–9, set. 1980.
WU, H. et al. A Reactive Batching Strategy of Apache Kafka for Reliable Stream Processing in Real-time. 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). Anais. 2020.
