Applying AWS and the Kafka Framework for Real-Time Weather Data Analysis

Authors

  • M. Jahir Pasha Associate Professor, Department Of Computer Science & Engineering. G. Pullaiah College of Engineering and Technology,Kurnool, India
  • V. Vijaya Chandra Rao Assistant Professor, Department Of Computer Science & Engineering. G. Pullaiah College of Engineering and Technology,Kurnool, India
  • P. Bhasha Assistant Professor, Department of Data Science,School of Computing, MOHAN BABU UNIVERSITY (Erstwhile Sree Vidyanikethan Engineering College (Autonomous))Tirupati – 517102, Andhra Pradesh
  • D. William Albert Professor, Ashoka Women's Engineering College Kurnool.
  • V. Sujatha Professor, Ashoka Women's Engineering College Kurnool.
  • K. K. Baseer Professor, Department of Data Science, School of Computing, MOHAN BABU UNIVERSITY (Erstwhile Sree Vidyanikethan Engineering College (Autonomous))Tirupati – 517102, Andhra Pradesh

Keywords:

Weather data, Big data analysis, Kafka, Amazon AWS, Amazon S3, Amazon Athena, Visualization, Zookeeper

Abstract

The weather forecasting assesses the change that is now taking place in the state of the atmosphere, making it an important and crucial process in people's everyday lives. Big data analysis is a technique for examining enormous quantity of information in order to discover undiscovered prototype and vital details that may improve outcomes The concept of Big Data has captured the attention of many spheres of society, including the climatology institution. Big data analytics will thereby develop climate prediction outcomes and help forecasters produce more precise weather predictions. Numerous big data techniques and technologies have been created to manage and analyse the substantial amount of weather data from diverse sources in order to accomplish this goal and provide useful responses. Issues with traditional data management approaches and technology solved by using analytics for big data in prediction of weather. The data is processed and analysed using cloud computing tools including Kafka, AWS S3,AWS Crawler, AWS Glue Data Catalogue, and Athena AWS. This framework helps in dealing and analyzing real-time information in other segments like  Stock exchange, e-business functionalities, gaming, etc. By using this framework useful features and hidden patterns in the real-time data can be extracted and analyzed. In addition, Machine Learning Model can be trained on the data that we generated..

Downloads

Download data is not yet available.

References

Y. -C. Tsao, Y. T. Tsai, Y. -W. Kuo and C. Hwang, "An Implementation of IoT-Based Weather Monitoring System," 2019 IEEE International Conferences on Ubiquitous Computing & Communications (IUCC) and Data Science and Computational Intelligence (DSCI) and Smart Computing, Networking and Services (SmartCNS), 2019, pp. 648-652, doi: 10.1109/IUCC/DSCI/SmartCNS.2019.00135.J.Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd ed., vol. 2. Oxford: Clarendon, 1892, pp.68–73.

Suciu, C. Beceanu, M. Anwar, C. M. Balaceanu and M. Dobrea, "Weather Monitoring for Predicting Thermal Comfort and Energy Efficiency," 2018 IEEE 24th International Symposium for Design and Technology in Electronic Packaging (SIITME), 2018, pp. 156-160, doi: 10.1109/SIITME.2018.8599260.K. Elissa, “Title of paper if known,” unpublished.

Mohapatra and B. Subudhi, "Development of a Cost-Effective IoT-Based Weather Monitoring System," in IEEE Consumer Electronics Magazine, vol. 11, no. 5, pp. 81-86, 1 Sept. 2022, doi: 10.1109/MCE.2021.3136833.

A. Suleykin and P. Panfilov, "Distributed Big Data Driven Framework for Cellular Network Monitoring Data," 2019 24th Conference of Open Innovations Association (FRUCT), 2019, pp. 430-436, doi: 10.23919/FRUCT.2019.8711912.

R. Wiska, N. Habibie, A. Wibisono, W. S. Nugroho and P. Mursanto, "Big sensor-generated data streaming using Kafka and Impala for data storage in Wireless Sensor Network for CO2 monitoring," 2016 International Workshop on Big Data and Information Security (IWBIS), 2016, pp. 97-102, doi: 10.1109/IWBIS.2016.7872896.

N. MAHESH, & A. VYSHNAVI. (2014). A Novel Control Strategy of PV making System with LPC for Loading Balance of Distribution Feeders. International Journal of Computer Engineering in Research Trends, 1(6), 570–574.

M. D'silva, A. Khan, Gaurav and S. Bari, "Real-time processing of IoT events with historic data using Apache Kafka and Apache Spark with dashing framework," 2017 2nd IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT), 2017, pp. 1804-1809, doi: 10.1109/RTEICT.2017.8256910.

A. Sgambelluri, A. Pacini, F. Paolucci, P. Castoldi and L. Valcarenghi, "Reliable and scalable Kafka-based framework for optical network telemetry," in Journal of Optical Communications and Networking, vol. 13, no. 10, pp. E42-E52, October 2021, doi: 10.1364/JOCN.424639.

N. Braunisch, S. Schlesinger and R. Lehmann, "Adaptive Industrial IoT gateway using kafka streaming platform," 2022 IEEE 20th International Conference on Industrial Informatics (INDIN), 2022, pp. 600-605, doi: 10.1109/INDIN51773.2022.9976153.

S. Winberg and S. Singh, "Real-Time Event-driven Air Quality Inspection Framework for City-wide Pollution Level Monitoring," 2021 International Conference on Electrical, Communication, and Computer Engineering (ICECCE), 2021, pp. 1-6, doi: 10.1109/ICECCE52056.2021.9514133.

Shweta P. Patil, Jyothi Digge, & Mahesh Kadam. (2020). Performance Analysis of Free Space Optical Communication Network with Different Modulation Techniques. International Journal of Computer Engineering in Research Trends, 7(10), 1–5.

Neha Narkhede, Gwen Shapira, and Todd Palino. 2017. Kafka: The Definitive Guide Real-Time Data and Stream Processing at Scale (1st.ed.).O'ReillyMedia, Inc.

Y. Drohobytskiy, V. Brevus and Y. Skorenkyy, "Spark Structured Streaming: Customizing Kafka Stream Processing," 2020 IEEE Third International Conference on Data Stream Mining & Processing (DSMP), 2020, pp. 296-299, doi: 10.1109/DSMP47368.2020.9204304.

K. Kato, A. Takefusa, H. Nakada and M. Oguchi, "A Study of a Scalable Distributed Stream Processing Infrastructure Using Ray and Apache Kafka," 2018 IEEE International Conference on Big Data (Big Data), 2018, pp. 5351-5353, doi: 10.1109/BigData.2018.8622415.

Suryakant Acharekar, Prashant Dawnade, Binay Kumar Dubey, & Prof. Prabhakar Mhadse. (2020). IoT Based Weather Monitoring System. International Journal of Computer Engineering in Research Trends, 7(4), 20–22.

B. D. D. Nayomi, D. W. Albert, V. Sujatha, K. K. Baseer and M. J. Pasha, "A Framework for Processing and Analysing Real-Time data in e-Commerce Applications," 2023 8th International Conference on Communication and Electronics Systems (ICCES), Coimbatore, India, 2023, pp. 1495-1500, doi: 10.1109/ICCES57224.2023.10192771.

Raj, R., & Sahoo, D. S. S. . (2021). Detection of Botnet Using Deep Learning Architecture Using Chrome 23 Pattern with IOT. Research Journal of Computer Systems and Engineering, 2(2), 38:44. Retrieved from https://technicaljournals.org/RJCSE/index.php/journal/article/view/31

Prasanthi, T. S. ., Rayavarapu, S. M. ., Lavanya, Y. L. ., Kumar, G. S. ., Rao, G. S. ., Goswami, R. K. ., & Yegireddy, N. K. . (2023). Performance Analysis of Different Applications of Image Inpainting Based on Exemplar Technique. International Journal on Recent and Innovation Trends in Computing and Communication, 11(4), 113–117. https://doi.org/10.17762/ijritcc.v11i4.6393

Rohokale, M. S., Dhabliya, D., Sathish, T., Vijayan, V., & Senthilkumar, N. (2021). A novel two-step co-precipitation approach of CuS/NiMn2O4 heterostructured nanocatalyst for enhanced visible light driven photocatalytic activity via efficient photo-induced charge separation properties. Physica B: Condensed Matter, 610 doi:10.1016/j.physb.2021.412902

Downloads

Published

02.09.2023

How to Cite

Pasha, M. J. ., Rao, V. V. C. ., Bhasha, P. ., Albert, D. W. ., Sujatha, V. ., & Baseer, K. K. . (2023). Applying AWS and the Kafka Framework for Real-Time Weather Data Analysis . International Journal of Intelligent Systems and Applications in Engineering, 12(1s), 457–465. Retrieved from https://ijisae.org/index.php/IJISAE/article/view/3428

Issue

Vol. 12 No. 1s (2024)

Section

Research Article

License

Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.

IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.

Most read articles by the same author(s)