Applying AWS and the Kafka Framework for Real-Time Weather Data Analysis - Pasha, M. Jahir; Rao, V. Vijaya Chandra; Bhasha, P.; Albert, D. William; Sujatha, V.; Baseer, K. K. (International Journal of Intelligent Systems and Applications in Engineering, 2023)
The weather forecasting assesses the change that is now taking place in the state of the atmosphere, making it an important and crucial process in people's everyday lives. Big data analysis is a technique for examining enormous quantity of information in order to discover undiscovered prototype and vital details that may improve outcomes The concept of Big Data has captured the attention of many spheres of society, including the climatology institution. Big data analytics will thereby develop climate prediction outcomes and help forecasters produce more precise weather predictions. Numerous big data techniques and technologies have been created to manage and analyse the substantial amount of weather data from diverse sources in order to accomplish this goal and provide useful responses. Issues with traditional data management approaches and technology solved by using analytics for big data in prediction of weather. The data is processed and analysed using cloud computing tools including Kafka, AWS S3,AWS Crawler, AWS Glue Data Catalogue, and Athena AWS. This framework helps in dealing and analyzing real-time information in other segments like Stock exchange, e-business functionalities, gaming, etc. By using this framework useful features and hidden patterns in the real-time data can be extracted and analyzed. In addition, Machine Learning Model can be trained on the data that we generated..