DE202022103462U1 - A system for streaming and storing heterogeneous sensor data streams using a microservices architecture model - Google Patents

Abstract

A system for streaming and storing the heterogeneous streamed sensor data using a microservices architecture model, the system comprising:
a data acquisition processing unit for acquiring the heterogeneous data from different types of sensors, the acquired data being raw and unstructured;
a data cleaning processing unit for eliminating the redundant data and for filtering the required data from the raw data collected from heterogeneous sensor data;
a data exploration processing unit for performing the analysis of the heterogeneous sensor data of the unstructured data and thereby identifying the data patterns;
a feature engineering processing unit for converting the raw data into features, where this data is considered the clean data for performing analysis and the data is kept and then stored in the cloud, and where the features represent the underlying problem in the predictive model ;
a predictive modeling processing unit for finding interesting patterns in the collected data by performing predictive analysis; and
a data visualization unit that presents the analysis results to the user using various visual elements, such as B. diagrams and tables.

Description

FIELD OF THE INVENTION

The present disclosure relates to an approach to improve streaming analytics. In particular, the present disclosure relates to a system for streaming and storing the heterogeneous streamed sensor data using a microservices architecture model.

BACKGROUND OF THE INVENTION

The drastic improvement in technology led to a sudden increase in the amount of data generated by these technologies. The volume of data to be managed and stored is increasing enormously from day to day. Because of this massive amount of data, data analysis plays an important role in analyzing the data, be it Covid-19 or elections.

A large amount of heterogeneous information (information from different sources) is regularly generated, so it becomes necessary to process and analyze this information, which has become a tedious task due to the huge amount of such information. Data from multiple sources provides better and more meaningful insights than data from a single source.

The heterogeneous, streamed sensor data from various sources such as e.g. B. Sensors are generated continuously, and this generated data must be analyzed immediately to gain valuable insights and find interesting patterns in the data. Due to the generation of enormous amounts of data that need to be stored, it has become necessary to store only the important data, which is challenging due to the variety and speed of the data. If the data is stored directly in the cloud without any pre-processing, the end-user incurs additional costs for the unwanted and redundant information, resulting in a data analysis bottleneck. Therefore, there is a need for such a system that can remove the unwanted and redundant data from the heterogeneous streamed data to store the important data on the cloud server and perform the data analysis.

In view of the previous discussion, it becomes clear that a system for streaming and storing heterogeneous sensor data using a microservices architectural model is needed.

SUMMARY OF THE INVENTION

The present disclosure relates to a system for streaming and storing the heterogeneous streamed sensor data using a microservices architecture model. The proposed system provides a storage mechanism for heterogeneous streamed sensor data, where the clean and pre-processed data is stored in the cloud server to perform the data analysis, where the stored data has the required feature vectors for the streaming analysis. The proposed system is able to determine the pattern, examine the results and identify the relationship between the data coming from multiple sources at the same time. The proposed system uses an event-driven microservices architecture model that has architectural features such as high throughput, scalability, availability, reliability, modularity, maintainability, and agility. Microservices architecture consists of different components namely data collection component, data cleansing component, data exploration component, feature engineering component, predictive modeling component and data visualization component. The experimental components of the microservices architecture are: event-based streaming on Flink (Flink Runtime) for streaming the data from multiple sources, microservices for performing operations on the collected data, dynamic integration using Apache Kafka as a message queuing system for generation and Data publication is used, analytics through machine learning algorithms and an efficient dashboard for reporting on the analyzed information. The proposed approach helps to process real-time data on the fly, and only the data that is ready for analysis is stored for future use. Therefore, the proposed approach reduces the costs associated with streaming and storing sensor data, as the proposed approach uses a microservices-based event-driven architecture with Flink for streaming, which can help in achieving scalability and modularity.

The present disclosure aims to provide a system for streaming and storing the heterogeneous streamed sensor data using a microservices architecture model. The system includes: a data acquisition processing unit for collecting the heterogeneous data from different sensor types, the collected data being raw and unstructured; a data cleaning processing unit to eliminate the redundant data and to filter the required data from the raw data collected from the heterogeneous sensor data; a data exploration processing unit for performing the analysis of the heterogeneous sensor data of the unstructured data and thereby identifying the data patterns; a feature engineering processing unit for converting the raw data into features, where this data is considered the clean data for performing analysis and the data is preserved and then stored in the cloud, and where the features represent the underlying problem in the predictive model ; a predictive modeling processing unit for finding the interesting patterns in the collected data by performing the predictive analysis; and a data visualization processing unit for reporting the analysis results to the user using various visual elements such as graphs and charts.

A goal of the present disclosure is to provide a system for streaming and storing the heterogeneous streamed sensor data using a microservices architecture model.

Another goal of the present disclosure is to clean and pre-process the collected raw data and then store the pre-processed data in the cloud.

Another goal of the present disclosure is to use a microservices-based event-driven architecture with Flink for streaming.

Another goal of the present disclosure is to achieve scalability and modularity in streaming analysis.

In order to further clarify the advantages and features of the present disclosure, a more detailed description of the invention is provided by reference to specific embodiments that are illustrated in the accompanying figures. It is understood that these figures represent only typical embodiments of the invention and therefore should not be considered as limiting the scope of the invention. The invention will be described and illustrated with additional specificity and detail with the accompanying figures.

character list

• 1 ein Blockdiagramm eines Systems für das Streaming und die Speicherung von heterogenen gestreamten Sensordaten unter Verwendung eines Microservices-Architekturmodells in Übereinstimmung mit einer Ausführungsform der vorliegenden Offenbarung zeigt;
• 2 eine Streaming-Architektur des Flink-Programms in Übereinstimmung mit einer Ausführungsform der vorliegenden Offenbarung zeigt; und
• 3 den Versuchsaufbau gemäß einer Ausführungsform der vorliegenden Offenbarung veranschaulicht.

These and other features, aspects, and advantages of the present disclosure will be better understood when the following detailed description is read with reference to the accompanying figures, in which like characters represent like parts throughout the figures, wherein:

• 1 10 shows a block diagram of a system for streaming and storing heterogeneous streamed sensor data using a microservices architecture model, in accordance with an embodiment of the present disclosure;
• 2 Figure 12 shows a streaming architecture of the Flink program in accordance with an embodiment of the present disclosure; and
• 3 illustrates the experimental setup according to an embodiment of the present disclosure.

Those skilled in the art will understand that the elements in the figures are presented for simplicity and are not necessarily drawn to scale. For example, the flow charts illustrate the method of key steps to enhance understanding of aspects of the present disclosure. In addition, one or more components of the device may be represented in the figures by conventional symbols, and the figures only show the specific details relevant to understanding the embodiments of the present disclosure, not to encircle the figures with details to overload, which are easily recognizable to those skilled in the art familiar with the present description.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the invention, reference will now be made to the embodiment illustrated in the figures and specific language will be used to describe the same. It should be understood, however, that no limitation on the scope of the invention is intended, and such changes and further modifications to the illustrated system and such further applications of the principles of the invention set forth therein are contemplated as would occur to those skilled in the art invention would normally come to mind.

Those skilled in the art will understand that the foregoing general description and the following detailed description are exemplary and explanatory of the invention and are not to be taken as limiting.

When this specification refers to "an aspect", "another aspect" or the like, it means that a particular feature, structure or particular characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Therefore, the phrases "in one embodiment,""in another embodiment," and similar phrases throughout this specification may or may not all refer to the same embodiment.

The terms "comprises," "including," or other variations thereof are intended to cover non-exclusive inclusion, such that a method or method that includes a list of steps includes not only those steps, but may also include other steps that are not expressly stated or pertaining to any such process or method. Likewise, any device or subsystem or element or structure or component preceded by "comprises...a" does not, without further limitation, exclude the existence of other devices or other subsystem or other element or other structure or other component or additional device or additional subsystems or additional elements or additional structures or additional components.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The system, methods, and examples provided herein are for purposes of illustration only and are not intended to be limiting.

Embodiments of the present disclosure are described in detail below with reference to the attached figures.

The functional units described in this specification are referred to as devices. A device may be implemented in programmable hardware devices such as processors, digital signal processors, central processing units, field programmable gate arrays, programmable logic arrays, programmable logic devices, cloud processing systems, or the like. The devices can also be implemented in software for execution by various types of processors. An identified device may include executable code and may include, for example, one or more physical or logical blocks of computer instructions, which may be organized as an object, procedure, function, or other construct, for example. However, the executable file of an identified device does not have to reside physically in one location, but may consist of various instructions stored in different locations which, when logically assembled, form the device and fulfill the stated purpose of the device.

The executable code of a device or module can be a single instruction or a multitude of instructions, and can even be distributed across several different code segments, different applications, and multiple storage devices. Similarly, operational data may be identified and represented within the device and may be in any suitable form and organized in any suitable type of data structure. The operational data may be collected as a single set of data or distributed across various locations, including various storage devices, and may exist, at least in part, as electronic signals in a system or network.

When this specification refers to "a selected embodiment," "an embodiment," or "an embodiment," it means that a particular feature, structure, or characteristic described in connection with the embodiment is at least one embodiment of the disclosed subject matter. As such, the phrases "a selected embodiment," "in one embodiment," or "in one embodiment" throughout this specification are not necessarily all referring to the same embodiment.

Furthermore, the features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to provide a thorough understanding of the embodiments of the disclosed subject matter. However, one skilled in the art will recognize that the disclosed subject matter may be practiced without one or more of the specific details, or using other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the disclosed subject matter.

According to the example embodiments, the disclosed computer programs or modules may be executed in many example ways, such as as an application residing in a device's memory or as a hosted application running on a server and with the device application or browser communicates via a number of standard protocols such as TCP/IP, HTTP, XML, SOAP, REST, JSON and other sufficient protocols. The disclosed computer programs may be written in example programming languages executed from memory on the device or from a hosted server, such as BASIC, COBOL, C, C++, Java, Pascal, or scripting languages such as JavaScript, Python, Ruby, PHP, Perl or other sufficient programming languages.

Some of the disclosed embodiments include or involve data transmission over a network, e.g. B. the transmission of various inputs or files over the network. The network can be, for example, the Internet, wide area networks (WANs), local area networks (LANs), analog or digital wired and wireless telephone networks (e.g. PSTN, Integrated Services Digital Network (ISDN), a cellular network and digital subscriber line (xDSL)), radio, television, cable, satellite and/or other transmission or tunneling mechanisms for the transmission of data. The network may comprise several networks or sub-networks, each of which e.g. B. may include a wired or wireless data path. The network may comprise a circuit switched voice network, a packet switched data network or any other network for the transmission of electronic communications. The network may for example comprise Internet Protocol (IP) or Asynchronous Transfer Mode (ATM) based networks and may support voice e.g. B. VoIP, Voice-over-ATM or other comparable protocols for voice data communication. In one implementation, the network includes a cellular telephone network configured to allow the exchange of text or SMS messages.

Examples of a network include a personal area network (PAN), a storage area network (SAN), a home area network (HAN), a campus area network (CAN), a local area network (LAN), a wide area Network (WAN), a Metropolitan Area Network (MAN), a Virtual Private Network (VPN), an Enterprise Private Network (EPN), the Internet, a Global Area Network (GAN), and so on.

1 10 shows a block diagram of a system for streaming and storing heterogeneous streamed sensor data using a microservices architecture model, in accordance with an embodiment of the present disclosure. The system (100) comprises a data acquisition processing unit 102 for acquiring the heterogeneous data from different sensor types, where the acquired data is raw and unstructured.

In one embodiment, a data cleaning unit 104 is used to eliminate the redundant data and filter the required data from the raw data collected from heterogeneous sensor data.

In one embodiment, a data exploration processing unit 106 is used to perform the analyzes of the heterogeneous sensor data of the unstructured data, thereby identifying the data patterns.

In one embodiment, a feature engineering processing unit 108 is used to convert the raw data into features, which data is considered the clean data for performing analysis, and the data is preserved and then stored in the cloud, and the features represent the underlying problem in the prediction model.

In one embodiment, a predictive modeling processing unit 110 is used to find the interesting patterns in the collected data by performing predictive analysis.

In one embodiment, a data visualization unit 112 is used to present the analysis results to the user using various visual elements such as graphs and charts.

In one embodiment, the system uses a microservices-based self-directed architecture with Flink for streaming, which allows the proposed system to achieve scalability and modularity, with some other architectural features being high throughput, scalability, availability, reliability and agility, and maintainability.

In one embodiment, a test bench for an experimental setup is created in which the environmental data such as temperature and humidity values are streamed hourly through multiple devices in order to demonstrate the redundancy of the data.

In one embodiment, the experimental components of the architecture are Flink for streaming the data from multiple sensors, microservices for collecting, analyzing and reporting, dynamic integration using Apache Kafka as a message queuing system for generating data and then publishing the relevant topics, Analy tik to perform the data analysis using machine learning component algorithms in Python and dashboard to efficiently report the information after performing the analysis, with the dashboard showing charts and graphs to present the results of the predictive analysis.

In one embodiment, the data is collected using sensors such as MQ135, DHT11, MQ131, DHT22, the data is collected using Arduino and Raspberry Pi microcontrollers, the data is collected in the platform using the Think-Speak platform are published, the streaming platform is integrated using the MQTT Flink bridge, the data is published in the platform using the Think Speak platform, the streaming platform is integrated using the MQTT Flink bridge, the Flink Streaming consumer receives the data and then publishes it to the MongoDB database program using the Kafka integration layer.

In one embodiment, air quality data is also streamed from different sensor devices using multiple parameters and this data is combined with respect to the timestamp value and weather data is also combined to produce a single data stream for each timestamp.

In one embodiment, a Python-based program is used for data exploration, where the data is cleaned and pre-processed, removing unwanted and incorrect data, replacing missing data with appropriate values, and extracting only the required features for analysis, after which this corrected and sanitized data is uploaded to the cloud for analysis, so this process of aggregating and pre-processing data only preserves the necessary data, resulting in a reduction in the storage required to store the data.

In one embodiment, the data collection processing unit (102), the data cleaning processing unit (104), the data exploration processing unit (106), the feature engineering processing unit (108), the predictive modeling processing unit (110), and the data visualization processing unit Processing unit (112) be implemented in programmable hardware devices such as processors, digital signal processors, central processing units, field programmable gate arrays, programmable logic arrays, programmable logic devices, cloud processing systems or the like.

1. 1. Flink-Streaming: Es handelt sich um ein ereignisbasiertes Streaming, bei dem Apache Flink für das Streaming von Daten aus mehreren Quellen verwendet wird.
2. 2. Microservices: Diese Aufgabe dient dem Sammeln, Analysieren und Berichten der Daten. Microservices sind modular mit der Interaktion mit den Komponenten wie Codefunktion, Daten und anderen Ressourcen.
3. 3. Dynamische Integration: Apache Kafka wird als Nachrichten-Warteschlangensystem für die Produktion von Daten und deren Veröffentlichung in den entsprechenden Topics verwendet, wobei Flink Kafka Consumer für das Abonnieren und Konsumieren eines Topics für die lose Kopplung der Lösung und die Bereitstellung einer Integrationsschicht für die Integration mit verteilten Systemen von Drittanbietern verwendet wird.
4. 4. Analytik: Ein Algorithmus der Komponente für maschinelles Lernen wird zur Durchführung der Analyse der gespeicherten Daten verwendet, wobei der Algorithmus durch eine Python-Komponente implementiert wird. Bei den Sensordaten handelt es sich um Zeitreighendaten Zur Untersuchung der Muster wie z. B. langfristiger Trend, saisonale Auswirkungen und Korrelation zwischen dem Merkmalsvektor und dem unabhängigen Vektor, wird eine Deep-Learning-Technik zur Durchführung der Prognoseanalyse verwendet.
5. 5. Dashboard: Das Dashboard wird für die Berichterstattung über die Endergebnisse der Datenanalyse verwendet, wobei für die Berichterstattung visuelle Darstellungen verwendet werden, bei denen Grafiken und Diagramme für die Berichterstattung über die Ergebnisse der prädiktiven Analyse verwendet wurden.

The individual experimental components of the proposed architecture are explained below.

1. 1. Flink Streaming: It is an event-based streaming that uses Apache Flink to stream data from multiple sources.
2. 2. Microservices: This task is used to collect, analyze and report the data. Microservices are modular with the interaction with the components like code function, data and other resources.
3. 3. Dynamic Integration: Apache Kafka is used as a message queuing system for producing data and publishing it in the appropriate topics, while Flink Kafka Consumer for subscribing and consuming a topic for loosely coupling the solution and providing an integration layer for the Integration with third-party distributed systems is used.
4. 4. Analytics: An algorithm of the machine learning component is used to perform the analysis of the stored data, the algorithm being implemented by a Python component. The sensor data is time-series data. To examine the patterns such as e.g. B. long-term trend, seasonal effects and correlation between the feature vector and the independent vector, a deep learning technique is used to perform the forecast analysis.
5. 5. Dashboard: The dashboard is used for reporting the final results of data analysis using visual representations for reporting where graphs and charts were used for reporting the results of predictive analysis.

2 10 shows a streaming architecture of the Flink program, in accordance with an embodiment of the present disclosure. A Flink runtime environment has two types of processes, namely a job manager and one or more task managers, where the job manager is considered to be the coordinator of the Flink system, while the task manager is considered to be a worker that runs parts of runs parallel programs. In this architecture, each task is considered a thread.

3 illustrates the experimental setup according to an embodiment of the present disclosure. The experimental setup, as shown in the figure, consists of different layers. The first layer is the data collection layer where the sensor data is collected, with collection agents collecting the heterogeneous data generated by different sensors of the multi-parameters to be measured, the collector being a single point for the collection of the sensor data and the internal software pattern for the Implementation could be based on a singleton design pattern. The next layer is the streaming data processor, which stores the raw data that needs to be examined and does the feature engineering before performing predictive analysis. The integration layer performs integration and interfacing with external systems and third-party software to achieve various functionalities. The next layer is data analysis, where the data is analyzed using a machine learning algorithm. Once the analysis has been performed, reporting on the data is initiated in the data visualization layer, reporting is done using charts and tables.

A test bench was set up for the experimental analysis of the proposed system, where environmental data is streamed every hour. Temperature and humidity readings are streamed across multiple devices to demonstrate data redundancy. Apache Kafka is used as a message queuing system for generating data and publishing relevant topics. Apache Kafka has multiple themes and multiple partitions to facilitate parallel processing. The Flink-based consumer task is used to consume the topic containing the message and group the data, aggregating the data according to the timestamp with the key value to remove redundancy. The air quality telemetry data is also streamed from various sensor devices with multiple parameters. This data is combined in terms of the timestamp value. The weather data is also combined to create a single data stream for each timestamp. The task responsible for data exploration analyzes task which is a Python based program used to remove unwanted and incorrect data, replace missing values and extract features needed for analysis, and then this extracted data is uploaded to the cloud to perform analysis. So, aggregation and pre-processing keeps only the necessary data. Therefore, with the proposed approach, the amount of disk space required to store unwanted and unclear data is significantly reduced and the costs incurred by directional streaming are also reduced. Once the analyzes have been performed, the next task is to create a report that will be called from a thread and produce a visual representation of the analyzed data to be displayed on the dashboard.

The test bench created for the development of the experimental setup consists of the MQ135, DHT11, MQ131 and DHT22 sensors that collect the environmental data, with the data being collected using Arduino and Raspberry Pi microcontrollers. The data is published on the platform using the Think Speak IoT platform, which uses the MQTT protocol for publication. The integration of the streaming platform is done via the MQTT Flink bridge, where the Flink streaming consumer receives the data and publishes it to the MongoDB (database) via the Kafka integration layer. Once the pre-processing is complete and the predictive analysis has been performed. The results obtained are presented on the dashboard with the help of visual elements such as charts and graphs. Sample data is collected in a single day for a city and combined into a single data stream prior to analysis.

In one embodiment, the proposed approach for storing the heterogeneous sensor data processes the data in real-time and only the data that is ready for analysis is stored in the database for future reference. The proposed approach reduces the costs associated with streaming and storing the sensor data. The proposed approach also helps in achieving scalability and modularity as in the proposed approach a microservices based self-directed architecture is used together with Flink for streaming, with microservices component vertical scaling and containerized architecture enabling modularity and together with all this, the cloud environment is horizontally scalable. Predictive analytics is applied to find interesting patterns, trends, and insights. The proposed approach delivers a data stream that can be used for machine learning by storing the data in data storage in the cloud. The real-time data is stored in the MongDB where it is used as historical data for future use. The stored data can be used to build a predictive model that uses a deep learning technique to produce more insightful results. In addition, the stored data can also be used for Examination of the relationship between the feature vector and the target variable represented in the charts and graphs on an analytical dashboard. The proposed approach attempts to bridge the gap between academic research and the practical approach by providing a containerized, loosely coupled architectural solution that enables practical implementation in industry.

The figures and the preceding description give examples of embodiments. Those skilled in the art will understand that one or more of the elements described may well be combined into a single functional element. Alternatively, certain elements can be broken down into multiple functional elements. Elements from one embodiment may be added to another embodiment. For example, the order of the processes described herein may be changed and is not limited to the manner described herein. Additionally, the actions of a flowchart need not be performed in the order shown; Also, not all actions have to be carried out. Also, those actions that are not dependent on other actions can be performed in parallel with the other actions. The scope of the embodiments is in no way limited by these specific examples. Numerous variations are possible, regardless of whether they are explicitly mentioned in the description or not, e.g. B. Differences in structure, dimensions and use of materials. The scope of the embodiments is at least as broad as indicated in the following claims.

Advantages, other benefits, and solutions to problems have been described above with respect to particular embodiments. However, the benefits, advantages, problem solutions, and components that can cause an advantage, benefit, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature or component of any or all claims.

Reference List

100

A system for streaming and storing heterogeneous sensor data streams using a microservices architecture.

102

Processing unit for data acquisition.

104

Processing unit for data cleaning.

106

Processing unit for data exploration

108

Processing unit for feature engineering.

110

Predictive modeling processing unit.

112

Processing unit for data visualization.

204

task management

206

Collect

208

Analyze

210

Storage and I/O manager

212

network manager

214

actor system

216

data stream

218

report

220

order management

222

data flow diagram

224 226

Planner test point coordinator

228

Flink program

230

program code

232

Optimizer/Graph Builder

234

client

236

Task Status Heartbeat Statistics

238

Deploy/Stop cancel tasks

240

Trigger checkpoint

242

status update

244

Statistics & Results

246

send order

248

Cancel/Update Order

302

data acquisition layer

304

sensor data

306

Raspberry Pi (data collector)

308

data streaming layer

310

cloud gateway

312

Processor for streaming data

314

data warehouse

316

integration layer

318

Kafka publisher

320

Kafka Subscriber

322

Data Analysis Layer

324

models

326

Machine Learning Algorithm

328

data visualization

330

Data presentation (charts and graphs)

332

Predictive Analysis

334

Results

Claims (7)

A system for streaming and storing the heterogeneous streamed sensor data using a microservices architecture model, the system comprising: a data acquisition processing unit for acquiring the heterogeneous data from different types of sensors, the acquired data being raw and unstructured; a data cleaning processing unit for eliminating the redundant data and for filtering the required data from the raw data collected from heterogeneous sensor data; a data exploration processing unit for performing the analysis of the heterogeneous sensor data of the unstructured data and thereby identifying the data patterns; a feature engineering processing unit to transform the raw data into features, where this data is considered the clean data to perform analysis and the data is kept and then stored in the cloud, and where the features represent the underlying problem in the predictive model ; a predictive modeling processing unit for finding patterns of interest in the collected data by performing predictive analysis; and a data visualization unit that presents the analysis results to the user using various visual elements, such as B. diagrams and tables. The system after claim 1 , where the system uses a microservices-based self-directed architecture with Flink for streaming, which allows the proposed system to achieve scalability and modularity, with some other architectural features being high throughput, scalability, availability, reliability and agility, and maintainability. system after claim 1 , in which a test bench is created for an experimental setup where the environmental data such as temperature and humidity values are streamed hourly via multiple devices to demonstrate the redundant data. The system after claim 3 , using experimental components of the Flink architecture for streaming the data from multiple sensors, Microservices for collecting, analyzing and reporting, Dynamic integration using Apache Kafka as a message queuing system for generating data and then publishing the relevant topics, Analytics for Performing data analysis using a machine learning component algorithm in Python and a dashboard for efficiently reporting the information after performing the analysis, the dashboard showing charts and graphs for representing the results of the predictive analysis. The system after claim 3 , where the data is collected using sensors such as MQ135, DHT11, MQ131, DHT22, where the data is collected using Arduino and Raspberry Pi microcontrollers, which publishes data in the platform using the Think-Speak platform the streaming platform will be integrated using the MQTT Flink bridge, the data will be published in the platform using the Think Speak platform, the streaming platform will be integrated using the MQTT Flink bridge, the Flink streaming -Consumer receives the data and then publishes it to the MongoDB database program using the Kafka integration layer. system after claim 3 , where air quality data telemetry is also streamed from different sensor devices using multiple parameters and this data is combined with respect to the timestamp value, and also where weather data is combined to produce a single data stream for each timestamp. system after claim 1 , which uses a Python-based program for data exploration, where the data is cleaned and pre-processed, removing unwanted and incorrect data, replacing missing data with appropriate values, and extracting only required features for analysis, after which these corrected and cleaned data is uploaded to the cloud for analysis, thus retaining only required data through this process of aggregating and pre-processing data, resulting in a reducing the memory required to store the data.

Images