AU2019100364A4

AU2019100364A4 - A Method of Air Quality Prediction Using Long Short-Term Memory Neural Network

Info

Publication number: AU2019100364A4
Application number: AU2019100364A
Authority: AU
Inventors: Shenyuan Huang; Zuotao Li; Ruolin Sun; Tianyue Xu; Xiaogang Yang; Jingqiao Zhu
Original assignee: Individual
Current assignee: Individual
Priority date: 2019-04-05
Filing date: 2019-04-05
Publication date: 2019-05-09
Anticipated expiration: 2027-04-05

Abstract

Air quality prediction is an effective way used to protect the citizens'health by providing an early warning of the atmospheric pollutants. Existing atmospheric pollutant concentration prediction models are unable to simulate the long-term relationship effectively and always neglect the spatio-temporal correlations. In this application, a novel long short-term memory neural network enhanced model (LSTME) which considers the spatiotemporal correlations between observation stations is proposed for accurate atmospheric pollutant concentration prediction. In this application, we used the long short-term memory (LSTM) layers to automatically extract inherent representative characteristics from the historical atmospheric pollutant data, and the meteorological data is integrated into the proposed model to enhance the prediction performance. Experiments using the PM2.5 hourly data collected at 12 air quality monitoring stations in Beijing city, China during the period from Jan/01/2014 to Jan/01/2015 proved the effectiveness of our proposed LSTME model. Compared with the state-of-the-art statistics-based prediction models, including LSTM and Time-Delayed Artificial Neural Network (TDNN), the results demonstrated that our LSTME model is better than other statistics-based models. The proposed model exhibited a satisfying performance on the one-hour prediction task, with an error of 11.93%. time t-2 °•.0 t-r .-. S1 S2 511 58 Station historical PMz.5 concentration data Long Short-term Neural Network Fully Connected Neural Network Fully Connected Neural Network Q Q(C) C time: t ... Figure 1

Description

LIELD OE INVENTION

The application is a device used to effectively predict the short-term air pollutant concentration. This application can bring great help to air quality prediction and monitoring. It can provide the citizens and the government with the data of future air pollutant concentration. It provides people with the reference of air quality so that people can take preventive and protective measures in advance for the haze weather. It also provides researchers in environmental engineering with complete spatial and temporal reference data.

BACKGROUND OF THE INVENTION

Accompany with the rapid development of the economy, air pollution has become a globally environmental problem. As air pollution has caused great damage to the health of residents, the problem of air pollution has become a major concern of the government and the public. Take PM2.5 as an example, it is a particle with a diameter of fewer than 2.5 microns

2019100364 05 Apr 2019 could directly affect the pulmonary ventilation function after entering the human body into the alveoli. There has been seriously healthy condition resulted from such pollutants which turn out to be a problem eager to be solved. Thus, it would be helpful when we are able to acquire the real-time concentration of the air pollutant.

There are already have some solutions in the prediction of air pollutants concentration. Most of them can be classified into the following two methods, but they both have its shortcomings.

1. Deterministic methods: These methods are based on atmosphere physical and chemical reactions, and the processes of pollutant diffusion, emission, transformation, removal and diffusion are modelled by using meteorological principles and statistical methods. Although sound theories support the pollutant diffusion mechanism, almost all of these models are related to empirical knowledge, unreliable and limited data, and various application limitations.

2.Statistical method. This kind of methods do not require complex theoretical models and can predict air quality based on statistical models. Traditional RNN models which can handle arbitrary sequences of inputs

2019100364 05 Apr 2019 still have two shortcomings: 1) RNN structure requires a lot of experiments to determine the optimal time lag ahead of time;2) the traditional RNNs cannot capture the long-time dependence in the input sequence, so it is difficult to train the RNNs with a long time delay, and it may encounter the problem of gradient disappearances and explosion.

In 1997, scientists developed a special RNN architecture called a long-short-memory neural network (LSTM NN) in order to handle these problems. Different from traditional neural networks, LSTM NNs have the ability to model learn long time series but are not affected by the vanishing gradient problem. This characteristic is particularly essential for the establishment of a spatiotemporal process model of the concentration of air pollutants in one monitoring station which is highly related to the historical concentration of air pollutants in this station and the concentration of air pollutants in nearby monitoring stations due to the pollutant transport process. Recent reseraches have tried to use LSTM NNs for air pollution risk prediction, but they only classified the risk ranking of pollution and did not predict the concentration of air pollutants by real values. In addition, they did not take into account the spatial correlation between monitoring stations but made separate predictions for each station. As far as we know, LSTM NNs have not been applied in the

2019100364 05 Apr 2019 field of air pollutant concentration prediction. In this application, LSTM NNs are extended to build a model for air pollutant concentration prediction and spatiotemporal correlation are effectively modeled.

SUMMARY OF THE INVENTION

In order to forecast future air quality, we proposed a method based on the deep learning structure. In our proposal, the main network was Long short-term memory (LSTM) and fully connected one (FC), which can be seen in Fig. 1. In our LSTME method, instead of using the historical air pollutant concentration data alone, our method used extra data to support the prediction process. The inputs can be classified as main inputs historical PM2.5 concentration data, and auxiliary inputs which includes meteorology (such as temperature, humidity, wind speed, visibility and etc.) Main inputs should be in a normalized format while the supplement data also went through the preprocessing process. Our experiments showed that this proposal can effectively predict future air quality.

A brief introduction to LSTM layer

The LSTM NN is generated from RNN. The basic structure consists of two layers serve as the input and output, respectively, several recurrently

2019100364 05 Apr 2019 connected hidden layers take the role of memory blocks. Those memory blocks can be used in solving time series problems, their inner structure (self-recurrent memory cell and recurrently self-connected linear unit-constant error carousel) can somehow memorize and make use of the previous status.

Fully connected (FC) layer

The FC is the mostly used building block in a neural network, in each layer, each node is connected with all nodes in the preceding layer. This network can be used in repeated supervisor learning and obtains a good performance in the data feature extraction stage.

In our proposal, there are two LSTM layers which served as the feature extractor for air pollutant concentration data. One FC layer was followed, its aim was the combination of features extracted by LSTM and supplement input, then generated the final output.

The global operation will be shown below.

Auxiliary data.

Before entering the FC layer, those auxiliary data will be preprocessed with normalization.

The Overall pipeline of our proposed method for air pollutant concentration prediction includes three parts.

2019100364 05 Apr 2019 .Main inputs enter LSTM network, then the network tries to extract the inherent features from the inputs.

2. Extracted features are combined with the auxiliary inputs and enter the FC network.

3. FC layer generates the final prediction output.

After training our network using enough training data, our model can directly produce air pollutant concentration for several stations simultaneously with a high prediction accuracy and inference speed.

DESCRIPTION OF THE DRAWINGS

Fig.1 illustrates the architecture of our proposed FSTMEM for air pollutant concentration prediction.

Fig.2 illustrates the overall pipeline for air pollutant concentration prediction.

Fig.3 illustrates the auto-correlation with different time delays.

Fig.4 illustrates the distributions of the 8 air quality monitoring stations (red triangle) and the meteorological monitoring station (blue triangle) in Beijing City, China.

Fig. 5 illustrates the predicted and observed PM2.5 concentration value.

2019100364 05 Apr 2019

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Network architecture

There are also parameters which were capable to influence the whole prediction process in other parts, those parameters can be grouped as: Number of Layers, which refers to LSTM layers and full-connected layers; Number of nodes, which refers to single LSTM layer and single full-connected layer; the time lag, which refers to the normalized historical PM2.5 concentration data; the learning rate, generally determined by the activated function chose in neural network, in our method, this function was set as ReLU. The number of layers had already been set to 2 (LSTM) and 1(FC), respectively. The number of nodes was set to 500. In order to optimize the number of time lags, we designed several comparison experiments, related parameters were well controlled, including the input data and other potential variables. The historical time lag r was set from 8 to 12, with an increment 2. The overall result showed that when time lag equaled to 10, our method reached its best performance.

Spatiotemporal correlation analysis

2019100364 05 Apr 2019

The Spatial Correlation

First, we analyzed PM2.5 concentrations among the all air quality monitoring stations in order to examine the spatial correlation which was measured by Pearson’s correlation coefficient (Pearson, 1895). As shown in Table, it significantly indicated that all the correlation coefficients between stations are above 0.8 (p-value<0.05) which indicated that the PM2.5 concentrations are closely correlated among the stations. This makes it possible for us to use a single model to predict the PM2.5 concentrations among these stations instead of using separate models for each station, and thus can also contribute to the improvement of the prediction performance by incorporating nearby related inputs.

The Autocorrelation

Subsequently, autocorrelation functions (Box and Jenkins, 1976) were measurements to calculate the temporal correlations in the PM2.5 concentration time series of each station. With regard to the time delay k, the temporal correlations were calculated as the following formula:

Cov(y(t),y(t + k))

P/c —°y(t)°y(t+k) y(t) denotes the air pollutant concentration at time t and y(t + k) denotes the air pollutant concentration at time t + k. Cov(·) is the covariance and σ(·) is the standard deviation.

2019100364 05 Apr 2019

According to Fig. 3, it shows the autocorrelation coefficients of each station and the 8 curves. A significant descending trend with increasing time lags indicates that earlier events have a weaker effect on the current status. Moreover, the correlation function is higher than 0.5 when the time lag is less than 20 indicating a high temporal correlation. We can select the appropriate time lags for the prediction tasks based on the correlation values.

In consideration of the high spatiotemporal correlations and the historical status of these stations, we can regard the time-delayed PM2.5 concentrations as inputs and applied the FSTM NN to confirm the spatiotemporal correlations.

Spatio-temporal feature extraction

First, an input tensor for the FSTM layers was made up of time-delayed historical data which had already been stacked (see “Main Inputs” in Fig. 1). Secondly, features of the spatially correlated data with long time dependencies were automatically extracted layer-by-layer (see “FSTMs” part in Fig. 1). A recursive arrow denotes that there are many times of the layer extraction process so as to optimum performance.

Supplement data processing

The addition of supplement data is another significant way to improve

2019100364 05 Apr 2019 prediction performance in addition to the modeling spatiotemporal correlations which are discussed above. According to previous studies, it is known that meteorological factors are indispensable parts of the daily variability of pollutants. Now, the use of current meteorological data can enhance the LSTM NN model.

Performance Inspection

After evaluating the prediction performance of our model under different network configuration, the optimum hyper-parameters have been determined. Comparison experiments between our proposal and LSTM NN model, time-delayed neural network (TDNN) model. Three indicators including RMSE, MAE and MAPE are used to evaluate the prediction performance, calculated as follows, m

RMSE = Τ V 7(Pi - y_ty m

i=l m

MAE= iV \_Pi-y_t\ m i=l m

ι v \Pi - yt\

MAPE = — > ——— mL-ι y_t i=l where m denotes the number of observation record, _Pi and y_t denote the predicted and observed air pollutant concentration at time stamp i, io

2019100364 05 Apr 2019 respectively.

The overall pipeline for air pollutant concentration prediction is shown in Fig. 2.

Experimental data

Main input: We collected hourly PM2.5 concentration data from 8 air quality monitoring stations in Beijing, China as the main input of our proposed method for evaluation. The distribution of the observation stations is shown in Fig. 4. Then we normalized the PM2.5 concentration data for our experiments using maximum-minimum normalization.

Auxiliary data: For auxiliary input, considering the meteorological knowledge and the influence of different meteorological factors to the process of pollutant diffusion, emission, transformation, we added auxiliary data to enhance our proposed model, including temperature, humidity, wind speed.

Network configuration

In our experiments, we tested different network configurations and find optimal hyper-parameters. Our model reaches the best performance when we use two LSTM layers for spatio-temporal air quality feature learning and one fully connected layer for auxiliary data feature embedding, along

2019100364 05 Apr 2019 with the time delays set to 10.

Experimental result

We compared the prediction performance of our proposal with LSTM NN model and time-delayed neural network (TDNN) model. The comparing experiments were conducted using the same training and test dataset. Table 1 shows the prediction performance of our proposed LSTME method and the comparing methods. As shown in Table 1, two LSTM based methods (LSTMNN and LSTME) got higher prediction performance as indicated by all evaluation metrics. This finding suggests that LSTM is better at capturing spatio-temporal correlations within air pollutant concentration data. Table 1 also proves that with auxiliary data, our proposed LSTME method performs better than the vanilla LSTM method, which demonstrates that the selected meteorological data can effectively boost the prediction performance.

Figure 5 shows the predicted and observed PM₂.5 concentrations. As shown in Figure 5, the predicted PM2.5 concentrations are statistically consistent with the observed values, which demonstrate the satisfying prediction performance of our model. The R² value between the predicted PM2.5 concentrations and the observed PM2.5 concentrations showed that

98.7% of the explanations variance was captured by our model.

2019100364 05 Apr 2019

Table 1. Prediction performance of our proposed LSTME method and the comparing methods.

Models	RMSE	MAE	MAPE(%)
LSTME	13.28	6.30	11.06
LSTMNN	16.19	8.18	15.29
STNN	23.75	12.83	22.42

Claims

1. A method of air quality prediction using long short-term memory neural network, is characterized in that the proposed model adopts the long shortterm memory neural network to extract high-level features from the input atmospheric pollutant concentrations data, which can effectively capture the long-term spatiotemporal dependency

2. The method according to claim 1, wherein the proposed method is able to extract the spatio-temporal interdependency in atmospheric pollutant concentration data with great efficiency and automation.

3. The method according to claim 1, wherein the highly related auxiliary data are embedded into a traditional LSTM model and the integrated model can perform better prediction than LSTM model alone.