CN114383295A - Ventilation control method based on indoor pollution concentration estimation model - Google Patents

Abstract

The invention relates to a ventilation control method based on an indoor pollution concentration estimation model. The method comprises the following steps: in the step 1, the method comprises the following steps of,building an indoor air detection system; step 2, collecting PM2.5 and CO collected in the step 1₂Carrying out normalization processing on the smoke signal data; step 3, establishing an indoor pollution concentration estimation model; step 4, establishing an indoor ventilation control algorithm model; step 5, embedding the designed indoor pollution concentration estimation model and the designed indoor ventilation control algorithm model into an air detection data processing module to complete control of indoor ventilation; and 6, detecting the outdoor wind speed and the raindrop value in real time by the indoor air detection data processing module, and closing the ventilation device by the controller when the outdoor wind speed and the raindrop value exceed a limited threshold value. The invention builds indoor PM2.5 and CO₂On the basis of a dynamic estimation model of concentration and smoke signals, PM2.5 and CO are dynamically estimated according to the reward value and the dynamic estimation of the ventilation control quantity₂The control of the window opening and the fan is finished by the concentration and smoke signals, and compared with the traditional mechanical ventilation system, the energy consumption can be saved, and the service life of equipment can be prolonged.

Description

Ventilation control method based on indoor pollution concentration estimation model

Technical Field

The invention relates to the field of indoor pollution, in particular to a ventilation control method based on an indoor pollution concentration estimation model.

Background

With the development of modern science and technology and the improvement of the living quality requirements of people, the functions of modern buildings are more and more perfect, and more human activities can be transferred from outdoor to indoor. Research shows that modern people spend most of their time in a building room, so the quality of indoor air has a very important influence on human health and comfort. In recent years, the problem of indoor air pollution in various regions around the world is frequent, and the problem of indoor air quality is more and more emphasized by people.

Indoor air pollutants are mainly divided into two main categories, solid and gas. The solid pollutants mainly comprise suspended particles, dust, pollen of partial plants, cells of microorganisms, partial smoke and the like; the gas pollutants mainly comprise Volatile Organic Compounds (VOCs), ozone (O3), nitrogen oxides (NOx), sulfur dioxide (SO2), radon gas and the like. The human body exposed to these indoor air pollutants for a long time may suffer from serious respiratory diseases, cardiovascular diseases, skin diseases, cancers, etc., and even death. The invention provides a ventilation control method based on an indoor pollution concentration estimation model, which comprises the steps of establishing an indoor PM2.5, CO2 and smoke concentration dynamic estimation model, and controlling ventilation equipment according to the dynamic estimation model.

Disclosure of Invention

In order to solve the existing problems, the invention collects PM2.5 and CO in a room₂Based on smoke concentration data and outdoor raindrop and wind speed data, a ventilation control method based on an indoor pollution concentration estimation model is provided, and indoor PM2.5 and CO are established₂On the basis of a dynamic estimation model of concentration and smoke signals, PM2.5 and CO are dynamically estimated according to the reward value and the dynamic estimation of the ventilation control quantity₂The control of the window opening and the fan is finished by the concentration and smoke signals, and the method comprises the following specific steps:

step 1, building an indoor air detection system: the indoor air detection system comprises a PM2.5 detection sensor and CO₂The system comprises a detection sensor, a smoke detection sensor, a wind speed detection sensor, a raindrop sensor, a signal conditioning circuit module, a power supply circuit module and an NI acquisition card;

step 2, collecting PM2.5 and CO collected in the step 1₂Normalizing the smoke signal data to obtain PM2.5 and CO₂Normalizing the smoke signal data to be in a range of 0 to 1 so as to reduce dimensional influence among the data;

step 3, establishing an indoor pollution concentration estimation model: normalizing the PM2.5 and CO obtained in the step 2₂The smoke data and the output value of the controller to the actuating mechanism are used as input, and the indoor PM2.5 and CO at the next moment are used as input₂Taking the smoke data as output to establish an indoor pollution concentration estimation model;

step 4, establishing an indoor ventilation control algorithm model: calculating the PM2.5 and CO in the room at the next moment by using the wind speed, the ventilation action and the indoor pollution concentration estimation model₂Smoke data and current indoor PM2.5, CO₂Training a deep certainty strategy gradient network by using smoke data as input, and outputting a ventilation value by using the deep certainty strategy gradient network;

step 5, embedding the designed indoor pollution concentration estimation model and the designed indoor ventilation control algorithm model into an air detection data processing module to complete control of indoor ventilation;

and 6, detecting the outdoor wind speed and the raindrop value in real time by the indoor air detection data processing module, and closing the ventilation device by the controller when the outdoor wind speed and the raindrop value exceed a limited threshold value.

Further, the process of building an indoor air detection system in step 1 can be represented as follows:

PM2.5 detection sensor and CO are respectively detected through a power supply circuit₂The detection sensor, the smoke detection sensor, the wind speed detection sensor and the raindrop sensor are powered, and meanwhile, the sensors are installed at proper positions to detect PM2.5 and CO₂The smoke, wind speed and raindrop signals are converted by the conditioning circuit and then are uniformly connected into the NI acquisition card, and the NI acquisition card transmits the signals into an indoor air detection data processing module of the upper computer through the PCIE interface.

Further, the process of establishing the indoor pollution concentration estimation model in step 3 can be expressed as follows:

step 3.1: the indoor pollution concentration estimation model consists of an input door, a forgetting door, a cell state and an output door; firstly, the input gate selectively memorizes the input, and the forgetting gate selectively forgets the input transmitted by the cell state; secondly, the outputs of the input gate and the forgetting gate are superposed to the next cell state; finally, the output gate scales the output of the cell state to output a predicted value at the next moment;

step 3.2: an input door of an indoor pollution concentration estimation model is constructed, and PM2.5 and CO are filtered₂Smoke data and controller pair actuatorOutputting invalid information of the value, inputting valid information to the cell state, and outputting

Entry is expressed as follows:

i_t＝g(W_xix_t+W_hih_t-1+W_cic_t-1+b_i+W_nin_t (1)

in the formula, x_tIs normalized PM2.5, CO₂Smoke data and data of the output value of the controller to the actuator, g () being the activation function, W_xiIs an input gate and x_tWeight coefficient between, h_t-1Is a hidden state at t-1, W_hiIs the weight coefficient between the input gate and the hidden state, c_t-1Is the cell state at t-1, W_ciIs the weight coefficient between the input gated sample and the cell state, b_iIs an offset term of the input gate, i_tFor input of the gate sample, n_tNoise interference at t, W_niThe weight coefficient between the input gate and the noise interference is used, and the robustness of the model is improved by adding the noise interference;

step 3.3: a forgetting gate of an indoor pollution concentration estimation model is constructed to delete part of invalid information in a cell state, and the forgetting gate is expressed as follows:

f_t＝g(W_xf x_t+W_hf h_t-1+W_cf c_t-1+b_f) (2)

in the formula (f)_tTo forget the gate output, W_xfIs forgetting gate and x_tInter weight coefficient, W_hfIs the weight coefficient between the forgetting gate and the hidden state, W_cfTo forget the gated sample and the cell status c_t-1Inter weight coefficient, b_fIs forgetting

A biasing term for the gate;

step 3.3: constructing a model cell state:

c_t＝f_tc_t-1+i_t tanh(W_xcx_t+W_hch_t-1+b_c) (3)

in the formula, W_xcIs a cellular stateAnd x_tWeight coefficient between, W_hcIs a cell state and a hidden state h_t-1Inter weight coefficient, b_cA bias term for a cellular state;

step 3.4: constructing a model output door and outputting indoor PM2.5 and CO at the next moment₂Smoke data:

s_t＝g(W_xs x_t+W_hsh_t-1+W_csc_t+b_s) (4)

in the formula, s_tIs the PM2.5 and CO output by the output gate at the next moment₂Smoke normalization data, W_xsIs an output gate and x_tInter weight coefficient, W_hsIs a sample of the output gate and h_t-1Weight coefficient between, W_csIs a sample of the output gate and c_tWeight coefficient of b_sBeing offset terms of output gates

Step 3.5: hidden state h to t moment through output gate_tUpdating:

h_t＝s_t tanh(c_t) (5)

and 3.6, repeating the step 3.2 to the step 3.5, and updating the weight and the bias parameters in the model by using the SGD algorithm until the loss function reaches a convergence threshold value which is set to be 1 e-4.

Further, the process of establishing the indoor ventilation control algorithm model in step 4 can be represented as follows:

step 4.1, the indoor ventilation control algorithm model comprises a strategy network and a value network, wherein the value network target is to select an optimal action value, the value network updates network parameters by taking minimization of a loss function as a target, and the strategy network target is to select a current action according to a current state and generate a next moment action according to an environment state;

step 4.2, the strategy network and the value network respectively comprise a real network and a target network, the strategy network and the value network are respectively initialized, and the parameters of the strategy network and the value network are randomly initialized: theta^Q、θ^μAnd order:

θ^Q′＝θ^Q (6)

θ^μ′＝θμ (7)

in the formula, theta^QRepresenting a realistic value network parameter, θ^Q′Representing a target value network parameter, theta^μRepresenting a realistic policy network parameter, θ^μ′Representing target policy network parameters

Step 4.3, selecting the actuator action a under the current state_t：

a_t＝μ(s_t|θ^μ)+N_t (8)

In the formula, N_tIs random noise, and the function mu () is an optimal behavior strategy;

step 4.4, by performing action a_tReceive a reward r_t+1And state s estimated by indoor pollution concentration estimation model_t+1Saving the current state s_tAnd action a_tPrize r_t+1And state s_t+1And putting the mixture into an experience pool; ,

step 4.5, when the experience pool data reaches the training condition, randomly selecting N samples from the experience pool(s)_t，a_t，r_t+1，s_t+1)}；

Step 4.6, utilizing the samples in the experience pool to pair the parameters theta of the strategy network and the value network^Q、θ^μUpdating, and calculating a target action value y through the value network:

y＝r_t+1+γQ(s_t+1，a_t+1/θ^Q′) (9)

gamma is a discount factor, a_t+1Is the action at time t + 1;

step 4.7, the target network is updated once at intervals:

θ^Q′＝τθ^Q+(1-τ)θ^Q′ (10)

θ^μ′＝τθ^μ+(1-τ)θ^μ′ (11)

in the formula, tau is a learning rate;

step 4.8, returning to the step 4.3 to carry out loop iteration, and knowing the maximum number of rounds;

step 4.9, outputting the ventilation control quantity a according to the trained indoor ventilation control algorithm model_t。

The ventilation control method based on the indoor pollution concentration estimation model has the advantages that:

according to the invention, on the basis of establishing a dynamic estimation model of the concentrations of PM2.5 and CO2 and smoke signals in a room, the control of the window opening and the fan is completed according to the reward value of the ventilation control quantity and the dynamic estimation of the concentrations of PM2.5 and CO2 and the smoke signals, and compared with the traditional mechanical ventilation system, the energy consumption can be saved, and the service life of equipment can be prolonged;

according to the invention, the outdoor wind speed and the raindrop value are detected in real time, and when the outdoor wind speed and the raindrop value exceed the limited threshold value, the controller closes the ventilation device, so that the safe operation of the ventilation system is ensured;

in order to increase the self-adaption and anti-interference capability of a ventilation system, the invention designs a new ventilation control method based on an indoor concentration estimation model, controls a system actuator by utilizing the on-line weight learning capability, and can automatically control the uncertainty and the interference of the ventilation system;

the invention provides an important technical means for an indoor pollution ventilation control method.

Drawings

FIG. 1 is a flow chart of the present invention;

FIG. 2 is a model diagram of an indoor ventilation control algorithm of the present invention.

Detailed Description

The invention provides a ventilation control method based on an indoor pollution concentration estimation model, aims to improve the indoor air quality and the running stability and the service life of ventilation control equipment, and provides a principle and a prediction framework based on an indoor pollution concentration estimation algorithm and a ventilation control algorithm. FIG. 1 is a flow chart of the present invention, which is further described below in conjunction with the accompanying drawings and detailed description:

step 1, building an indoor air detection system: indoor air detection system is detected by PM2.5Sensor, CO₂The system comprises a detection sensor, a smoke detection sensor, a wind speed detection sensor, a raindrop sensor, a signal conditioning circuit module, a power supply circuit module and an NI acquisition card;

PM2.5 detection sensor and CO are respectively detected through a power supply circuit₂The detection sensor, the smoke detection sensor, the wind speed detection sensor and the raindrop sensor are powered, and meanwhile, the sensors are installed at proper positions to detect PM2.5 and CO₂The smoke, wind speed and raindrop signals are converted by the conditioning circuit and then are uniformly connected into the NI acquisition card, and the NI acquisition card transmits the signals into an indoor air detection data processing module of the upper computer through the PCIE interface.

Step 2, collecting PM2.5 and CO collected in the step 1₂Normalizing the smoke signal data to obtain PM2.5 and CO₂Normalizing the smoke signal data to be in a range of 0 to 1 so as to reduce dimensional influence among the data;

step 3, establishing an indoor pollution concentration estimation model: normalizing the PM2.5 and CO obtained in the step 2₂The smoke data and the output value of the controller to the actuating mechanism are used as input, and the indoor PM2.5 and CO at the next moment are used as input₂Taking the smoke data as output to establish an indoor pollution concentration estimation model;

step 3.1: the indoor pollution concentration estimation model consists of an input door, a forgetting door, a cell state and an output door; firstly, the input gate selectively memorizes the input, and the forgetting gate selectively forgets the input transmitted by the cell state; secondly, the outputs of the input gate and the forgetting gate are superposed to the next cell state; finally, the output gate scales the output of the cell state to output a predicted value at the next moment;

step 3.2: an input door of an indoor pollution concentration estimation model is constructed, and PM2.5 and CO are filtered₂Smoke data and invalid information of the output value of the controller to the actuator, and inputting valid information to the cell state, the input gate being expressed as follows:

i_t＝gW_xix_t+W_hih_t-1+W_cic_t-1+b_i+W_nin_t (I)

in the formula, x_tIs normalized PM2.5, CO₂Smoke data and data of the output value of the controller to the actuator, g () being the activation function, W_xiIs an input gate and x_tWeight coefficient between, h_t-1Is a hidden state at t-1, W_hiIs the weight coefficient between the input gate and the hidden state, c_t-1Is the cell state at t-1, W_ciIs the weight coefficient between the input gated sample and the cell state, b_iIs an offset term of the input gate, i_tFor input of the gate sample, n_tNoise interference at t, W_niThe weight coefficient between the input gate and the noise interference is used, and the robustness of the model is improved by adding the noise interference;

step 3.3: a forgetting gate of an indoor pollution concentration estimation model is constructed to delete part of invalid information in a cell state, and the forgetting gate is expressed as follows:

f_t＝g(W_xf x_t+W_hf h_t-1+W_cf c_t-1+b_f) (2)

in the formula (f)_tTo forget the gate output, W_xfIs forgetting gate and x_tInter weight coefficient, W_hfIs the weight coefficient between the forgetting gate and the hidden state, W_cfTo forget the gated sample and the cell status c_t-1Inter weight coefficient, b_fIs a biased term for a forgetting gate;

step 3.3: constructing a model cell state:

c_t＝f_t c_t-1+i_t tanh(W_xc x_t+W_hch_t-1+b_c) (3)

in the formula, W_xcIs the cell state and x_tWeight coefficient between, W_hcIs a cell state and a hidden state h_t-1Inter weight coefficient, b_cA bias term for a cellular state;

step 3.4: constructing a model output door and outputting indoor PM2.5 and CO at the next moment₂Smoke data:

s_t＝g(W_xs x_t+W_hs h_t-1+W_cs c_t+b_s) (4)

in the formula, s_tIs the PM2.5 and CO output by the output gate at the next moment₂Smoke normalization data, W_xsIs an output gate and x_tInter weight coefficient, W_hsIs a sample of the output gate and h_t-1Weight coefficient between, W_csIs a sample of the output gate and c_tWeight coefficient of b_sBeing offset terms of output gates

Step 3.5: hidden state h to t moment through output gate_tUpdating:

h_t＝s_t tanh(c_t) (5)

and 3.6, repeating the step 3.2 to the step 3.5, and updating the weight and the bias parameters in the model by using the SGD algorithm until the loss function reaches a convergence threshold value which is set to be 1 e-4.

Step 4, establishing an indoor ventilation control algorithm model: calculating the PM2.5 and CO in the room at the next moment by using the wind speed, the ventilation action and the indoor pollution concentration estimation model₂Smoke data and current indoor PM2.5, CO₂Training a deep certainty strategy gradient network by using smoke data as input, and outputting a ventilation value by using the deep certainty strategy gradient network;

the indoor ventilation control algorithm model comprises a strategy network and a value network, wherein the value network target is to select an optimal action value, the value network updates network parameters by taking minimization of a loss function as a target, the strategy network target is to select a current action according to a current state and generate a next moment action according to an environment state, and an indoor ventilation control algorithm model diagram is shown in FIG. 2;

step 4.1, the strategy network and the value network respectively comprise a real network and a target network, the strategy network and the value network are respectively initialized, and the parameters of the strategy network and the value network are randomly initialized: theta^Q、θ^μAnd order:

θ^Q′＝θ^Q (6)

θ^μ′＝θμ (7)

in the formula, theta^QRepresenting a realistic value network parameter, θ^Q′Representing a target value network parameter, theta^μRepresenting a realistic policy network parameter, θ^μ′Representing target policy network parameters

Step 4.2, selecting the actuator action a under the current state_t：

a_t＝μ(s_t|θ^μ)+N_t (8)

In the formula, N_tIs random noise, and the function mu () is an optimal behavior strategy;

step 4.3, by performing action a_tReceive a reward r_t+1And state s estimated by indoor pollution concentration estimation model_t+1Saving the current state s_tAnd action a_tPrize r_t+1And state s_t+1And putting the mixture into an experience pool; ,

step 4.4, when the experience pool data reaches the training condition, randomly selecting N samples from the experience pool(s)_t，a_t，r_t+1，s_t+1)}；

Step 4.5, utilizing the samples in the experience pool to pair the parameters theta of the strategy network and the value network^Q、θ^μUpdating, and calculating a target action value y through the value network:

y＝r_t+1+γQ(s_t+1，a_t+1/θ^Q′) (9)

gamma is a discount factor, a_t+1Is the action at time t + 1;

step 4.6, the target network is updated once at intervals:

θ^Q′＝τθ^Q+(1-τ)θ^Q′ (10)

θ^μ′＝τθ^μ+(1-τ)θ^μ′ (11)

in the formula, tau is a learning rate;

step 4.7, returning to the step 4.2 to carry out loop iteration, and knowing the maximum number of rounds;

step 4.8, outputting the ventilation control quantity a according to the trained indoor ventilation control algorithm model_t。

Step 5, embedding the designed indoor pollution concentration estimation model and the designed indoor ventilation control algorithm model into an air detection data processing module to complete control of indoor ventilation;

and 6, detecting the outdoor wind speed and the raindrop value in real time by the indoor air detection data processing module, and closing the ventilation device by the controller when the outdoor wind speed and the raindrop value exceed a limited threshold value.

The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any way, but any modifications or equivalent variations made according to the technical spirit of the present invention are within the scope of the present invention as claimed.

Claims (4)

1. The ventilation control method based on the indoor pollution concentration estimation model comprises the following specific steps, and is characterized in that:

step 1, building an indoor air detection system: the indoor air detection system comprises a PM2.5 detection sensor and CO₂The system comprises a detection sensor, a smoke detection sensor, a wind speed detection sensor, a raindrop sensor, a signal conditioning circuit module, a power supply circuit module and an NI acquisition card;

step 2, collecting PM2.5 and CO collected in the step 1₂Normalizing the smoke signal data to obtain PM2.5 and CO₂Normalizing the smoke signal data to be in a range of 0 to 1 so as to reduce dimensional influence among the data;

step 3, establishing an indoor pollution concentration estimation model: normalizing the PM2.5 and CO obtained in the step 2₂The smoke data and the output value of the controller to the actuating mechanism are used as input, and the indoor PM2.5 and CO at the next moment are used as input₂Taking the smoke data as output to establish an indoor pollution concentration estimation model;

step 4, establishing an indoor ventilation control algorithm model: calculating the PM2.5 and CO in the room at the next moment by using the wind speed, the ventilation action and the indoor pollution concentration estimation model₂Smoke data and current indoor PM2.5, CO₂Training a deep certainty strategy gradient network by using smoke data as input, and outputting a ventilation value by using the deep certainty strategy gradient network;

step 5, embedding the designed indoor pollution concentration estimation model and the designed indoor ventilation control algorithm model into an air detection data processing module to complete control of indoor ventilation;

and 6, detecting the outdoor wind speed and the raindrop value in real time by the indoor air detection data processing module, and closing the ventilation device by the controller when the outdoor wind speed and the raindrop value exceed a limited threshold value.

2. The ventilation control method based on the indoor pollution concentration estimation model according to claim 1, wherein: the process of building an indoor air detection system in step 1 can be represented as follows:

PM2.5 detection sensor and CO are respectively detected through a power supply circuit₂The detection sensor, the smoke detection sensor, the wind speed detection sensor and the raindrop sensor are powered, and meanwhile, the sensors are installed at proper positions to detect PM2.5 and CO₂The smoke, wind speed and raindrop signals are converted by the conditioning circuit and then are uniformly connected into the NI acquisition card, and the NI acquisition card transmits the signals into an indoor air detection data processing module of the upper computer through the PCIE interface.

3. The ventilation control method based on the indoor pollution concentration estimation model according to claim 1, wherein: further, the process of establishing the indoor pollution concentration estimation model in step 3 can be expressed as follows:

step 3.1: the indoor pollution concentration estimation model consists of an input door, a forgetting door, a cell state and an output door; firstly, the input gate selectively memorizes the input, and the forgetting gate selectively forgets the input transmitted by the cell state; secondly, the outputs of the input gate and the forgetting gate are superposed to the next cell state; finally, the output gate scales the output of the cell state to output a predicted value at the next moment;

step 3.2: construction of indoor pollution concentrationInput gate of degree estimation model, filtering PM2.5, CO₂Smoke data and invalid information of the output value of the controller to the actuator, and inputting valid information to the cell state, the input gate being expressed as follows:

i_t＝g(W_xix_t+W_hih_t-1+W_cic_t-1+b_i)+W_nin_t (1)

in the formula, x_tIs normalized PM2.5, CO₂Smoke data and data of the output value of the controller to the actuator, g () being the activation function, W_xiIs an input gate and x_tWeight coefficient between, h_t-1Is a hidden state at t-1, W_hiIs the weight coefficient between the input gate and the hidden state, c_t-1Is the cell state at t-1, W_ciIs the weight coefficient between the input gated sample and the cell state, b_iIs an offset term of the input gate, i_tFor input of the gate sample, n_tNoise interference at t, W_niThe weight coefficient between the input gate and the noise interference is used, and the robustness of the model is improved by adding the noise interference;

step 3.3: a forgetting gate of an indoor pollution concentration estimation model is constructed to delete part of invalid information in a cell state, and the forgetting gate is expressed as follows:

f_t＝g(W_xfx_t+W_hfh_t-1+W_cfc_t-1+b_f) (2)

in the formula (f)_tTo forget the gate output, W_xfIs forgetting gate and x_tInter weight coefficient, W_hfIs the weight coefficient between the forgetting gate and the hidden state, W_cfTo forget the gated sample and the cell status c_t-1Inter weight coefficient, b_fIs a biased term for a forgetting gate;

step 3.3: constructing a model cell state:

c_t＝f_tc_t-1+i_ttanh(W_xcx_t+W_hch_t-1+b_c) (3)

in the formula, W_xcIs the cell state andx_tweight coefficient between, W_hcIs a cell state and a hidden state h_t-1Inter weight coefficient, b_cA bias term for a cellular state;

step 3.4: constructing a model output door and outputting indoor PM2.5 and CO at the next moment₂Smoke data:

s_t＝g(W_xsx_t+W_hsh_t-1+W_csc_t+b_s) (4)

in the formula, s_tIs the PM2.5 and CO output by the output gate at the next moment₂Smoke normalization data, W_xsIs an output gate and x_tInter weight coefficient, W_hsIs a sample of the output gate and h_t-1Weight coefficient between, W_csIs a sample of the output gate and c_tWeight coefficient of b_sBeing offset terms of output gates

Step 3.5: hidden state h to t moment through output gate_tUpdating:

h_t＝s_ttanh(c_t) (5)

and 3.6, repeating the step 3.2 to the step 3.5, and updating the weight and the bias parameters in the model by using the SGD algorithm until the loss function reaches a convergence threshold value which is set to be 1 e-4.

4. The ventilation control method based on the indoor pollution concentration estimation model according to claim 1, wherein: the process of establishing the indoor ventilation control algorithm model in step 4 can be represented as follows:

step 4.1, the strategy network and the value network respectively comprise a real network and a target network, the strategy network and the value network are respectively initialized, and the parameters of the strategy network and the value network are randomly initialized: theta^Q、θ^μAnd order:

θ^Q′＝θ^Q (6)

θ^μ′＝θ^μ (7)

in the formula, theta^QRepresenting realistic value network parameters，θ^Q′Representing a target value network parameter, theta^μRepresenting a realistic policy network parameter, θ^μ′Representing target policy network parameters

Step 4.2, selecting the actuator action a under the current state_t：

a_t＝μ(s_t|θ^μ)+N_t (8)

In the formula, N_tIs random noise, and the function mu () is an optimal behavior strategy;

step 4.3, by performing action a_tReceive a reward r_t+1And state s estimated by indoor pollution concentration estimation model_t+1Saving the current state s_tAnd action a_tPrize r_t+1And state s_t+1And putting the mixture into an experience pool; ,

step 4.4, when the experience pool data reaches the training condition, randomly selecting N samples from the experience pool(s)_t，a_t，r_t+1，s_t+1)}；

Step 4.5, utilizing the samples in the experience pool to pair the parameters theta of the strategy network and the value network^Q、θ^μUpdating, and calculating a target action value y through the value network:

y＝r_t+1+γQ(s_t+1，a_t+1/θ^Q′) (9)

gamma is a discount factor, a_t+1Is the action at time t + 1;

step 4.6, the target network is updated once at intervals:

θ^Q′＝τθ^Q+(1-τ)θ^Q′ (10)

θ^μ′＝τθ^μ+(1-τ)θ^μ′ (11)

in the formula, tau is a learning rate;

step 4.7, returning to the step 4.2 to carry out loop iteration, and knowing the maximum number of rounds;

step 4.8, outputting the ventilation control quantity a according to the trained indoor ventilation control algorithm model_t。

Images