CN111637614A - Intelligent control method for active ventilation floor in data center - Google Patents

Abstract

The intelligent control method of the active ventilation floor of the data center, establishes a Markov decision process model for the hotspot problem of the data center rack, and provides three model solving algorithms, including the basic intelligent algorithm, the sample value variant intelligence algorithm and the structural variant intelligence algorithm , respectively as the core of the active ventilation floor control algorithm. The model consists of four parts: system state, behavior, reward and value function. The solution of the model is that the optimal behavior is continuously selected under a series of system states to maximize the cumulative reward of the system. The active ventilation floor control algorithm, through continuous exploration and Learn the complex relationship between the temperature distribution of the rack air inlet and the fan speed of the active ventilation floor. Finally, according to the temperature distribution of the rack air inlet, the optimal PWM signal duty cycle value can be generated, and the speed of the active ventilation floor fan can be adjusted so that the temperature of the rack air inlet can be adjusted. Uniform distribution to alleviate rack hotspot issues. Compared with other solutions, the present invention is more universal, easier to deploy, and more cost-effective.

Description

Intelligent control method for active ventilation floor in data center

technical field

The invention belongs to the technical field of automatic control, and particularly relates to an intelligent control method for an active ventilation floor of a data center.

Background technique

Rack hotspots are high-temperature spots where the temperature of one or several locations in the data center rack is significantly higher than that of other locations. Excessive temperatures can cause some servers in the data center to work less efficiently, thereby reducing their overall power density and reducing their reliability, which is obviously contrary to the needs of the data center.

Using global regulation to alleviate or eliminate rack hot spots, such as increasing the power of the air conditioner in the equipment room to provide sufficient cooling air, will inevitably lead to excessive cooling in most of the rack areas. While causing waste of cooling resources, the data center can always be The cooling energy consumption, which accounts for nearly half of the consumption, is even more huge. Therefore, rack-level cooling solutions are more suitable for alleviating rack hotspot issues.

Rack-level cooling solutions exist, such as installing adaptive ventilation floors, installing baffles, and enclosing and ducting individual racks. However, these solutions are all "passive" cooling solutions, which cannot actively provide cold airflow to the racks. When the cooling air supply is insufficient, these solutions are powerless.

As another rack-level cooling solution, active ventilation floors can alleviate the problem of rack hot spots by actively transporting cold air. Compared with the above solutions, it is easier to deploy and more cost-effective, but the difficulty of its control mainly lies in the placement environment. Diversity and dynamism, such as different computer room air conditioners, relative positions of racks, and server distribution inside the rack; different closed states of cold and hot aisles, different server rack standards and sealing conditions; computer room air conditioner power, heat load of servers in different racks different, wait. Therefore, the thermal energy efficiency and airflow models of data centers are generally difficult to describe with analytical models.

Most of the existing active ventilation floor related research is based on measurement or simulation performance modeling and evaluation, and there is no research literature on the control problem of active ventilation floor.

SUMMARY OF THE INVENTION

In order to overcome the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide an intelligent control method for an active ventilation floor of a data center, which can automatically learn the optimal operation strategy, plan the air flow of the rack, and make The temperature distribution of the air inlet of the rack is uniform, which alleviates the problem of rack hot spots. And there is no need to build and calibrate complex airflow and heat exchange models, thereby improving the universality of active ventilation floors.

In order to achieve the above object, the technical scheme adopted in the present invention is:

The intelligent control method of the active ventilation floor of the data center, establishes a Markov decision process model for the hotspot problem of the data center rack, and provides three model solving algorithms, including the basic intelligent algorithm, the sample value variant intelligence algorithm and the structural variant intelligence algorithm , respectively as the core of the active ventilation floor control algorithm. The model consists of four parts: system state, behavior, reward, and value function. The solution of the model is that the optimal behavior is continuously selected under a series of system states to maximize the cumulative reward of the system. The active ventilation floor controls The algorithm, by continuously exploring and learning the complex relationship between the temperature distribution of the rack air inlet and the fan speed of the active ventilation floor, can finally generate the optimal PWM signal duty cycle value according to the temperature distribution of the air inlet of the rack, and adjust the fan speed of the active ventilation floor. The temperature distribution of the air inlet of the rack is uniform, and the hot spot problem of the rack is alleviated.

Compared with the prior art, the beneficial effects of the present invention are:

The present invention does not need to establish and calibrate complex airflow and heat exchange models, uses intelligent control algorithms, overcomes the diversity and dynamics of the placement environment of the active ventilation floor, automatically matches the temperature distribution of the air inlet of the rack and the optimal active ventilation floor fan speed, only It is necessary to replace the original ordinary ventilation floor with the active ventilation floor running the present invention, and the present invention can operate autonomously, improve the temperature distribution of the air inlet of the rack, and alleviate the problem of hot spots of the rack. Compared with other solutions, the present invention has higher universality. Easier to deploy and more cost-effective.

Description of drawings

Figure 1 is an active ventilation floor design and deployment diagram.

Detailed ways

The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples.

1 is a schematic diagram of the detailed deployment implementation of the present invention. A certain number of temperature sensors 1 are evenly distributed at the air inlets of rack 2 to monitor the temperature distribution of the air inlets of rack 2. At the same time, another temperature sensor 2 is installed under the active ventilation floor. Monitors actively ventilated underfloor supply air temperature.

In the art, the rack 2 is a rectangular iron box in which a certain number of servers are placed, and many racks are placed in rows. In a row of racks, the left and right panels of a rack are generally close to other racks. The front panel of the rack is the air inlet, which is used to suck in cold air to cool the server, and the rear panel of the rack is the air outlet, which is used to discharge cooling. After the hot air, monitoring the temperature distribution of the air inlet of the rack is to monitor the temperature of certain positions on the front panel of the rack. The temperature of these positions constitutes the temperature distribution of the air inlet of the rack. Therefore, the number of temperature sensors 1 depends on the number of these positions. .

The intelligent control method of the active ventilation floor of the present invention runs on the PC side, the PC6 is connected to the microcontroller 3, the microcontroller 3 is connected to the driving board 4, and the driving board 4 is connected to the active ventilation floor fan 7 after being connected to the switching power supply 5 (12V, 20A). connect. According to the temperature distribution returned by the temperature sensor 1, the duty cycle value of the PWM signal is generated and transmitted to the microcontroller 3. The microcontroller 3 generates the corresponding PWM signal according to the duty cycle value, and transmits it to the driver board 4. The driver board 4. Control the voltage provided by the switching power supply 5 to the active ventilation floor fan 7 according to the PWM signal, and achieve the purpose of adjusting the fan speed by controlling the fan power supply voltage.

The control method includes the following parts:

1. For the raised floor structure (the air supply structure of the data center, the floor of the data center computer room is raised, leaving a 60-100cm high under-floor space for the air conditioner of the computer room to deliver cold air. This structure is the raised floor structure. At present, domestic Most data centers use this structure) The data center rack hotspot problem establishes a Markov decision process model, which consists of the following four parts: ABCD:

A system state, which is a collection of temperature distributions of the air inlets of the rack with history, and its formula is:

φ _t = {s _tp ,…,s _x …,s _t-1 ,s _t }, where

为温度传感器一的集合，

为温度传感器一的总数。where φ _t is the system state at time t, s _tp , s _x , s _t-1 , and s _t are the temperature distribution of the air inlet of the rack at time tp, x, t-1, and t, respectively, x∈[tp,t], p is the history length; T _i is the reading of temperature sensor one numbered i,

is a set of temperature sensors,

is the total number of temperature sensors one.

定义为离散化的PWM信号占空比值，其公式为：B behavior space

Defined as the discretized PWM signal duty cycle value, its formula is:

中某个行为，DC为PWM信号占空比，max(DC)为最大占空比，D_DRL为DC离散化等分比，k表示某个行为中D_DRL的个数；where a is

In a certain behavior, DC is the PWM signal duty cycle, max(DC) is the maximum duty cycle, D _DRL is the DC discretization equal division ratio, and k represents the number of D _DRLs in a certain behavior;

The C reward R _t+1 is composed of two parts: the quantitative index of the uniformity of the temperature distribution of the air inlet of the rack and the energy consumption of the active ventilation floor fan. The formula is:

表示机架入风口温度分布均匀程度，该式值全为负，越接近0，表明机架入风口温度分布越均匀，其中T_t,i为t时刻编号为i的传感器一的温度读数，

为t时刻机架参考温度，

T_t,under为t时刻所述温度传感器二的读数，Δ_T为根据主动通风地板上下冷热气流混合程度设置的固定温度差，为正数；-(A_ref×DC_t)³表示主动通风地板风扇能耗，该式的值全为负，越接近0，表明风扇能耗越低，其中A_ref为保持与机架入风口温度分布均匀程度同一量级的参考行为值，DC_t为t时刻PWM信号方波占空比。where R _t+1 is the reward obtained by the system after taking a certain behavior at time t,

Indicates the uniformity of the temperature distribution at the air inlet of the rack. The value of this formula is all negative. The closer to 0, the more uniform the temperature distribution of the air inlet of the rack is. T _t,i is the temperature reading of the sensor numbered i at time t.

is the rack reference temperature at time t,

T _t,under is the reading of the temperature sensor 2 at time t, Δ _T is the fixed temperature difference set according to the mixing degree of hot and cold air above and below the active ventilation floor, which is a positive number; -(A _ref ×DC _t ) ³ represents active ventilation The energy consumption of the floor fan. The values of this formula are all negative. The closer to 0, the lower the energy consumption of the fan. A _ref is a reference behavior value that maintains the same level of uniformity as the temperature distribution of the air inlet of the rack. DC _t is t The duty cycle of the square wave of the PWM signal at the moment.

The D value function Q(φ _t , at _t ) is the behavioral value function, and its formula is:

为t时刻系统采取的行为，

为期望函数，y为相对于t时刻的未来时刻，R_t+y+1表示系统在t+y时刻采取行为后获得的奖励，γ表示衰减因子，表示在某状态下采取某行为对系统未来奖励即环境影响的重视程度，0≤γ＜1，γ^y为γ的y次方，是t+y时刻R_t+y+1的衰减因子。where the value function Q(φ _t , at _t ) is called the Q function,

is the action taken by the system at time t,

is the expectation function, y is the future time relative to time t, R _t+y+1 represents the reward obtained by the system after taking action at time t+y, γ represents the decay factor, which means that taking a certain behavior in a certain state will affect the future of the system Reward is the importance of environmental impact, 0≤γ<1, γy is the ^y power of γ, which is the decay factor of R _t+y+1 at time t+y.

The E model can be summarized as, in the system state at any time t, by selecting the optimal behavior to maximize the cumulative reward, the model formula is:

bound to

Among them, γ ^t is the decay factor of R _t+1 at time t.

2. Model solution and solution algorithm

For the solution of the model described in a, after the optimal Q function is obtained by calculation, the optimal behavior can be selected under the system state at any time t according to the optimal Q function, so as to maximize the cumulative reward. The calculation formula of the optimal Q function is:

At any time t, the optimal behavior selection formula is:

中的某一行为。where Q ^* (φ _t , a _t ) represents the optimal Q function, φ _t+1 represents the system state at time t+1, and a represents any one of all actions that the system may take at time t+1, that is, behavior space

one of the behaviors.

The b solution algorithm is to calculate the optimal Q function and select the optimal behavior in the decision-making, so as to maximize the cumulative reward. The solution algorithms include basic intelligence algorithms, sample value variant intelligence algorithms and structural variant intelligence algorithms, all of which are accumulated through continuous decision-making (φ _t , at , R _{t +1} , φ _t+1 ) sample records Train the neural network so that the neural network can approximate the Q function, and then select the optimal behavior to maximize the cumulative reward of the model, where φ _t is the state of the system at time t, a _t is the behavior taken by the system at time t, and R _{t +1} is the reward obtained by the system after taking a _t , and φ _t+1 is the state of the system at time t+1. The three algorithms are designed as follows:

a) The basic intelligent algorithm uses two neural networks with the same structure to approximate the Q function, one is used to approximate the Q sample function and calculate the Q sample value, which is called targ network; the other is used to approximate the Q prediction function and calculate the Q prediction value, called eval network; use the sample record to calculate the difference between the Q sample value and the Q predicted value, train and update the neural network, and the Q sample value calculation formula is:

为行为空间，θ_t,target为t时刻targ网络参数集合。where Q _t+1,target is the Q sample value, R _t+1 and φ _t+1 are taken from the sample records, 0≤γ<1 is the attenuation factor, Q(φ _t+1 ,a; θ _t,target ) is the set of Q samples output by the targ network, a represents all actions that the system may take at time t+1,

is the behavior space, θ _{t, target} is the set of targ network parameters at time t.

The neural network update method is as follows:

为

关于θ_t,eval的梯度，α为神经网络学习步长，θ_target是时刻t为N的整数倍(包括0)时，targ网络参数集合，θ_eval是时刻t为N的整数倍(包括0)时，eval网络参数集合。Where δ _t+1 is the difference between the Q sample value and the corresponding Q predicted value, Q(φ _t , at ; θ _{t , eval} ) is the Q predicted value corresponding to a _t in the Q prediction set output by the eval network, φ _t and a _t are taken from the sample records, θ _{t, eval} is the parameter set at time t of the eval network, θ _{t+1, eval} is the parameter set at time t+1 of the eval network,

for

Regarding the gradient of θ _{t and eval} , α is the learning step size of the neural network, θ _target is the set of targ network parameters when time t is an integer multiple of N (including 0), and θ _eval is an integer multiple of N (including 0) at time t ), the set of eval network parameters.

b) The sample value variant intelligent algorithm uses the formula when calculating the Q sample value:

为targ网络输出的Q样本集合中，使Q_eval(φ_t+1,a；θ_t,eval)最大的行为对应的Q样本值，Q_eval(φ_t+1,a；θ_t,eval)为eval网络输出的Q预测集合，a表示在t+1时刻系统可能采取的所有行为中的任一行为，亦即行为空间

中的某一行为，θ_t,eval为t时刻eval网络参数集合，θ_t,target为t时刻targ网络参数集合；Wherein Q _{t+1, target} is the Q sample value, R _t+1 and φ _t+1 are taken from the sample records, Q (φ _t+1 , a; θ _{t, target} ) is the set of Q samples output by the targ network ,

It is the Q sample value corresponding to the behavior that maximizes Q _eval (φ _t+1 , a; θ _{t, eval} ) in the set of Q samples output by the targ network, Q _eval (φ _t+1 , a; θ _{t, eval} ) is the Q prediction set output by the eval network, a represents any of all the actions that the system may take at time t+1, that is, the action space

For a certain behavior in , θ _{t, eval} is the set of eval network parameters at time t, and θ _{t, target} is the set of targ network parameters at time t;

The neural network structure of the sample value variant intelligent algorithm and its update method are the same as those of the basic intelligent algorithm.

c) The structural variant intelligent algorithm uses two neural networks with the same structure, and sets a DN layer on the penultimate layer of each neural network. The DN layer is divided into V segment and A segment, and the number of neurons in the V segment is 1. Represents the state of the system at time t. The number of neurons in segment A is the number of elements in the behavior space, representing all possible behaviors in this system state. The calculation formula of the DN layer is:

为行为空间中元素个数。Among them, Q(φ _t , at ; θ _t , θ _{t , V} , θ _{t, A} ) is the final output of the neural network, φ _t and at _t are taken from the sample records, θ _t is time t, the structural variant The set of network parameters before the _{DN layer of} the intelligent algorithm _neural network _. ) is the V segment output, A(φ _t , at _t ; θ _t , θ _{t, A} ) is the output value corresponding to a _t in the A segment, A (φ _t , a'; θ _t , θ _{t, A} ) is All outputs of segment A, a' represents all possible actions of the system in the state φ _t ,

is the number of elements in the behavior space.

After that, adopt the same Q sample value calculation and neural network update method as the sample value variant intelligent algorithm to train and update the neural network.

3. By continuously exploring and learning the complex relationship between the temperature distribution of the rack air inlet and the fan speed of the active ventilation floor, and finally according to the temperature distribution of the air inlet of the rack, the optimal PWM signal duty cycle value is generated, and the speed of the active ventilation floor fan is adjusted so that the The temperature distribution of the air inlet of the rack is uniform, which alleviates the problem of rack hot spots. Its operation logic on the PC side is as follows:

1: In different control algorithms, construct and initialize different neural networks, and make the targ network parameters the same as the eval network parameters; set the sample record buffer array; set the reference temperature T _t ;

2: Set the initial time t=0, the time of the sample record in the cache array is recorded as τ; the initial behavior exploration probability ε, the exploration rate decreases with t Δ _ε , the minimum exploration probability ε _min ;

3: Randomly select actions in Z moments, and record (φ _z∈[0,Z) ,a _z∈[0,Z) ,R _z+1∈[0,Z] ,φ z∈[0,Z) ,R z+1∈[0,Z] , _z+1∈[0,Z] ) is stored in the cache array;

4: Obtain the initial rack air inlet temperature distribution

5: The loop body starts;

6: Obtain p historical rack air inlet temperature distributions to form a system state φ _t ={s _tp ,...,s _t-1 ,s _t };

7: If _t =0, select the behavior at =max(DC) and turn to 9, otherwise turn to 8;

8: Use the following formula to select the behavior:

9: Execute a _t , the PC sends a duty cycle command to the microcontroller, changes the fan speed, and obtains the rack air inlet temperature distribution s _t+1 at the next moment of the system, and calculates R _t+1 according to the reward formula in claim 4 ;

10: According to the latest p temperature distribution history, form the next state φ _t+1 ={s _t+1-p ,...,s _t ,s _{t +1} }, and combine (φ _t ,at ,R _{t +1} ,φ _t+1 ) is stored in the cache array;

11: Randomly extract Y sample records from the cache array (φ _τ , a _τ , R _τ+1 , φ _τ+1 );

12: According to different control algorithms, use Y records to calculate the Q sample value, the formula is as follows:

13: Update the eval network with the learning step size α and the following loss function:

14: The exploration probability ε takes the minimum value of ε- _Δε and ε _min ;

15: If t mod N=0, the targ network copies the eval network parameters, otherwise go to 16;

16: time t increases by 1;

17: The loop body ends.

In summary, the present invention establishes a Markov decision process model for the hotspot problem of data center racks, and provides three model solving algorithms, including the basic intelligent algorithm, the sample value variant intelligence algorithm and the structural variant intelligence algorithm, which are respectively used as active ventilation. The core of the floor control algorithm. The model consists of four parts: system state, behavior, reward and value function. The solution of the model is that the optimal behavior is continuously selected under a series of system states to maximize the cumulative reward of the system. The active ventilation floor control algorithm, through continuous exploration and Learn the complex relationship between the temperature distribution of the rack air inlet and the fan speed of the active ventilation floor. Finally, according to the temperature distribution of the rack air inlet, the optimal PWM signal duty cycle value can be generated, and the speed of the active ventilation floor fan can be adjusted so that the temperature of the rack air inlet can be adjusted. Uniform distribution to alleviate rack hotspot issues. Compared with other solutions, the present invention is more universal, easier to deploy, and more cost-effective.

Claims (6)

1. the intelligent control method of the active ventilation floor of the data center, is characterized in that, comprises the steps: Step 1, set a certain number of temperature sensors at the air inlet of the rack for monitoring the temperature distribution of the air inlet of the rack; Step 2, establish a Markov decision process model for the data center rack hotspot problem, the model consists of system state φ _t , behavior space

The reward R _t+1 and the value function Q (φ _t , a _t ) are composed of four parts; Among them: the system state φ _t at time t is defined as the temperature distribution set of the air inlet of the rack with history, and its formula is: φ _t = {s _tp ,…,s _x …,s _t-1 ,s _t }, where

where s _tp , s _x , s _t-1 , and s _t are the temperature distribution of the air inlet of the rack at tp, x, t-1, and t, respectively, x∈[tp,t], p is the history length; T _i is the serial number is the reading of temperature sensor one of i,

is a set of temperature sensors,

is the total number of temperature sensors one; behavior space

Defined as the discretized PWM signal duty cycle value, its formula is:

where a is

In a certain behavior, DC is the PWM signal duty cycle, max(DC) is the maximum duty cycle, D _DRL is the DC discretization equal division ratio, and k represents the number of D _DRLs in a certain behavior; The reward R _t+1 is composed of the quantitative index of the uniformity of the temperature distribution of the air inlet of the rack and the energy consumption of the active ventilation floor fan. The formula is:

where R _t+1 is the reward obtained by the system after taking a certain behavior at time t,

Indicates the uniformity of the temperature distribution at the air inlet of the rack. The value of this formula is all negative. The closer to 0, the more uniform the temperature distribution of the air inlet of the rack is. T _t,i is the temperature reading of the sensor numbered i at time t.

is the rack reference temperature at time t,

T _t,under is the reading of the temperature sensor 2 at time t, Δ _T is the fixed temperature difference set according to the mixing degree of hot and cold air above and below the active ventilation floor, which is a positive number; -(A _ref ×DC _t ) ³ represents active ventilation The energy consumption of the floor fan. The values of this formula are all negative. The closer to 0, the lower the energy consumption of the fan. A _ref is a reference behavior value that maintains the same level of uniformity as the temperature distribution of the air inlet of the rack. DC _t is t The duty cycle of the square wave of the PWM signal at the moment; The value function Q(φ _t , at _t ) is the behavioral value function, and its formula is:

where the value function Q(φ _t , at _t ) is called the Q function,

is the action taken by the system at time t,

is the expectation function, y is the future time relative to time t, R _t+y+1 represents the reward obtained by the system after taking action at time t+y, γ represents the decay factor, which means that taking a certain behavior in a certain state will affect the future of the system Reward is the importance of environmental impact, 0≤γ<1, γy is the ^y power of γ, which is the decay factor of R _t+y+1 at time t+y; The Markov decision process model is summarized as: in the system state at any time t, by selecting the optimal behavior to maximize the cumulative reward of the system, the formula is:

bound to

Among them, γ ^t is the decay factor of R _t+1 at time t; Step 3: Solve the model, through continuous exploration and learning of the complex relationship between the rack air inlet temperature distribution and the fan speed of the active ventilation floor, and finally generate the optimal PWM signal duty cycle value according to the rack air inlet temperature distribution, and adjust The active ventilation floor fan speed makes the temperature distribution of the rack air inlets uniform and alleviates the problem of rack hot spots. 2. The intelligent control method for the active ventilation floor of the data center according to claim 1, wherein in the step 2, the optimal Q function is obtained by calculation, and the optimal Q function can be selected according to the system state at any time t. The optimal behavior maximizes the cumulative reward, and the optimal Q function calculation formula is:

At any time t, the optimal behavior selection formula is:

where Q ^* (φ _t , a _t ) represents the optimal Q function, φ _t+1 represents the system state at time t+1, and a represents any one of all actions that the system may take at time t+1, that is, behavior space

one of the behaviors. 3. The intelligent control method for the active ventilation floor of the data center according to claim 1, wherein in the step 3, the basic intelligent algorithm, the sample value variant intelligent algorithm and the structural variant intelligent algorithm are used to solve the model, and the model is solved by continuously The decision accumulation (φ _t , at , R _{t +1} , φ _t+1 ) sample records to train the neural network, so that the neural network can approximate the Q function, and then select the optimal behavior to maximize the cumulative reward of the model, where φ _t+1 represents the system state at time t+1. 4. The intelligent control method of the active ventilation floor of the data center according to claim 3, wherein the basic intelligent algorithm uses two neural networks with the same structure to approximate the Q function, one is used to approximate the Q sample function, and calculates the Q function. The sample value is called the targ network; the other is used to approximate the Q prediction function, and the Q prediction value is calculated, which is called the eval network; the difference between the Q sample value and the Q prediction value is calculated by using the sample record, and the neural network is trained and updated. The formula for calculating the Q sample value is:

In the sample value variant intelligent algorithm, the Q sample value calculation formula is:

Wherein Q _{t+1, target} is the Q sample value, R _t+1 and φ _t+1 are taken from the sample records, Q (φ _t+1 , a; θ _{t, target} ) is the set of Q samples output by the targ network ,

It is the Q sample value corresponding to the behavior that maximizes Q _eval (φ _t+1 , a; θ _{t, eval} ) in the set of Q samples output by the targ network, Q _eval (φ _t+1 , a; θ _{t, eval} ) is the Q prediction set output by the eval network, a represents any of all the actions that the system may take at time t+1, that is, the action space

For a certain behavior in , θ _{t, eval} is the set of eval network parameters at time t, and θ _{t, target} is the set of targ network parameters at time t; The neural network update method is as follows:

Where δ _t+1 is the difference between the Q sample value and the corresponding Q predicted value, Q(φ _t , at ; θ _{t , eval} ) is the Q predicted value corresponding to a _t in the Q prediction set output by the eval network, φ _t and a _t are taken from the sample records, θ _{t+1, eval} is the set of eval network parameters at time t+1,

for

Regarding the gradient of θ _{t and eval} , α is the learning step size of the neural network, θ _target is the set of targ network parameters when time t is an integer multiple of N including 0, and θ _eval is the time t is N including 0 The set of eval network parameters when an integer multiple of . 5. The intelligent control method for the active ventilation floor of the data center according to claim 4, wherein the structural variant intelligent algorithm uses two neural networks with the same structure, and is set on the penultimate layer of each neural network DN layer, the DN layer is divided into V segment and A segment, where the number of neurons in segment V is 1, which represents the system state at time t, and the number of neurons in segment A is the number of elements in the behavior space, indicating that it is possible in this system state. All actions taken, the DN layer calculation formula is:

Among them, Q(φ _t , at ; θ _t , θ _{t , V} , θ _{t, A} ) is the final output of the neural network, φ _t and at _t are taken from the sample records, θ _t is time t, the structural variant The set of network parameters before the _{DN layer of} the intelligent algorithm _neural network _. ) is the V segment output, A(φ _t , at _t ; θ _t , θ _{t, A} ) is the output value corresponding to a _t in the A segment, A (φ _t , a'; θ _t , θ _{t, A} ) is All outputs of segment A, a' represents all possible actions of the system in the state φ _t ,

is the number of elements in the behavior space; After that, adopt the same Q sample value calculation and neural network update method as the sample value variant intelligent algorithm to train and update the neural network. 6. The intelligent control method of the active ventilation floor of the data center according to claim 1, wherein the operation logic of the intelligent control method is as follows: 1: In different control algorithms, construct and initialize different neural networks, and make the targ network parameters the same as the eval network parameters; set the sample record cache array; set the reference temperature

2: Set the initial time t=0, the time of the sample record in the cache array is recorded as τ; the initial behavior exploration probability ε, the exploration rate decreases with t Δ _ε , the minimum exploration probability ε _min ; 3: Randomly select actions in Z moments, and record (φ _z∈[0,Z) ,a _z∈[0,Z) ,R _z+1∈[0,Z] ,φ z∈[0,Z) ,R z+1∈[0,Z] , _z+1∈[0,Z] ) is stored in the cache array; 4: Obtain the initial rack air inlet temperature distribution

5: The loop body starts; 6: Obtain p historical rack air inlet temperature distributions to form a system state φ _t ={s _tp ,...,s _t-1 ,s _t }; 7: If _t =0, select the behavior at =max(DC) and turn to 9, otherwise turn to 8; 8: Use the following formula to select the behavior:

9: Execute a _t , the PC sends a duty cycle command to the microcontroller, changes the fan speed, and obtains the rack air inlet temperature distribution s _t+1 at the next moment of the system, and calculates R _t+1 according to the reward formula in claim 4 ; 10: According to the latest p temperature distribution history, form the next state φ _t+1 ={s _t+1-p ,...,s _t ,s _{t +1} }, and combine (φ _t ,at ,R _{t +1} ,φ _t+1 ) is stored in the cache array; 11: Randomly extract Y sample records from the cache array (φ _τ , a _τ , R _τ+1 , φ _τ+1 ); 12: According to different control algorithms, use Y records to calculate the Q sample value, the formula is as follows:

13: Update the eval network with the learning step size α and the following loss function:

14: The exploration probability ε takes the minimum value of ε- _Δε and ε _min ; 15: If t mod N=0, the targ network copies the eval network parameters, otherwise go to 16; 16: time t increases by 1; 17: The loop body ends.

Images