CN114126365B - High-energy-efficiency liquid cooling method and liquid cooling system for data center - Google Patents
High-energy-efficiency liquid cooling method and liquid cooling system for data center Download PDFInfo
- Publication number
- CN114126365B CN114126365B CN202111406849.5A CN202111406849A CN114126365B CN 114126365 B CN114126365 B CN 114126365B CN 202111406849 A CN202111406849 A CN 202111406849A CN 114126365 B CN114126365 B CN 114126365B
- Authority
- CN
- China
- Prior art keywords
- hardware
- temperature
- water
- liquid
- pipeline
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H05—ELECTRIC TECHNIQUES NOT OTHERWISE PROVIDED FOR
- H05K—PRINTED CIRCUITS; CASINGS OR CONSTRUCTIONAL DETAILS OF ELECTRIC APPARATUS; MANUFACTURE OF ASSEMBLAGES OF ELECTRICAL COMPONENTS
- H05K7/00—Constructional details common to different types of electric apparatus
- H05K7/20—Modifications to facilitate cooling, ventilating, or heating
- H05K7/2029—Modifications to facilitate cooling, ventilating, or heating using a liquid coolant with phase change in electronic enclosures
- H05K7/20381—Thermal management, e.g. evaporation control
-
- H—ELECTRICITY
- H05—ELECTRIC TECHNIQUES NOT OTHERWISE PROVIDED FOR
- H05K—PRINTED CIRCUITS; CASINGS OR CONSTRUCTIONAL DETAILS OF ELECTRIC APPARATUS; MANUFACTURE OF ASSEMBLAGES OF ELECTRICAL COMPONENTS
- H05K7/00—Constructional details common to different types of electric apparatus
- H05K7/20—Modifications to facilitate cooling, ventilating, or heating
- H05K7/2029—Modifications to facilitate cooling, ventilating, or heating using a liquid coolant with phase change in electronic enclosures
- H05K7/20327—Accessories for moving fluid, for connecting fluid conduits, for distributing fluid or for preventing leakage, e.g. pumps, tanks or manifolds
Abstract
The application discloses a high-energy-efficiency liquid cooling method and system for a data center, wherein the method comprises the following steps: determining a key heat production parameter x1 causing the computing hardware to produce heat and a key heat production parameter x2 causing the memory hardware to produce heat in the data center; obtaining water temperature T, flow velocity v and key heat production parametersCounting the hardware temperature T 'of x1 at different values, obtaining a first data set of the computing hardware, fitting, and obtaining a computing hardware temperature function T' ═ F 1 (T, v, x 1); obtaining the memory hardware temperature change rate delta of the current hardware temperature Td, the water temperature T and the key heat production parameter x2 at different values, obtaining a second data set of the memory hardware and fitting the second data set to obtain a memory hardware temperature change rate function delta F 2 (T, Td, x 2); and solving the target water temperature and flow rate required to be regulated according to the safe temperature of each hardware. According to the heat dissipation requirement of each hardware, the method can be used for adjusting pertinently, and the refrigeration energy consumption is minimized on the premise of meeting the heat dissipation requirement of each hardware.
Description
Technical Field
The application belongs to the technical field of energy consumption optimization of data centers, and particularly relates to a high-energy-efficiency liquid cooling method and a high-energy-efficiency liquid cooling system for the data centers.
Background
During operation of the data center, the various types of hardware with which the data center is equipped generate a large amount of heat. In order to timely take these heat generating zones away from the data center and ensure that the data center operates safely and stably, the refrigeration system plays a crucial role. Typically, refrigeration systems consume significant electrical resources, accounting for approximately 30% -50% of IT equipment energy consumption. For smaller scale edge data centers, the energy consumption duty of the refrigeration system tends to be higher. Therefore, improving the refrigeration efficiency of the refrigeration system has great significance for reducing the total energy consumption of the data center and practicing the strategic goals of energy conservation and emission reduction.
The liquid cooling data center mainly uses a cold plate type liquid cooling technology, namely, a water cooling head with refrigerating water flowing through is directly attached to the surface of hardware through silicone grease, and when the refrigerating water passes through, the water cooling head absorbs and takes away heat from the hardware. The refrigerating water flows out from water cooling heads of different hardware and is gathered together, and then the heated refrigerating water is cooled again to a certain temperature through refrigerating equipment such as a refrigerator, a cooling tower and the like. To ensure safety of all hardware, the temperature is typically set low, e.g., 7-10 ℃. In a conventional coarse-grained cold water refrigeration system, different hardware uses the same refrigeration water temperature and flow rate. However, these different pieces of hardware may have different utilization and power, and different components within the same piece of hardware may also have different utilization and power distributions. Such hot spot problems between and within hardware can lead to differentiated heat dissipation requirements, which in turn can cause the refrigeration efficiency of conventional coarse-grained water cooling systems to be very low, especially when the edge data center is equipped with heterogeneous hardware with different characteristics. In order to improve the refrigeration efficiency and reduce the refrigeration energy consumption, the academic world and the industrial world have proposed warm water refrigeration technology, and such coarse-grained warm water refrigeration system uses uniform warm water to dissipate heat of all hardware, and the temperature of the refrigeration water is usually set at a high level, for example, 40-45 ℃. However, excessive water temperature may cause performance degradation, life reduction, failure, etc. of hardware with high utilization and power, and even cause system downtime.
Disclosure of Invention
Aiming at the defects or the improvement requirements in the prior art, the application provides a high-energy-efficiency liquid cooling method and a liquid cooling system for a data center, and aims to utilize the high-energy-efficiency liquid cooling system with fine granularity to dissipate heat according to the heat dissipation requirement and the characteristics of each hardware, so that the refrigeration energy consumption is reduced.
To achieve the above object, according to one aspect of the present application, there is provided a data center energy-efficient liquid cooling method, including:
determining a key heat production parameter x1 causing the computing hardware to produce heat and a key heat production parameter x2 causing the memory hardware to produce heat in the data center;
obtaining the calculated hardware temperature T ' of the water temperature T, the flow velocity v and the key heat production parameter x1 at different values to obtain a first data set (T ', (T, v, x1)) of the calculated hardware, fitting the first data set to obtain a calculated hardware temperature function T ' ═ F of the water temperature T, the flow velocity v and the key heat production parameter x1 as variables 1 (T，v，x1)；
Obtaining the current temperature Td of hardware, the water temperature T and the temperature change rate delta of the memory hardware of a key heat production parameter x2 at different values to obtain a second data set (delta, (Td, T, x2)) of the memory hardware, fitting the second data set to obtain the current temperature T of the hardwared. The water temperature T and the memory hardware temperature change rate function delta F with the key heat production parameter x2 as variables 2 (T，Td，x2)；
And calculating a solution set of the water temperature T and the flow velocity v when the water temperature T and the flow velocity v do not exceed the safety temperature of the calculation hardware according to the temperature function of the calculation hardware, selecting a group of solutions (T, v) from the solution set for regulation, calculating a water temperature solution set when the water temperature T and the flow velocity v do not exceed the safety temperature of the calculation hardware according to the temperature change rate function of the memory hardware, and selecting a water temperature solution from the water temperature solution set for regulation.
Preferably, the regulation flow rate aiming at the memory hardware is a fixed value set in advance, and the range is 20L/h-40L/h.
Preferably, the method further comprises the following steps:
the method comprises the steps of building a liquid cooling pipeline, wherein the liquid cooling pipeline comprises a cold water supply pipeline, a hot water supply pipeline, a plurality of mixing pipelines and control valves, the input end of each mixing pipeline is communicated with the cold water supply pipeline and the hot water supply pipeline respectively, the output end of each mixing pipeline is used for supplying mixed liquid to target hardware, each mixing pipeline is provided with the control valve, the control valves are used for regulating and controlling the flow rate of cold water and the flow rate of hot water entering the mixing pipelines, and after solutions used for regulation and control are obtained, the control valves corresponding to positions are controlled according to the corresponding demodulation so as to supply the required liquid to computing hardware and memory hardware respectively for cooling.
Preferably, the hot water supply pipeline comprises a hot water collecting pipe, a water tank and a first hot water output pipe which are communicated, wherein the hot water collecting pipe is used for collecting liquid after the temperature of the hardware is reduced, conveying the liquid to the water tank and pumping the liquid through the first hot water output pipe;
the cold water supply pipe comprises a second hot water output pipe, a refrigerator and a cold water output pipe which are communicated, wherein the second hot water output pipe is also communicated with the water tank and is used for pumping part of liquid in the water tank to the refrigerator for refrigeration and then outputting the part of liquid through the cold water output pipe;
each mixing pipeline is provided with two input ends, one input end is communicated with the cold water output pipe through a control valve, the other input end is communicated with the first hot water output pipe through a control valve, and the mixing pipeline is used for mixing the liquid regulated and controlled by the control valve and then conveying the mixed liquid to a corresponding hardware for cooling.
Preferably, for a power of P i The computing hardware i of (a) selecting a set of solutions (T) from the solution set warm,i ，v warm,i ) The method comprises the following steps of (1),
calculating the water temperature at the water outlet of the hardware i
T out,i ＝T warm,i +P i /cμv warm,i
Wherein, P i Represents the power of hardware i, μ and c are the density and specific heat capacity of the liquid, respectively;
calculating the natural heat dissipation power of the liquid in the ith pipeline with the length L
Wherein c, mu and v represent the specific heat capacity, density and flow rate of the liquid in the current pipeline, and T o Indicates the ambient temperature outside the pipeline, r i And r o Respectively representing the internal and external diameters, lambda, of the current pipeline 1 The heat conductivity coefficient of the current pipeline is represented, h represents the convective heat transfer coefficient of air, and beta is an error correction coefficient obtained according to simulation software;
calculating the water temperature flowing from the water outlet of the hardware i to the water tank
Calculating natural heat dissipation power of water tank
Wherein, T tank Indicating the temperature of the water tank, lambda 2 Denotes the heat conductivity of the water tank, r e 、r d And H represents the inside diameter, outside diameter and height of the water tank, respectively;
calculating the natural heat dissipation power of liquid in a liquid cooling pipeline
P total ＝∑ i P pipe,i +P tank
Calculating the gain due to heat dissipation
R HDO ＝P total /COP–P pump
Where COP is the coefficient of performance of the refrigerator, P pump Representing the total power of the pump, P pump ＝α∑ i v warm,i And α is a constant measured experimentally;
in return R HDO Taking the combination T of the water temperature and the flow rate corresponding to the maximum warm,i And v warm,i As a solution for regulation.
Preferably, for a power of P i The computing hardware i of (a) selecting a set of solutions (T) from the solution set warm,i ，v warm,i ) The method comprises the following steps of (1),
calculating cold water supply of liquid cooling pipeline
∑ i v cold,i ＝∑ i v warm,i (T hot -T warm,i )/(T hot -T cold )
Wherein, T hot For the known temperature of the hot water in the hot water supply line, T cold A known cold water temperature in the cold water supply line;
calculating the profit from the refrigerator
R CPO ＝-∑ i v cold,i
In return R CPO Taking the combination T of the water temperature and the flow rate corresponding to the maximum warm,i And v warm,i As a solution for regulation.
Preferably, the control valve for controlling the corresponding position according to the corresponding demodulation after obtaining the solution for the modulation includes:
construction of a set of equations
T warm,i ＝(v hot,i T hot +v cold,i T cold )/(v hot,i +v cold,i )
v warm,i ＝v hot,i +v cold,i
Wherein, T hot And T cold Respectively representing the temperature of the hot and cold water before mixing, v hot,i And v cold,i Respectively representing the flow rates of hot and cold water to be passed through the mixing circuit, T warm,i And v warm,i Respectively representing the water temperature and the flow rate of the mixed corresponding hardware i regulation solution;
and solving the equation set to obtain the flow rates of the hot water and the cold water which need to be regulated and controlled by the control valve.
Preferably, the computing hardware includes a CPU and a GPU, and the memory hardware includes a DRAM, where a key heat-generating parameter of the CPU is a utilization rate of the hardware, a key heat-generating parameter of the GPU is a power of the hardware, and a key heat-generating parameter of the memory hardware is a utilization rate of the hardware.
According to another aspect of the present application, there is provided a data center high-energy liquid cooling system, which includes a liquid cooling pipeline and a control module, wherein
The liquid cooling pipeline comprises a cold water supply pipeline, a hot water supply pipeline, a plurality of mixing pipelines and a control valve:
the hot water supply pipeline comprises a hot water collecting pipe, a water tank and a first hot water output pipe which are communicated, wherein the hot water collecting pipe is used for collecting liquid after the hardware is cooled, conveying the liquid to the water tank and pumping the liquid out through the first hot water output pipe;
the cold water supply pipe comprises a second hot water output pipe, a refrigerator and a cold water output pipe which are communicated, wherein the second hot water output pipe is also communicated with the water tank and is used for pumping part of liquid in the water tank to the refrigerator for refrigeration and then outputting the part of liquid through the cold water output pipe;
each mixing pipeline is provided with two input ends, one input end is communicated with the cold water output pipe through a control valve, the other input end is communicated with the first hot water output pipe through a control valve, the control valve is used for regulating and controlling the flow rate of cold water and the flow rate of hot water entering the mixing pipeline, and the mixing pipeline is used for mixing liquid regulated and controlled by the control valve and then conveying the mixed liquid to a corresponding hardware part for cooling;
the control module is used for regulating and controlling the control valve at the corresponding position according to the regulation and control solution obtained by the high-energy-efficiency liquid cooling method of any data center so as to respectively supply required liquid to the computing hardware and the memory hardware for cooling.
Preferably, the device further comprises a two-phase cavity, the two-phase cavity comprises a sealed vacuum chamber and a capillary structure in the vacuum chamber, the two-phase cavity comprises an evaporation side surface and a condensation side surface which are opposite, the evaporation side surface is attached to the hardware and used for absorbing heat from the hardware, vaporizing the heat and rising to the condensation side under the action of pressure difference, the condensation side surface is attached to the water cooling head, the gas returns to the evaporation side through capillary action after being condensed at the condensation side, and the liquid of the mixing pipeline flows through the water cooling head.
Generally speaking, the application provides a fine-grained energy-efficient liquid cooling method and system, which are adjusted in a targeted manner according to the heat dissipation requirement of each hardware, so that the corresponding hardware is maintained within a safe temperature, and the refrigeration energy consumption is minimized on the premise of meeting the heat dissipation requirement of each hardware. Meanwhile, the hardware types are divided into computing hardware and memory hardware, and the applicant finds that the temperature influence parameters of the liquid cooling system on the two types of hardware are different, and for the computing hardware, the temperature of the hardware is influenced most by the water temperature and the flow rate in the liquid cooling system, so that when the applicant regulates and controls the temperature of the computing hardware, the corresponding water temperature and the corresponding flow rate can be mainly regulated. For the memory hardware, the applicant finds that the influence of the flow rate of the refrigerating liquid of the liquid cooling system on the temperature of the hardware is small, and the temperature of the refrigerating liquid only influences the change rate of the temperature of the hardware, so that when the applicant regulates and controls the temperature of the memory hardware, the corresponding water temperature is mainly regulated, and the flow rate can be set to be a fixed small value in advance. And by adopting different adjustment strategies for different types of hardware, the refrigeration energy consumption is further reduced.
Drawings
FIG. 1 is a flow chart illustrating steps of a method for energy efficient liquid cooling of a data center according to an embodiment of the present application;
FIG. 2 is a measured data set for a CPU in one embodiment of the present application;
FIG. 3 is a measured data set for a GPU in one embodiment of the present application;
FIG. 4 is a data set measured for a DRAM in one embodiment of the present application;
fig. 5 is a block diagram of an energy-efficient liquid cooling system of a data center according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. In addition, the technical features mentioned in the embodiments of the present application described below may be combined with each other as long as they do not conflict with each other.
Fig. 1 is a flowchart illustrating steps of a method for energy efficient liquid cooling of a data center according to an embodiment of the present application, where the method includes:
step S100: the key heat production parameter x1 that the data center causes to calculate hardware heat production and the key heat production parameter x2 that causes to memory hardware heat production are determined.
In the step, hardware needing heat dissipation of the data center is divided into computing hardware and memory hardware, the computing hardware is mainly hardware participating in data processing, key heat production parameters mainly enabling the computing hardware to produce heat are determined, the memory hardware is mainly hardware participating in data storage, and the key heat production parameters mainly enabling the memory hardware to produce heat are determined. In one embodiment, the computing hardware generally includes a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and the like, and the Memory hardware generally includes a Dynamic Random Access Memory (DRAM), and the like. The key heat-generating parameter causing the heat generation of the CPU is the utilization rate of hardware, the key heat-generating parameter causing the heat generation of the GPU is the power of the hardware, and the key heat-generating parameter causing the heat generation of the DRAM is the utilization rate of the hardware.
In an embodiment, the data center may be an edge data center.
Step S200: obtaining the calculated hardware temperature T ' of the water temperature T, the flow velocity v and the key heat production parameter x1 at different values to obtain a first data set (T ', (T, v, x1)) of the calculated hardware, fitting the first data set to obtain a calculated hardware temperature function T ' ═ F of the water temperature T, the flow velocity v and the key heat production parameter x1 as variables 1 (T，v，x1)。
In the study of calculating the cooling of the hardware, the applicant found that the water temperature and the flow rate of the liquid cooling system can directly affect the real-time temperature of the hardware, and therefore, the water temperature T, the flow rate v and the key heat generation parameter x1 are used as variables, the calculation of the hardware temperature (i.e. the cooled hardware temperature) T 'is used as a target, and the corresponding target values of the variables with different values are obtained through multiple experiments, so as to obtain the first data set (T', (T, v, x 1)). Then, fitting is performed by a fitting method such as quadratic polynomial regression or linear regression to obtain a calculated hardware temperature function T' ═ F 1 (T，v，x1)。
Taking computing hardware as a CPU and a GPU as an example for explanation, temperature functions of the CPU and the GPU are respectively obtained.
For a CPU, the key heat generation parameter x1 is a hardware utilization rate, a plurality of sets of CPU data sets are obtained by changing the hardware utilization rate, the flow rate and the temperature and measuring the CPU temperature under the corresponding parameters, as shown in fig. 2, the sets of CPU data sets are measured in one embodiment, and the functional relationship between the hardware temperature and the hardware utilization rate, the water temperature and the flow rate can be obtained by fitting the data sets.
For the GPU, the key heat generation parameter x1 is the hardware power, and a plurality of sets of data sets of the GPU are obtained by changing the hardware power, the flow rate, and the temperature and measuring the GPU temperature under the corresponding parameters, as shown in fig. 3, which is the data set of the GPU measured in one embodiment, and the functional relationship between the hardware temperature and the hardware power, the water temperature, and the flow rate can be obtained by fitting the data sets.
Step S300:obtaining the memory hardware temperature change rate delta of the current hardware temperature Td, the water temperature T and the key heat production parameter x2 at different values, obtaining a second data set (delta, (Td, T, x2)) of the memory hardware, fitting the second data set, and obtaining a memory hardware temperature change rate function delta-F with the current hardware temperature Td, the water temperature T and the key heat production parameter x2 as variables 2 (T，Td，x2)。
In the process of researching refrigeration of the memory hardware, the applicant finds that the flow velocity of the liquid cooling system has little influence on the temperature of the memory hardware, and the water temperature can directly influence the temperature change rate of the memory hardware, so that the current temperature Td, the water temperature T and a key heat production parameter x2 of the hardware are taken as variables, the temperature change rate delta of the memory hardware is taken as a target, corresponding target values of the variables with different values are obtained through multiple experiments, and a second data set delta is obtained 2 (T, Td, x 2). Then, fitting is carried out through a fitting mode such as linear regression, and the function delta of the temperature change rate of the memory hardware is obtained to be F 2 (T，Td，x2)。
Taking DRAM as an example for illustration, the key heat generation parameter that causes DRAM to generate heat is hardware utilization. By changing the current hardware temperature, water temperature and hardware power and measuring the DRAM temperature under the corresponding parameters, a plurality of sets of data sets of the DRAM are obtained, which are part of the data sets of the DRAM measured in one embodiment as shown in fig. 4 and (a) and (b), and by fitting the data sets, the functional relationship between the hardware temperature and the current hardware temperature, water temperature and hardware power can be obtained.
Step S400: and calculating a solution set of the water temperature T and the flow velocity v when the water temperature T and the flow velocity v do not exceed the safety temperature of the calculation hardware according to the temperature function of the calculation hardware, selecting a group of solutions (T, v) from the solution set for regulation, calculating a water temperature solution set when the water temperature T and the flow velocity v do not exceed the safety temperature of the calculation hardware according to the temperature change rate function of the memory hardware, and selecting a water temperature solution from the water temperature solution set for regulation.
And determining a temperature function of the calculation hardware so that the temperature of the calculation hardware does not exceed the safe temperature, respectively determining the water temperature to be regulated and controlled of each hardware and the optional range of the flow rate, obtaining a series of optional solutions, and selecting a group of solutions.
After the temperature change rate function of the memory hardware is determined, the cooling temperature of the memory hardware does not exceed the safe temperature, the water temperature of the memory hardware needing to be regulated is determined, a series of optional solutions are obtained, and one water temperature is selected from the solutions. For example, the flow rate range of the regulation and control aiming at the memory hardware is 20L/h-40L/h, which is far smaller than the conventional flow rate of the common hardware about 100L/h.
It can be understood that the above scheme is to determine the water temperature and the flow rate which are finally input to the data center to cool various hardware, and the processing of the refrigerating liquid in the early stage pipeline is not limited, so the above scheme is not limited to the applicable liquid cooling pipeline.
The mixing and adjustment of cold water and hot water will be described below as an example.
In one embodiment, the scheme further comprises the step of building a liquid cooling pipeline. As shown in fig. 5, in the computer room 1, each server 2 is composed of a series of heterogeneous hardware, and includes a Central Processing Unit (CPU) 12, a Graphics Processing Unit (GPU) 13, a Dynamic Random Access Memory (DRAM) 14, and the like. The liquid cooling lines include a cold water supply line, a hot water supply line, a plurality of mixing lines, and a control valve 6. The input end of each mixing pipeline is respectively communicated with the cold water supply pipeline and the hot water supply pipeline, and the output end of each mixing pipeline is used for supplying mixed liquid to target hardware. For example, a first mixing circuit may be used to provide the mixed refrigerant fluid to the CPU 12, a second mixing circuit may be used to provide the mixed refrigerant fluid to the GPU 13, and a third mixing circuit may be used to provide the mixed refrigerant fluid to the DRAM 14. Each mixing pipeline is provided with a control valve 6, the control valves 6 are used for regulating and controlling the flow rate of cold water and the flow rate of hot water entering the mixing pipelines, and after a solution for regulation and control is obtained, the control valves at corresponding positions are regulated and controlled according to the corresponding demodulation so as to respectively supply required liquid to the computing hardware and the memory hardware for cooling.
More specifically, with continued reference to fig. 5, the hot water supply pipeline includes a hot water collecting pipe, a water tank and a first hot water output pipe, wherein the hot water collecting pipe is used for collecting the liquid output after the hardware is cooled, conveying the liquid to the water tank, and pumping the liquid by a water pump on the first hot water output pipe. The cold water supply pipe comprises a second hot water output pipe, a refrigerator 15 and a cold water output pipe which are communicated, wherein the second hot water output pipe is also communicated with the water tank and is used for pumping part of liquid in the water tank to the refrigerator 15 through a water pump for refrigeration and then outputting the part of liquid through the cold water output pipe. Each mixing pipeline is provided with two input ends, one of the input ends is communicated with a cold water output pipe through a control valve 6 and used for controlling the flow rate of cold water entering the mixing pipeline through a cold water output end, the other input end is communicated with a first hot water output pipe through the control valve 6 and used for controlling the flow rate of hot water entering the mixing pipeline through the first hot water output end, and liquid regulated and controlled by the control valve 6 is mixed by the mixing pipeline to reach the target flow rate and temperature and is conveyed to a corresponding hardware for cooling. It will be appreciated that the liquid cooling system is comprised primarily of three components, the inner and outer loop 3 and the control module 8. The internal and external cycles 3 provide refrigeration for various heterogeneous hardware, including internal cycle 4 and external cycle 5. The internal circulation 4 is a hot water circulation and directly recovers hot water at the outlet of each hardware; the external circulation 5 is a cold water circulation, and the recovered hot water is pumped to the refrigerator 15 to form cooled refrigerating water again. The control module 8 includes a central control module 10 and a plurality of sub-control modules 11, and each server 2 is directly controlled by one sub-control module 11. The valve 6 at each hardware inlet provides the best cooling water temperature and flow rate for each heterogeneous hardware with different heat dissipation requirements by mixing a certain proportion of the hot water of the internal circulation 4 and the cold water of the external circulation 5 according to the instruction of the control module 8. The fine-grained refrigeration regulation can meet the heat dissipation requirements of various hardware, guarantees the safety of the hardware, and can fully utilize natural heat dissipation to reduce refrigeration energy consumption and improve refrigeration efficiency.
In one embodiment, with continued reference to fig. 5, the liquid cooling circuit is constructed to further include a micro-circulation 7, wherein the micro-circulation 7 is a small gas-liquid circulation realized in the two-phase chamber 16. The two-phase chamber 16 is composed of a sealed vacuum chamber and a plurality of capillary structures inside, and is installed on the modified traditional water cooling head 9. The surface where the water cooling head is attached to is called evaporation side, and the surface where the water cooling head is attached to is called condensation side. The liquid therein absorbs heat from the hardware on the evaporation side and vaporizes, and then rises to the condensation side under the effect of the pressure differential. The gas condenses again on the condensing side and returns to the evaporating side by capillary action. The gas-liquid circulation not only enables local hot spots inside hardware to become more moderate and more uniform temperature distribution to be realized, but also enables the overall heat dissipation capacity to become stronger.
In an embodiment, after obtaining a set of rows in step S400, a solution may be randomly selected for adjustment, or a better solution may be selected according to the following strategy.
One of the strategies is a Heat Dissipation Oriented (HDO) strategy, which comprises the following specific processes:
step S411: calculating the water temperature at the water outlet of the hardware i
T out,i ＝T warm,i +P i /cμv warm,i
Wherein, P i Representing the power of hardware i, and μ and c are the density and specific heat capacity of the liquid, respectively.
Step S412: calculating the natural heat dissipation power of the liquid in the ith pipeline with the length L
Wherein c, mu and v represent the specific heat capacity, density and flow rate of the liquid in the current pipeline, and T o Indicates the ambient temperature outside the pipeline, r i And r o Respectively representing the internal and external diameters, lambda, of the current pipeline 1 The heat conductivity coefficient of the current pipeline is represented, h represents the convective heat transfer coefficient of air, and beta is obtained according to simulation softwareAnd obtaining an error correction coefficient. Specifically, the expression of β can be estimated through Fluent software simulation, and β can be specifically calculated according to the following formula. Further, when h is less than or equal to 100W/m 2 The temperature is controlled, and the water flow velocity sigma is less than or equal to 1m/s, and the water flow velocity sigma is calculated according to the following formula
Step S413: calculating the water temperature flowing from the water outlet of the hardware i to the water tank
Step S414: calculating the natural heat dissipation power of the water tank
Wherein, T tank Indicating the temperature of the tank, which may be based on the temperature of the tank from the previous cycle and the above T tank.i Weighted to obtain λ 2 Denotes the heat conductivity of the water tank, r e 、r d And H denotes an inner diameter, an outer diameter and a height of the water tank, respectively.
Step S415: calculating the natural heat dissipation power of liquid in a liquid cooling pipeline
P total ＝∑ i P pipe,i +P tank
Step S416: calculating the gain due to heat dissipation
R HDO ＝P total /COP–P pump
Where COP is the coefficient of performance of the refrigerator, P pump Representing the total power of the pump, P pump ＝α∑ i v warm,i And α is a constant measured experimentally;
step S417: in return R HDO Taking the combination T of the water temperature and the flow rate corresponding to the maximum warm,i And v warm,i As a solution for regulation.
The other strategy is a refrigerating machine Power Oriented (CPO) strategy, and the specific process is as follows:
step S421: calculating cold water supply of liquid cooling pipeline
∑ i v cold,i ＝∑ i v warm,i (T hot -T warm,i )/(T hot -T cold )
Wherein, T hot For the known temperature of the hot water in the hot water supply line, T cold A known cold water temperature in the cold water supply line;
step S422: calculating the profit from the refrigerator
R CPO ＝-∑ i v cold,i
Step S423: in return R CPO Taking the combination T of the water temperature and the flow rate corresponding to the maximum warm,i And v warm,i As a solution for regulation.
The above strategies are all constructed by the applicant according to a large number of analyses, and experiments verify that the solutions obtained by the above schemes have good refrigeration effect and low refrigeration energy consumption, so that the better solution can be determined according to any one of the above two strategies.
In one embodiment, when the mixed modulation of the cold water and the hot water is required to be synthesized, after the solution of the water temperature and the flow rate is obtained, the parameters of the cold water and the hot water during mixing are required to be modulated and controlled through the control valve so as to obtain the mixed water temperature and the flow rate in the solution.
Specifically, the solution can be performed according to the following scheme:
step S510: construction of a set of equations
T warm,i ＝(v hot,i T hot +v cold,i T cold )/(v hot,i +v cold,i )
v warm,i ＝v hot,i +v cold,i
Wherein, T hot And T cold Respectively represent a mixtureThe temperature of the pre-heated and cold water, which is a known parameter, v hot,i And v cold,i Respectively representing the flow rates of hot and cold water entering the mixing line to be regulated, which is a parameter to be solved, T warm,i And v warm,i Respectively represent the water temperature and the flow rate of the mixed corresponding hardware i control solution, which are known parameters obtained.
Step S520: and solving the equation set to obtain the flow rates of the hot water and the cold water which need to be regulated and controlled by the control valve.
After the flow rates of the cold water and the hot water which are required to be regulated and controlled by the control valves are obtained, the liquid in the mixing pipeline can be regulated and controlled by the corresponding control valves, so that the liquid in the mixing pipeline is at the target water temperature and the target flow rate, and the corresponding hardware is in a safe temperature range.
It should be noted that the execution sequence of each step in the present application is not limited to the above embodiment, and it should be understood that the corresponding step can be executed as long as the execution condition is satisfied.
Based on the data center high-energy-efficiency liquid cooling method, the scheme also relates to a data center high-energy-efficiency liquid cooling system, as shown in fig. 5, the system comprises a liquid cooling pipeline and a control module:
wherein the content of the first and second substances,
the liquid cooling pipeline comprises a cold water supply pipeline, a hot water supply pipeline, a plurality of mixing pipelines and a control valve:
the hot water supply pipeline comprises a hot water collecting pipe, a water tank and a first hot water output pipe which are communicated, wherein the hot water collecting pipe is used for collecting liquid after the hardware is cooled, conveying the liquid to the water tank and pumping the liquid out through the first hot water output pipe;
the cold water supply pipe comprises a second hot water output pipe, a refrigerator 15 and a cold water output pipe which are communicated, wherein the second hot water output pipe is also communicated with the water tank and is used for pumping part of liquid in the water tank to the refrigerator 15 for refrigeration and then outputting the part of liquid through the cold water output pipe;
each mixing pipeline is provided with two input ends, one input end is communicated with a cold water output pipe through a control valve, the other input end is communicated with a first hot water output pipe through a control valve, the control valve is used for regulating and controlling the flow rate of cold water and the flow rate of hot water entering the mixing pipeline, and the mixing pipeline is used for mixing liquid regulated and controlled by the control valve and then conveying the mixed liquid to a corresponding hardware for cooling;
the control module is used for regulating and controlling the control valve at the corresponding position according to the regulation and control solution obtained by the data center high-energy-efficiency liquid cooling method so as to respectively supply required liquid to the computing hardware and the memory hardware for cooling.
Further, the liquid cooling system further comprises a two-phase cavity 16, the two-phase cavity 16 comprises a sealed vacuum chamber and a capillary structure in the vacuum chamber, the two-phase cavity comprises an evaporation side surface and a condensation side surface which are opposite, the evaporation side surface is attached to the hardware 12/13/14, liquid in the liquid is evaporated by absorbing heat from the hardware and rises to the condensation side under the action of pressure difference, the condensation side surface is attached to the water cooling head 9, gas is condensed on the condensation side and then returns to the evaporation side through capillary action, and liquid in the mixing pipeline flows through the water cooling head 9.
In general, the liquid cooling system mainly comprises three parts, namely an internal and external circulation 3, a microcirculation 7 and a control module 8. The internal and external cycles 3 provide refrigeration for various heterogeneous hardware, including internal cycle 4 and external cycle 5. The internal circulation 4 is a hot water circulation and directly recovers hot water at the outlet of each hardware; the external circulation 5 is a cold water circulation, and the recovered hot water is pumped to the refrigerator 15 to form cooled refrigerating water again. The valve 6 at each hardware inlet provides the best cooling water temperature and flow rate for each heterogeneous hardware with different heat dissipation requirements by mixing a certain proportion of the hot water of the internal circulation 4 and the cold water of the external circulation 5 according to the instruction of the control module 8. The fine-grained refrigeration regulation can meet the heat dissipation requirements of various hardware, ensure the safety of the hardware, fully utilize natural heat dissipation to reduce refrigeration energy consumption and improve refrigeration efficiency. The micro-circulation 7 is a small gas-liquid circulation realized in the two-phase chamber 16. The two-phase chamber 16 is composed of a sealed vacuum chamber and a plurality of capillary structures inside, and is installed on the modified traditional water cooling head 9. The surface where the water cooling head is attached to is called evaporation side, and the surface where the water cooling head is attached to is called condensation side. The liquid therein absorbs heat from the hardware on the evaporation side and vaporizes and then rises to the condensation side under the effect of the pressure differential. The gas condenses again on the condensing side and returns to the evaporating side by capillary action. The gas-liquid circulation not only enables local hot spots inside hardware to become more moderate and more uniform temperature distribution to be realized, but also enables the overall heat dissipation capacity to become stronger.
In summary, the present application provides a fine-grained energy-efficient liquid cooling method and system, which are adjusted in a targeted manner according to the heat dissipation requirement of each hardware, so that the corresponding hardware is maintained within a safe temperature, and the refrigeration energy consumption is minimized on the premise of meeting the heat dissipation requirement of each hardware. Meanwhile, the hardware types are divided into computing hardware and memory hardware, and the applicant finds that the temperature influence parameters of the liquid cooling system on the two types of hardware are different, and for the computing hardware, the temperature of the hardware is influenced most by the water temperature and the flow rate in the liquid cooling system, so that when the applicant regulates and controls the temperature of the computing hardware, the corresponding water temperature and the corresponding flow rate can be mainly regulated. For the memory hardware, the applicant finds that the influence of the flow rate of the refrigerating liquid of the liquid cooling system on the temperature of the hardware is small, and the temperature of the refrigerating liquid only influences the change rate of the temperature of the hardware, so that when the applicant regulates and controls the temperature of the memory hardware, the corresponding water temperature is mainly regulated, and the flow rate can be set to be a fixed small value in advance. And by adopting different adjustment strategies for different types of hardware, the refrigeration energy consumption is further reduced.
It will be understood by those skilled in the art that the foregoing is merely a preferred embodiment of this application and is not intended to limit the application, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the application should be included within the scope of protection of the application.
Claims (8)
1. A method for high-energy-efficiency liquid cooling of a data center is characterized by comprising the following steps:
determining a key heat production parameter x1 causing heat production of computing hardware of a data center and a key heat production parameter x2 causing heat production of memory hardware, wherein the computing hardware comprises a CPU and a GPU, the memory hardware comprises a DRAM, the key heat production parameter of the CPU is the utilization rate of the hardware, the key heat production parameter of the GPU is the power of the hardware, and the key heat production parameter of the memory hardware is the utilization rate of the hardware;
obtaining the calculated hardware temperature T ' of the water temperature T, the flow velocity v and the key heat production parameter x1 at different values to obtain a first data set (T ', (T, v, x1)) of the calculated hardware, fitting the first data set to obtain a calculated hardware temperature function T ' ═ F of the water temperature T, the flow velocity v and the key heat production parameter x1 as variables 1 (T，v，x1)；
Obtaining the memory hardware temperature change rate delta of the memory hardware when the current temperature Td, the water temperature T and the key heat production parameter x2 of the memory hardware are at different values, obtaining a second data set (delta, (Td, T, x2)) of the memory hardware, fitting the second data set, and obtaining a memory hardware temperature change rate function delta-F with the current temperature Td, the water temperature T and the key heat production parameter x2 of the memory hardware as variables 2 (T，Td，x2)；
Calculating a solution set of water temperature T and flow velocity v when the water temperature T and the flow velocity v do not exceed the safety temperature of the calculation hardware according to the calculation hardware temperature function, selecting a group of solutions (T, v) from the solution set for regulation, calculating a water temperature solution set when the water temperature T and the flow velocity v do not exceed the safety temperature of the memory hardware according to the memory hardware temperature change rate function, and selecting a water temperature solution from the water temperature solution set for regulation, wherein the regulation flow velocity of the memory hardware is a fixed value set in advance and ranges from 20L/h to 40L/h.
2. The data center energy efficient liquid cooling method of claim 1, further comprising:
the method comprises the steps of building a liquid cooling pipeline, wherein the liquid cooling pipeline comprises a cold water supply pipeline, a hot water supply pipeline, a plurality of mixing pipelines and control valves, the input end of each mixing pipeline is communicated with the cold water supply pipeline and the hot water supply pipeline respectively, the output end of each mixing pipeline is used for supplying mixed liquid to target hardware, each mixing pipeline is provided with the control valve, the control valves are used for regulating and controlling the flow rate of cold water and the flow rate of hot water entering the mixing pipelines, and after solutions used for regulation and control are obtained, the control valves corresponding to positions are controlled according to the corresponding demodulation so as to supply the required liquid to computing hardware and memory hardware respectively for cooling.
3. The method for high-energy-efficiency liquid cooling in the data center according to claim 2, wherein the hot water supply pipeline comprises a hot water collection pipe, a water tank and a first hot water output pipe which are communicated with each other, wherein the hot water collection pipe is used for collecting liquid after cooling the hardware, conveying the liquid to the water tank and pumping the liquid through the first hot water output pipe;
the cold water supply pipeline comprises a second hot water output pipe, a refrigerator and a cold water output pipe which are communicated, wherein the second hot water output pipe is also communicated with the water tank and is used for pumping part of liquid in the water tank to the refrigerator for refrigeration and then outputting the part of liquid through the cold water output pipe;
each mixing pipeline is provided with two input ends, one input end is communicated with the cold water output pipe through a control valve, the other input end is communicated with the first hot water output pipe through a control valve, and the mixing pipeline is used for mixing the liquid regulated and controlled by the control valve and then conveying the mixed liquid to a corresponding hardware for cooling.
4. The energy efficient liquid cooling method of claim 3, wherein the power P is selected for i The computing hardware i of (a) selecting a set of solutions (T) from the solution set warm,i ，v warm,i ) The method comprises the following steps of (1),
calculating the water temperature at the water outlet of the hardware i
T out,i ＝T warm,i +P i /cμv warm,i
Wherein, P i Represents the power of hardware i, μ and c are the density and specific heat capacity of the liquid, respectively;
calculating the natural heat dissipation power of the liquid in the ith pipeline with the length L
Wherein c, mu and v represent the specific heat capacity, density and flow rate of the liquid in the current pipeline, and T o Indicates the ambient temperature outside the pipeline, r i And r o Respectively representing the internal and external diameters, lambda, of the current line 1 The heat conductivity coefficient of the current pipeline is represented, h represents the convective heat transfer coefficient of air, and beta is an error correction coefficient obtained according to simulation software;
calculating the water temperature flowing from the water outlet of the hardware i to the water tank
Calculating natural heat dissipation power of water tank
Wherein, T tank Indicating the temperature of the water tank, lambda 2 Denotes the heat conductivity of the water tank, r e 、r d And H represents the inside diameter, outside diameter and height of the water tank, respectively;
calculating the natural heat dissipation power of liquid in a liquid cooling pipeline
P total ＝∑ i P pipe,i +P tank
Calculating the gain due to heat dissipation
R HDO ＝P total /COP–P pump
Where COP is the coefficient of performance of the refrigerator, P pump Representing the total power of the pump, P pump ＝α∑ i v warm,i And α is a constant measured experimentally;
in return R HDO Taking the combination T of the water temperature and the flow rate corresponding to the maximum warm,i And v warm,i As a solution for regulation.
5. The energy efficient liquid cooling method of claim 3, wherein the power P is used for the data center i The computing hardware i of (a) selecting a set of solutions (T) from the solution set warm,i ，v warm,i ) The method comprises the following steps of (1),
calculating cold water supply of liquid cooling pipeline
∑ i v cold,i ＝∑ i v warm,i (T hot -T warm,i )/(T hot -T cold )
Wherein, T hot For the known temperature of the hot water in the hot water supply line, T cold A known cold water temperature in the cold water supply line;
calculating the profit from the refrigerator
R CPO ＝-∑ i v cold,i
In return R CPO Taking the combination T of the water temperature and the flow rate corresponding to the maximum warm,i And v warm,i As a solution for regulation.
6. The energy-efficient liquid cooling method for data centers according to any one of claims 2 to 5, wherein controlling the control valve of the corresponding position according to the corresponding demodulation after obtaining the solution for regulation comprises:
construction of a set of equations
T warm,i ＝(v hot,i T hot +v cold,i T cold )/(v hot,i +v cold,i )
v warm,i ＝v hot,i +v cold,i
Wherein, T hot And T cold Respectively representing the temperature of the hot and cold water before mixing, v hot,i And v cold,i Respectively representing the flow rates of hot and cold water to be passed through the mixing circuit, T warm,i And v warm,i Respectively representing the water temperature and the flow rate of the mixed corresponding hardware i regulation solution;
and solving the equation set to obtain the flow rates of the hot water and the cold water which need to be regulated and controlled by the control valve.
7. A high-energy-efficiency liquid cooling system of a data center is characterized by comprising a liquid cooling pipeline and a control module, wherein the liquid cooling pipeline and the control module are arranged in parallel
The liquid cooling pipeline comprises a cold water supply pipeline, a hot water supply pipeline, a plurality of mixing pipelines and a control valve:
the hot water supply pipeline comprises a hot water collecting pipe, a water tank and a first hot water output pipe which are communicated, wherein the hot water collecting pipe is used for collecting liquid after the hardware is cooled, conveying the liquid to the water tank and pumping the liquid out through the first hot water output pipe;
the cold water supply pipeline comprises a second hot water output pipe, a refrigerator and a cold water output pipe which are communicated, wherein the second hot water output pipe is also communicated with the water tank and is used for pumping part of liquid in the water tank to the refrigerator for refrigeration and then outputting the part of liquid through the cold water output pipe;
each mixing pipeline is provided with two input ends, one input end is communicated with the cold water output pipe through a control valve, the other input end is communicated with the first hot water output pipe through a control valve, the control valve is used for regulating and controlling the flow rate of cold water and the flow rate of hot water entering the mixing pipeline, and the mixing pipeline is used for mixing liquid regulated and controlled by the control valve and then conveying the mixed liquid to a corresponding hardware part for cooling;
the control module is used for regulating and controlling the control valve at the corresponding position according to the regulation and control solution obtained by the data center energy-efficient liquid cooling method in any one of claims 1 to 6, so as to respectively supply required liquid to the computing hardware and the memory hardware for cooling.
8. The data center energy-efficient liquid cooling system of claim 7 further comprising a two-phase chamber comprising a sealed vacuum chamber and a capillary structure within the vacuum chamber, the two-phase chamber comprising opposing evaporation side surfaces and condensation side surfaces, the evaporation side surfaces being attached to the hardware for absorbing heat from the hardware to vaporize and rise to the condensation side under pressure differential, the condensation side surfaces being attached to the water-cooled head, the gas being condensed at the condensation side and returning to the evaporation side by capillary action, the liquid in the mixing line flowing through the water-cooled head.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111406849.5A CN114126365B (en) | 2021-11-24 | 2021-11-24 | High-energy-efficiency liquid cooling method and liquid cooling system for data center |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111406849.5A CN114126365B (en) | 2021-11-24 | 2021-11-24 | High-energy-efficiency liquid cooling method and liquid cooling system for data center |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114126365A CN114126365A (en) | 2022-03-01 |
| CN114126365B true CN114126365B (en) | 2022-09-16 |
Family
ID=80372280
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111406849.5A Active CN114126365B (en) | 2021-11-24 | 2021-11-24 | High-energy-efficiency liquid cooling method and liquid cooling system for data center |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114126365B (en) |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9423172B2 (en) * | 2010-10-13 | 2016-08-23 | Weldtech Technology (Shanghai) Co., Ltd. | Energy-saving optimized control system and method for refrigeration plant room |
| CN105302182B (en) * | 2015-11-02 | 2018-03-02 | 张存勇 | A kind of server liquid-cooling heat radiation energy-saving control system and method |
| CN107562155A (en) * | 2017-09-07 | 2018-01-09 | 郑州云海信息技术有限公司 | A kind of liquid-cooled suit business device and its flow-measuring method |
| CN109472413A (en) * | 2018-11-14 | 2019-03-15 | 南方电网科学研究院有限责任公司 | Consider the garden integrated energy system Optimization Scheduling of hot pipe network transmission characteristic |
| CN112996368A (en) * | 2021-04-12 | 2021-06-18 | 上海有孚智数云创数字科技有限公司 | Heat dissipation method and device of data center |
| CN113206319B (en) * | 2021-04-27 | 2022-03-11 | 吉林大学 | Fuzzy sliding mode optimization method for liquid cooling system of power battery pack of new energy automobile |
-
2021
- 2021-11-24 CN CN202111406849.5A patent/CN114126365B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN114126365A (en) | 2022-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104024749B (en) | Air-conditioning system that adjusts temperature and humidity | |
| CN108489069A (en) | A kind of air-conditioning heat dissipation structure control method and system | |
| Zinet et al. | A numerical model for the dynamic simulation of a recirculation single-effect absorption chiller | |
| CN109460861A (en) | A kind of chemical circulation cooling water system water pump fan combination frequency conversion optimization operating scheme determines method | |
| CN109827270A (en) | A kind of air-conditioning system for evaporating cooling lithium bromide absorbing type refrigeration | |
| JP2016125771A (en) | Power force optimization system | |
| Zhang et al. | The model predictive control strategy of the transcritical CO2 air conditioning system used in railway vehicles | |
| Hu et al. | Modeling and simulation of a falling film evaporator for a water vapor heat pump system | |
| JP2011089722A (en) | Method and device for refrigeration/air conditioning | |
| CN114126365B (en) | High-energy-efficiency liquid cooling method and liquid cooling system for data center | |
| Li et al. | Experimental study on reliable operation strategy for multi-split backplane cooling system in data centers | |
| Mokarram et al. | Design and modeling of novel two-phase heat exchangers for a home cooling system with ice energy storage | |
| Sun et al. | Experimental study on a novel pump-driven heat pipe/vapor compression system for rack-level cooling of data centers | |
| CN111898260A (en) | Variable flow optimization control method and controller for central air conditioning system | |
| JP2008064404A (en) | Heat storage medium direct-mixing type temperature regulation method | |
| CN103697568B (en) | A kind of air conditioning control method, device and apply the air conditioning unit of its | |
| CN110793240A (en) | Large-scale air source heat pump low-energy consumption operation system and control method | |
| Zhang et al. | Dynamic model based study on mode switching characteristics and reliability analysis of data center composite air conditioning system | |
| Goodheart | Low firing temperature absorption chiller system | |
| Kærn et al. | Compensation of airflow maldistribution in fin-and-tube evaporators | |
| Wang et al. | Air conditioning system with dual-temperature chilled water for air grading treatment in data centers | |
| CN215121650U (en) | Underground works heat extraction temperature control system that airs exhaust | |
| CN114867313B (en) | Distributed energy storage system of data center | |
| Walgama et al. | Indoor and Outdoor Conditions Utilized Energy Saving Scheme for HVAC Cooling Water Systems in Smart Commercial Buildings | |
| Saha et al. | Analytical modeling of a radiant cooling system for thermal performance analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |