AU2017348453A1

AU2017348453A1 - Data centre cooling regulation system and method

Info

Publication number: AU2017348453A1
Application number: AU2017348453A
Authority: AU
Inventors: Mark Kenneth Hubner; Jeffrey Daniel Van Zetten
Original assignee: Nextdc Ltd
Current assignee: Nextdc Ltd
Priority date: 2016-10-25
Filing date: 2017-10-25
Publication date: 2019-05-16
Anticipated expiration: 2037-10-25
Also published as: WO2018076056A1; AU2017348453B2; SG11201903461WA

Abstract

A method of cooling a plurality of computing units accommodated in a space using a cooling fluid, said cooling fluid being introduced into the space via one or more inlets and being arranged to leave the space via one or more outlets, said method comprising: sensing an electrical load of one or more of the computing units, receiving desired temperature control data, determining a required flow rate Vr of the cooling fluid based on the sensed electrical load and the received temperature control data, regulating the flow of the cooling fluid to achieve the required flow rate.

Description

Data centre cooling regulation system and method

Field of the invention

The present invention relates to methods and systems for cooling regulation. Most specifically, it provides a method for cooling a plurality of computing units, such as might exist in a data centre or the like. A cooling regulation system and cooling fluid circulation system are also disclosed.

Background of the invention

Cooling computing units such as servers, data storage systems and networking gear is a key function that must be performed in any data centre. In the past, it has been the case that the energy used in the cooling of data centres has almost equalled the energy used to perform the data centre’s core computing functions. Accordingly, data centre operators have invested a great deal of effort into seeking more efficient ways to cool them.

In addition to cost and resource usage concerns, proper cooling of computing equipment contributes to the reliability and longevity of the computing device.

One proposal to more efficiently provide cooling for a computing system including a plurality of computing units is described in US patent publication 20130098593 A1 (Busch et al.). Busch et al. described a system in which a real-time cooling controller monitors the power consumption of computing devices in each of a plurality of zones, and in response to changes in the power consumption, changes the flow rate of a cooling liquid to a heat exchanger corresponding to the zone. Busch et al. directly seeks to control operation of the computer room air-conditioner (CRAC) that supplies cooling air to the space in which the computing units reside.

Another proposal is described in US patent publication 20080092577 A1 (Martin). Martin describes a system in which electrical load sensors measure power consumption of respective computing units to be cooled, and in response to these measurements controls the relative proportion of cooling fluid delivered to each of the computing units. This is done by controlling an air damper or liquid flow valve that controls the level of cooling fluid arriving at the “cold side” of each computing unit. In Martin, the cooling

WO 2018/076056

PCT/AU2017/051172 fluid is controlled at the “outlet” (using the term of Martin) from the CRAC unit prior to its entry into the space containing the computing units to be cooled.

The present inventors have determined that even though these systems offer advantages over conventional, uncontrolled cooling, further improvements, or at least an alternative would be useful.

Reference to any prior art in the specification is not an acknowledgment or suggestion that this prior art forms part of the common general knowledge in any jurisdiction or that this prior art could reasonably be expected to be understood, regarded as relevant, and/or combined with other pieces of prior art by a skilled person in the art.

Summary of the invention

In a first aspect, the invention provides a method of cooling a plurality of computing units accommodated in a space using a cooling fluid, said cooling fluid being introduced into the space via one or more inlets and being arranged to leave the space via one or more outlets, said method comprising:

sensing an electrical load of one or more of the computing units, receiving desired temperature control data, determining a required flow rate Vr of the cooling fluid based on the sensed electrical load and the received temperature control data, regulating the flow of the cooling fluid to achieve the required flow rate.

In one form, the desired temperature control data includes at least a differential temperature set-point dT and/or a desired exit temperature of the cooling fluid Tr.

In one form, the step of regulating the flows includes causing the cooling fluid to leave the space at, at least one of:

the required flow rate Vr; and the desired exit temperature Tr.

WO 2018/076056

PCT/AU2017/051172

The method preferably includes receiving pressure data indicating the pressure differential dP which causes movement of the cooling fluid between the one or more inlets and the one or more outlets. The method can include determining the pressure differential by measurement of fluid pressure at at least two points. The two points can include a point associated with the one or more inlets (e.g. at or near the inlet, in a duct or plenum used to distribute cooling fluid to the one or more inlets, on the inlet-side of the computing unit); and a point associated with the one or more outlet (e.g. at or near an outlet, or on the outlet-side of the computing unit, in a duct or plenum downstream from the outlet.)

In one form, one or both of:

the desired exit temperature; and the differential temperature control set-point;

is adjustable.

In one embodiment, the desired exit temperature and/or differential temperature control set-point is determined by an operator and/or based on operational tolerances of the one or more computing units.

In one form, the method further includes:

• monitoring the supply temperature of the cooling fluid as it is being introduced into the space, and/or • monitoring the exit temperature of the cooling fluid as it exits the space from the one or more outlets, and • adjusting the required flow rate based on the monitored temperature(s).

In one form, at least one of the outlets is provided with a control valve to regulate the flow of cooling fluid leaving the outlet.

In one form, the control valve has a plurality of adjustable open positions allowing the cooling fluid to leave the space at different flow rates.

Preferably the method further includes:

WO 2018/076056

PCT/AU2017/051172 based on the required flow rate of the cooling fluid, adjusting the open position of the control valve.

Preferably the flow rate of fluid through the control valve is calibrated. Calibration can be performed at a plurality of cooling fluid pressures or over a continuous range of fluid pressures. In preferred forms of the present invention, this enables the method to be applied regardless of the pressure differential between the inlet and outlet.

In one form, each of the one or more outlets is associated with a computing unit, or a group of computing units.

In one form, the one or more outlets are positioned at a location generally adjacent e.g. above the one or more computing units.

In one form, the control valve is a damper operated by a motorized modulating actuator. Preferably the damper is fail-safe open when there is a loss of power, loss of control signal, or a disaster such as fire, earthquake, and similar.

In one form, the inlet(s) is mounted in a vertical wall of the space in which the computing units are accommodated. The inlet(s) may be located at other suitable locations for example in a floor of the space or in a ceiling of the space.

In some embodiments the inlet(s) can be provided with a control valve to regulate the rate of cooling fluid being introduced into the space.

In one form, the electrical load is sensed by: a current sensor, a voltage sensor, a power meter, or a combination of the above.

In one form, the electrical load is sensed continuously, on an ad-hoc basis, periodically, or in accordance with a predetermined sensing schedule.

In one form, the electrical load of each of the computing units is sensed and monitored.

In one form, the method further comprises:

directing the cooling fluid from the one or more outlets to a cooling system to subject the cooling fluid to a cooling process, to thereby lower the temperature of the cooling fluid from Tr to a desired supply temperature Ts; and

WO 2018/076056

PCT/AU2017/051172 directing the cooling fluid from an outlet of the cooling system back to the one or more inlets of the space.

In one form, the method further comprises:

receiving an adjustable differential pressure dPs set-point of the cooling fluid within and outside of the space, obtaining an actual differential pressure dPa measurement of the cooling fluid within the space and outside of the space, and controlling the flow of cooling fluid entering the space via the one or more inlets.

In one form, the step of controlling the flow of cooling fluid includes controlling the operation of one or more supply fans.

In one form, the step of controlling the flow of cooling fluid entering the space includes:

determining a required supply fan speed based on the differential pressure set point dPs and the actual differential pressure dPa, and controlling the supply fan to operate at the required supply fan speed.

Preferably, the supply fan’s speed is increased when dPs is greater than dPa, and the supply fan’s speed is decreased when dPs is less than dPa. In alternative forms which include a plurality of supply fans, one or more fans can be activated or deactivated to increase or decrease the dPa. Other modifications of fan operation are also possible, e.g. by adjusting the blade angle of attack, such as by feathering a fan’s blades to cause a change in dPa.

In one form, the cooling system includes a computer room air conditioning unit or air handling unit (herein: CRAC unit or AHU) for cooling the cooling fluid from Tr to Ts, and an associated cooling controller for controlling the operation of the CRAC or AHU unit.

In one form, the method further includes:

delivering electrical power to the one or more computing units.

WO 2018/076056

PCT/AU2017/051172

The electrical power may be provided from a power distribution unit, a power rail, a bus-duct tap-off box, or a combination of the above.

In one form, the one or more computing units are mounted within one or more standing rack cabinets. The rack cabinets can be arranged to stand side by side in a plurality of spaced rows. Preferably, the cooling fluid is introduced into the space and then caused to flow generally horizontally from the front of the cabinet to the back of the cabinet.

In one form, the one or more computing units are arranged in an aisle containment structure. For example, the rows of cabinets are arranged in groups, wherein each group may include two rows of cabinets that are arranged in a back to back relationship. Alternatively the group may include one row of cabinets and containment panelling. The cooling fluid is directed to flow onto the first row of the computing units, generally referred to as the cold aisle, and is then collected from the second row, generally referred to as the hot aisle. Preferably the one or more outlets of the space are associated with hot aisles. The system may have containment panels to ensure that cold aisle supply fluid is segregated from the hot aisle return fluid.

In one form, the method further may include:

detecting an occupancy status of the one or more hot aisles;

adjusting the temperature control data upon the detection of an occupant at the one or more hot aisles temporarily;

re-setting the temperature control data back to default value upon detection of the occupant leaving the hot aisle and/or after a predetermined time period has elapsed.

In one form, the step of detecting the occupancy status can include:

receiving an input indicating the occupancy status.

The input can be supplied by a manual input, e.g. operation of a push button, switch or software based input or other control, an input from a sensor which detects the presence of a person, proximity of a person or the like.

Preferably the one or more computing units are server racks that are accommodated in a data centre or a server room.

WO 2018/076056

PCT/AU2017/051172

In a second aspect, the invention provides a cooling regulation system for a plurality of computing units accommodated in a space by a cooling fluid, said cooling fluid being introduced into the space via one or more inlets and being arranged to leave the space via one or more outlets, said system comprising:

sensor for measuring an electrical load of one or more of the computing units, a controller, said controller being configured to:

receive the sensor output, receive desired temperature control data, said desired temperature control data includes at least a differential temperature set-point dT and/or a desired exit temperature of the cooling fluid Tr, determine a required flow rate of the cooling fluid based on the sensed electrical load and the received temperature control data, regulate the flows of the cooling fluid to achieve the required flow rate Vr.

The system can include at least one control valve associated with said outlet from the space. The controller may regulate the flows of the cooling fluid to achieve the required flow rate Vr by sending a control signal to said at least one control valve.

The control valve can have a plurality of adjustable open positions allowing the cooling fluid to leave the space at different flow rates, under control form the controller.

Preferably the flow rate of fluid through the control valve is calibrated. Calibration can be performed at a plurality of cooling fluid pressures or over a continuous range of fluid pressures. In preferred forms of the present invention this enables the method to be applied regardless of the pressure at which the cooling fluid is admitted to the space.

In one form, each of the one or more outlets is associated with a computing unit, or a group of computing units. In one form, the one or more outlets are positioned at a location generally adjacent e.g. above the one or more computing units.

In one form, the valve is a damper operated by a motorized modulating actuator. Preferably the damper is fail-safe open when there is a loss of power, loss of control signal, or a disaster such as fire, earthquake, and similar.

WO 2018/076056

PCT/AU2017/051172

In one form, the system further includes:

a cooling system for generating the cooling fluid required for the space, said cooling system being configured to:

receive an adjustable differential pressure dPs set-point of the cooling fluid within and outside of the space, obtain an actual differential pressure dPa measurement of the cooling fluid within the space and outside of the space, and controlling the flow of cooling fluid entering the space via the one or more inlets.

Preferably, the supply fan’s speed is increased when dPs is greater than dPa, and the supply fan’s speed is decreased when dPs is less than dPa.

In one form, the system further includes:

means for guiding the cooling fluid from the one or more outlets of the space to an inlet of the cooling system, and means for guiding the cooling fluid from an outlet of the cooling system to the one or more inlets of the space.

In a third aspect, the invention provides a cooling fluid circulation system for a plurality of computing units accommodated in a space by a cooling fluid, said cooling fluid being introduced into the space via one or more inlets and being arranged to leave the space via one or more outlets, said system comprising:

WO 2018/076056

PCT/AU2017/051172 a cooling system for generating the cooling fluid required for cooling the space, said cooling system being configured to:

receive an adjustable differential pressure dPs set-point of the cooling fluid within and outside of the space, obtain an actual differential pressure dPa measurement of the cooling fluid within the space and outside of the space, control the flow of cooling fluid entering the space via the one or more inlets based on dPs and dPa;

receive the sensor output, receive desired temperature control data, said desired temperature control data includes at least a differential temperature set-point dT and/or a desired exit temperature of the cooling fluid Tr, determine a required volume flow rate of the cooling fluid based on the sensed electrical load and the received temperature control data, regulate the flows of the cooling fluid exiting the space to achieve the required flow rate Vr.

determining a required supply fan speed based on the differential pressure set point dPs and the actual differential pressure dPa, and controlling the one or more supply fans to operate at the (or respective) required supply fan speed.

WO 2018/076056

PCT/AU2017/051172

The system can further include at least one control valve associated with said outlet from the space. The controller may regulate the flows of the cooling fluid to achieve the required flow rate Vr by sending a control signal to said at least one control valve.

The control valve includes a plurality of adjustable open positions allowing the cooling fluid to leave the space at different flow rates, under control from the controller.

As used herein, except where the context requires otherwise, the term comprise and variations of the term, such as comprising, comprises and comprised, are not intended to exclude further additives, components, integers or steps.

Further aspects of the present invention and further embodiments of the aspects described in the preceding paragraphs will become apparent from the following description, given by way of example and with reference to the accompanying drawings.

Brief description of the drawings

Figure 1 is a plan view of a space containing a plurality of computing units, which may be cooled using an embodiment of the present invention. The space includes cooling fluid inlets and fluid outlets from the space as well as a containment structure to assist in controlling the flow of cooling fluid.

WO 2018/076056

PCT/AU2017/051172

Figure 1A shows a side view, along line A-A in Figure 1 to further illustrate the inlet and outlet to the space, and the path of the cooling fluid through the computing units.

Figures 2 and 2A illustrate views corresponding to that of Figures 1 and 1A, but illustrate a different space with computing units arranged into rows and containment structure defining a “cold aisle” and a “hot aisle”.

Figures 3 and 3A illustrate views corresponding to the previous figures, but differ in that the cooling fluid inlet to the space is from an underfloor plenum, and the outlet from the space is into an overhead plenum.

Figures 4 and 4A illustrate a variation on the embodiment of Figures 3 and 3A, in which multiple inlets and outlets are provided.

Figures 5 and 5A show views corresponding to the previous figures, however the inlet to the space is in a wall, and the outlets move cooling fluid into an overhead plenum.

Figures 6 and 6A illustrate a further embodiment, which again uses hot aisle containment. However, it differs from the previous embodiments in that the cooling fluid inlet is through a wall of the space and the outlets are into an overhead plenum.

Figure 7 is a schematic view of the space of Figure 1, which further includes a cooling regulation system according to an embodiment of the present invention.

Figure 8 is a flow chart showings steps in a process performed by the system of Figure 7.

Figure 9 is a schematic cross-sectional side view of a space containing a plurality of computing units and cooling fluid containment structure which define a hot aisle and cold aisle. The figure also schematically illustrates the components of cooling regulation system according to an embodiment of the present invention and a CRAC.

Figure 10 illustrates a further embodiment of the system of Figure 9 which additionally includes a sub-system for occupancy sensing.

Figure 10A is a top view of the illustration of Figure 10 better showing the operation of the occupancy sensing system.

WO 2018/076056

PCT/AU2017/051172

Detailed description of the embodiments

Illustrative embodiments of the aspects of the present invention will now be described. The illustrative embodiments will be described in connection with performing cooling regulation for a plurality of computing units. The computing units can take the form of any computing device, such as servers, hard drives, other data processing or data storage equipment, networking equipment such as routers, switches or the like; and any of their associated peripherals or components. The computing units may be collocated in one or more rack cabinets. As would be appreciated by those skilled in the art, a rack cabinet is an enclosure in which computing hardware may be mounted. Such a configuration offers advantages in terms of convenience in so far as the computing units are provided as standardised modules which are mounted in the cabinet. The computing units will be located, in a space, such as a room or partitioned zone within a room. In order to cool the computing units accommodated within the space, a cooling fluid is introduced into the space via one or more inlets. The cooling fluid then passes through respective computing units or circulates around them, and passes out of the space via one or more outlets. In preferred forms of the present invention, to pass between the inlet and an outlet, the cooling fluid must flow through or around inputting units e.g. it must flow through a grill in the front of a rack cabinet, through a computing unit and out of the cabinet. Such a flow path can be enforced by the provision of containment panelling or the like to stop air from cycling back towards the inlet and mixing with the newly introduced, cold, cooling fluid. In most embodiments of the present invention, the cooling fluid will be provided by a computer room air conditioner (CRAC) or an air handling unit (AHU). The cooling fluid supply system will typically recirculate air that leaves the outlet of the space and cool it before it is at least partly reintroduced to the inlet.

Figures 1-6A illustrate four exemplary spaces, each housing a plurality of computing units in which embodiments of the present invention can be implemented. In each case the space includes one or more cooling fluid inlets, and one or more cooling fluid outlets. A flow path between the inlets and outlets is provided that passes through the computing units. The flow path ensures the cooling fluid passes through the units before exiting the space to maximise cooling efficiency. As would be appreciated by those skilled in the art, the nature and configuration of spaces in which computing units

WO 2018/076056

PCT/AU2017/051172 can be housed are myriad, and these examples are merely a small selection. In the figures, elements performing a common function have the same reference numerals.

Turning firstly to Figure 1 and 1A which show a top view and a side view from a position A-A in Figure 1. Figures 1 and 1A illustrate a space 10, in which is located a schematic block representing a plurality of computing units 20. The plurality of computing units 20, may include a plurality of computing units housed within a rack cabinet. The space 10 includes an air inlet 30 and an air outlet 40. Cooling fluid, which in this case is air, enters the space 10 by the inlet 30, passes through the plurality of computing units 20, and exits the space 10 via the outlet 40. The cooling fluid is forced to pass through the computing units 20 by the presence of the containment partition 50. Accordingly, the space 10 has a cold side 60 into which cooling fluid is supplied, before cooling the computing units 20, and a hot side 70 into which the cooling fluid passes after passing through the computer units 20. As would be appreciated by those skilled in the art, as a cooling fluid passes through and over and around computing units, heat is transferred from the hot computing units to the cooling fluid. The cooling fluid then passes out of the space 10 to extract the heat from the system within the space 10.

Figures 2 and 2A show a second embodiment of a space 10 containing a plurality of computing units 20. In this example the plurality of computing units 20 are split amongst multiple cabinets. Each cabinet may include one or more computing units, or even, in some examples some of the enclosures may temporarily have no computing units within them. In such a case, these empty enclosures will typically be blocked and effectively operate as an extension of the containment partition 50 of the space 10. In this example, the computing units 20 are in two rows, leaving an aisle between them. Accordingly, such an arrangement may be known as having a “hot aisle” 70 and a “cold aisle” 60. In the example of Figures 2 and 2A the space 10 is provided with two inlets 30 and one outlet 40.

Figures 3 and 3A show a further embodiment of the space 10 in which an embodiment of the present invention can be used. This embodiment differs from the previous embodiments in that the cooling fluid is delivered to the inlet from an underfloor plenum 80 and is drawn away from the outlet 40 through an overhead plenum 90. Air may be introduced to the underfloor plenum 80 by plenum inlet 85 and exhausted from the overhead plenum 90 by a plenum exhaust 95. A containment partition 50 is also

WO 2018/076056

PCT/AU2017/051172 provided to ensure that the cooling fluid flows from the inlet 30 through the computing units 20 to the outlet 40.

Figures 4 and 4A illustrate a variation on the embodiment of Figures 3 and 3A and include a plurality of separate computing units 20 (e.g rack cabinets) and the plurality of inlets 30 for delivering cooling fluid to the cold side 60 of the space 10. A plurality of outlets 40 are also provided. Again, an underfloor plenum 80 is used for delivering air to the inlets 30 and an overhead plenum 90 is used to remove cooling fluid from the outlet 40. In this embodiment, each rack cabinet (possibly each computing unit) is provided with a corresponding outlet 40. However, there are fewer inlets than rack cabinets. However, it should be noted that any number of inlets 30 and outlets 40 could be provided. It is preferred however that each outlet 40 generally corresponds to a defined subset of the plurality of computing units. The correspondence may be that each outlet 40 corresponds to a rack cabinet, individual computing units, or a certain number of rack cabinets or computing units. The purpose for this correspondence is so that airflow through a particular subset of computing units (e.g. a rack cabinet) can be more closely controlled, based on their corresponding power usage.

Figures 5A and 5B illustrate a further space 10 in which an embodiment of the present invention may be used. In this example, an overhead plenum identical to that of Figure 4 and 4A is employed. Air is introduced into the space 10 via an inlet 30 mounted on a wall and exit the hot aisle 70 of the space 10 via outlet 40 into the overhead plenum 90.

Figures 6 and 6A illustrate another hot aisle containment arrangement. In this example, air enters the cold aisle 60 by the inlet 30 and passes through the computing units 20 into the hot aisle 70, as in previous embodiments. The hot cooling fluid exits the hot aisle 70 via a series of outlets 40 into the overhead plenum 90. As in the previous embodiments, each outlet 40 generally corresponds to a pair of rack units containing computing units 20. As can be seen, hot aisle 70 in the present example is confined at both ends by a containment partition 50 to define a smaller enclosure within a space 10.

Figure 7 illustrates a space 10 that is the same as that illustrated in Figure 1 and 1A, but which additionally includes a cooling regulation system according to an embodiment of the present invention. The cooling regulation system 700 comprises a controller 710, a sensor system 720, and a valve 760 in the form of a controllable

WO 2018/076056

PCT/AU2017/051172 damper at the outlet 40. In effect the valve 760 changes the flow resistance of the outlet with which it is associated and thereby control the flow rate through it. The cooling fluid, which in this case is air, is provided by a CRAC unit 730. The CRAC unit 730 provides air at a temperature Ts to the inlet 30 and causes a pressure differential dP that drives the cooling air through the airflow path of the cooling system. In particular it causes a pressure differential dP between the inlet-side 60 of the space 10 and the exhaust side of the outlet 40. The sensor system 720 includes a sensor 740 for determining the power consumption (preferably instantaneous or short term average electrical load) of the computing units 20 located within the space 10. The sensor output is provided to the controller 710. The electrical load can be sensed in any known manner, e.g. by: a current sensor, a voltage sensor, a power meter, or a combination of the above. Preferably the electrical load is sensed continuously, but may be sensed on an ad-hoc basis, periodically, or in accordance with a predetermined sensing schedule. As will be appreciated from the following description the electrical load of each of the computing units may be sensed separately or on a group basis (e.g. a whole rack) depending on how many computing units are associated with a given valve to be controlled.

The control unit 710 also receives an input of the differential pressure dP driving the cooling fluid through the cooling circuit. This can optionally be provided by a pair of pressure sensors 742 comprising part of the sensor system 720. In this example the pressure sensors monitor the pressure difference between the inlet-side 60 of the space 10 and outside the exhaust of the outlet 40. The controller 710 also receives control data which sets the desired level of cooling to be provided to the computing units 20. The temperature control data can comprise either a set point temperature, being the temperature of the air after it has passed through the computing units 20, or a differential temperature set point dT, being the temperature difference between air at the “cold aisle” 60 and the “hot aisle” 70 of the space 10. The control unit 710 determines a cooling fluid flow rate needed to achieve the target temperature data control point being either dT or exit temperature, given the current power consumption of the computing units as determined by the power sensor 740. Next, the controller determines the necessary setting for the valve 760 given the differential pressure dP driving the cooling fluid through the outlet 40, and generates a control signal on control line 750 to adjust the position of an adjustable damper of the valve 760.

WO 2018/076056

PCT/AU2017/051172

The valve 760 can be continuously adjustable between a maximally open position to a maximally closed position. The maximally closed position may be fully closed. In some embodiments the valve may instead have a plurality of discrete adjustable open positions allowing the cooling fluid to leave the space at different flow rates.

As can be seen from the various embodiments set out in figures 1 to 6A, the number, size and position of outlets relative to the computing units may vary between installations. Embodiments of the present invention can be arranged to handle such a variation. In one form, the one or more outlets are positioned at a location generally adjacent e.g. above the one or more computing units. For example, in some forms each valve 760 can be associated with a single computing unit. The control system 710 can control the valve to individually control the flow rate of cooling fluid through the computing unit, based on the power consumption of individual computing units. More commonly however, a set of computing units (e.g. a number of computing units housed in a rack cabinet) will be have a shared outlet and valve 760 to control fluid flow through that set, based on their combined power usage. This could be achieved by placing an outlet near the hot air exhaust of each rack cabinet. In other embodiments a single outlet and valve 760 can be assigned to multiple rack cabinets.

As noted above, in this example the valve 760 is a damper with movable vanes operated by a motorized modulating actuator. In preferred forms the valve/damper is fail-safe open. Thus, in the event of a loss of power, or loss of control signal the damper moves to an open position to permit maximal flow. In order to determine the correct damper position setting, the flow rate through the damper at a plurality of different pressure differentials must be known. This can be known in a number of ways, e.g. by performing empirical tests of flow and determining an equation modelling flow over a range of differential pressures, by storing discrete values in a database and using it (or a subset of it) as a lookup table, or using theoretical modelling techniques.

It should be noted that in this example the control system 710 is separate to the cooling fluid supply system i.e. the CRAC, and does not exercise any direct control over the CRAC, i.e. it does not set the temperature of the input air or the target dP driving the cooling fluid around the system. Instead, the control system 710 controls the airflow through the computing units to achieve either a desired exit air temperature or differential temperature across the computing units, in order to achieve the necessary

WO 2018/076056

PCT/AU2017/051172 cooling of the computing resources. In preferred embodiments, because the system knows the level of cooling needed, based on the measured power consumption, and the level of cooling actually achieved by measuring output temperature, it is possible to apply the minimum possible cooling whilst still operating the computing units within their operational specifications. Moreover the operation of the system can cause a feedback loop that reduces energy use by the CRAC unit as it may also measure pressure differential to control its fan speed. For example, when power consumption of the computing units drops, the level of cooling needed also drops, the controller 710 will restrict flow though the outlet 40 so that the exit temperature of air remains suitably high. This restriction in airflow causes the CRAC to sense that it is oversupplying air, because the differential pressure is elevated, and in response the CRAC will reduce its output.

As noted the desired exit temperature and/or differential temperature control setpoint is determined by an operator, and may be based on: operational tolerances of the one or more computing units and or operator choice. The operator’s choice may be based on experience or operational tolerances of the equipment with a safety margin built in. In some cases the operator of a data centre that houses computing units owned or managed by different entities (e.g. businesses or users) may allow the owner/manager to specify the maximum temperature that a given piece of equipment may be exposed to.

In order to save cooling energy, or use the cooling capacity more efficiently, using embodiments of the present invention, the exit temperature can be set closer to the desired equipment tolerance level or user specified limits, meaning that the minimum amount of cooling can be used. As should be appreciated this generally means increasing the exit air temperature or maximising actual dT achieved.

To aid in this process, some embodiments of the present invention can also measure dT or exit temperature. This can be performed by monitoring the supply temperature of the cooling fluid as it is being introduced into the space. The exit temperature of the cooling fluid can also be measured after passing through a computing unit or as it exits the space from the one or more outlets. In response to such measurements the control system 710 can adjust the required flow rate based on the monitored temperature(s).

WO 2018/076056

PCT/AU2017/051172

Figure 8 illustrates a flowchart showing the steps in the control process performed by the control system 700 of Figure 7. The process 800 begins at step 801 by receiving (or sensing) data representing the electrical load drawn by the computing units 20. As there is only one outlet in this example, a combined electrical load for all computing units 20 will suffice. In step 802 temperature control data is received. This will typically be received from an interface adapted to receive a user input. Next, based on the electrical load being consumed, and the temperature data, the required fluid flow rate Vr is determined in step 803. Then in step 804 the fluid flow rate is regulated towards Vr. This can involve the following sub-steps:

In sub-step 810, the differential pressure dP across the system can be received or determined. This data is used in sub-step 812 to determine the valve positions, e.g. damper opening position, needed to achieve the required fluid flow rate Vr. A control signal representing the required damper position is transmitted to the damper 760 in sub-step 814. The adjustable damper 760, if necessary, then adjusts its position in accordance with the control signal.

Figure 9 illustrates a further embodiment applied to a space, similar to that of figures 5 and 5A. The space 10 houses a plurality of individual computing units 20.1 to 20.3 mounted in a cabinet 20.4. Cooling fluid (e.g. refrigerated air) enters the space 10 via an inlet 30. The air flows through the front of the cabinet 20.4 to cool the computing units 20.1 to 20.3 into a hot aisle 70. The hot aisle 70 is separated from the cold aisle 60 by containment panelling 50. The heated cooling fluid then passes out of the outlet 40. The outlet is fitted with a variable position damper 760 to control flow out of the space 10 into the overhead plenum 90. The position of the damper is controlled by the controller 710 as described in connection with figure 7 and 8. Power is supplied to the plurality of computing units 20 from a power supply system 715. The power supply system may be a power distribution unit, a power rail, a bus-duct tap-off box, or the like. Power usage is monitored by the sensor 740, which sends a power level input to the controller 710.

The controller 710 also receives input from a pressure sensing system 734. The same pressure sensing system may form part of the CRAC unit 730, and receive pressure input from a pair of pressure sensors 742, one of which is located in the cold aisle 60 of the space 10, and one after the outlet 40. In this case the second pressure sensor is located at the exhaust from the overhead plenum 90.

WO 2018/076056

PCT/AU2017/051172

In this example, the controller 710 additionally includes a pair of temperature sensors 910 and 920. Optional sensor 920 measures the inlet air temperature. Sensor 910 measures the outlet temperature. This data is used by the controller to measure the temperature differential caused by dispersing heat from the computing units into the cooling fluid. The data is used by the controller to monitor system cooling performance and provides a feedback loop to adjust the damper 760 if more or less flow is needed to achieve the desired temperature data.

The CRAC unit 730 includes a CRAC controller 731, cooling coil 732, supply fan 733 and pressure sensing system 734. Temperature sensors are also provided but not shown. The controller 731 adjusts flow rate based on the pressure differential measured by the sensing system and adjusts the refrigerant flow in the coil 732 based on a user defined set-point temperature and flow rate. In one form, the flow rate is controlled by adjusting the speed of the supply fan 733.

The CRAC unit 730 directs the return cooling fluid received from the outlet(s) 40 to a cooling system 732 to subject the cooling fluid to a cooling process, The CRAC cools the return fluid (in a recirculating system) or input fluid (in a system that draws fresh air) from Tr to a desired supply temperature Ts and directs it to the inlet 30.

The CRAC controller 731 receives an adjustable differential pressure dPs setpoint and an actual differential pressure dPa measurement from the pressure sensing system 734. The CRAC controller 731 uses the differential pressure set point dPs and the actual differential pressure dPa to control the flow of cooling air entering the space. In one embodiment the CRAC controller 731 receives an adjustable differential pressure dPs set point and an actual differential pressure dPa measurement from the pressure sensing system 734, and determines the supply fan 733 speed directly based on dPs and dPa received to control the rate at which the cooling fluid is introduced into the space. As will be appreciated, the CRAC can include multiple fans, which can be controlled independently or together according to dPa and dPs. Controlling the fan speed can include stopping or starting one or more fans. In some embodiments the CRAC could be controlled on the basis of air supply speed Vs instead or in addition to differential pressure. In this case Vs can be monitored at or near the inlet(s) to the space, and the one or more supply fans of the CRAC are controlled to supply air at a set point Vs.

WO 2018/076056

PCT/AU2017/051172

As noted above, the cooling regulation system may be separate from the cooling fluid supply system i.e. the CRAC. Advantageously this allows a system according to the present invention to be retrofitted to existing computer rooms or data centres with minimal disruption, and no need to interface with the cooling system. However the systems can be combined into a cooling fluid circulation system that perform both functions. In such systems the same controller can control both the cooling process and cooling regulation process through the computing units themselves.

A further embodiment of the present invention is shown in figures 10 and 10a. This embodiment is identical to that of figure 9, except that the controller 710 additionally receives an occupancy status input. The input may come from either a user input, such as a push button 1000, or a sensor such as an occupancy sensor (e.g. PIR or other motion sensor) 1010. The motion sensor 1010 detects the presence of a person within a field of view 1020. The occupancy sensing input is used by the controller to modify the cooling fluid flow based on the occupancy of the hot aisle 70.

The present inventors have identified that personnel may need to enter the hot aisle from time to time, and it may be unpleasant or potentially dangerous for personnel to do so due to the elevated temperature of the hot aisle. For example the hot aisle may operate at 45 degrees Celsius or higher. Thus In the present embodiment the system detects an occupancy status of a hot aisle, and adjusts the cooling fluid flow temporarily to account for the occupancy. This may involve fully opening the valve/damper 760 to allow maximum cooling fluid flow into the hot aisle to lower the temperature in the hot aisle. Alternatively this may involve temporarily changing the temperature data used to control the Vr of the system to lower the temperature in the hot aisle 70. The temperature control data can be re-set back to the previous value or a default value upon detection of the occupant leaving the hot aisle and/or after a predetermined time period has elapsed.

It will be understood that the invention disclosed and defined in this specification extends to all alternative combinations of two or more of the individual features mentioned or evident from the text or drawings. All of these different combinations constitute various alternative aspects of the invention.

Claims

1. A method of cooling a plurality of computing units accommodated in a space using a cooling fluid, said cooling fluid being introduced into the space via one or more inlets and being arranged to leave the space via one or more outlets, said method comprising:

2. The method of claim 1, wherein the desired temperature control data includes at least a differential temperature set-point dT and/or a desired exit temperature of the cooling fluid Tr.

3. The method of claim 1, wherein the step of regulating the flows includes causing the cooling fluid to leave the space at, at least one of:

the required flow rate Vr; and the desired exit temperature Tr.

4. The method of claim 2, wherein one or both of:

is adjustable.

5. The method of claim 2, wherein the desired exit temperature and/or differential temperature control set-point is determined by an operator and/or based on operational tolerances of the one or more computing units.

6.

The method of claim 1, further including:

WO 2018/076056

PCT/AU2017/051172 monitoring the supply temperature of the cooling fluid as it is being introduced into the space, and/or monitoring the exit temperature of the cooling fluid as it exits the space from the one or more outlets, and adjusting the required flow rate based on the monitored temperature(s).

7. The method of claim 1, wherein at least one of the outlets is provided with a control valve to regulate the flow of cooling fluid leaving the outlet.

8. The method of claim 7, wherein the control valve has a plurality of adjustable open positions allowing the cooling fluid to leave the space at different flow rates.

9. The method of claim 1, wherein the method further includes:

based on the required flow rate of the cooling fluid, adjusting the open position of the control valve.

10. The method of claim 7, wherein the control valve is a damper operated by a motorized modulating actuator.

11. The method of claim 1, wherein at least one of the inlets is provided with a control valve to regulate the rate of cooling fluid being introduced into the space.

12. The method of claim 1, wherein the electrical load is sensed by a current sensor, a voltage sensor, a power meter, or a combination of the above.

13. The method of claim 1, wherein the electrical load is sensed continuously, on an ad-hoc basis, periodically, or in accordance with a predetermined sensing schedule.

14. The method of claim 1, further comprising:

directing the cooling fluid from the one or more outlets to a cooling system to subject the cooling fluid to a cooling process, to thereby lower the temperature of the cooling fluid from Tr to a desired supply temperature Ts; and directing the cooling fluid from an outlet of the cooling system back to the one or more inlets of the space.

WO 2018/076056

PCT/AU2017/051172

15. The method of claim 1, further comprising:

16. The method of claim 15, wherein the step of controlling the flow of cooling fluid entering the space includes:

controlling the operation of one or more supply fans to control the flow of cooling fluid entering the space.

17. The method according to claim 16, which includes:

determining a required supply fan speed based on the differential pressure set point dPs and the actual differential pressure dPa, and controlling the operation of the supply fan based on the determined required supply fan speed.

18. The method of claim 1, further including:

delivering electrical power to the one or more computing units from a power distribution unit, a power rail, a bus-duct tap-off box, or a combination of the above.

19. The method of claim 1, further including:

detecting an occupancy status of the one or more hot aisles;

20. The method of claim 19, wherein the step of detecting the occupancy status includes:

WO 2018/076056

PCT/AU2017/051172 receiving an input indicating the occupancy status, wherein the occupancy status is supplied by a manual input, e.g. operation of a push button, switch or software based input or other control, or an input from a sensor which detects the presence of a person, or proximity of a person.

21. A cooling regulation system for a plurality of computing units accommodated in a space by a cooling fluid, said cooling fluid being introduced into the space via one or more inlets and being arranged to leave the space via one or more outlets, said system comprising:

22. The cooling regulation system of claim 21, further includes at least one control valve associated with said outlet(s) from the space, and the controller regulates the flows of the cooling fluid through the outlet(s) to achieve the required flow rate Vr by sending a control signal to said at least one control valve.

23. The cooling regulation system of claim 22, wherein the control valve includes a plurality of adjustable open positions allowing the cooling fluid to leave the space at different flow rates, under control from the controller.

24. The cooling regulation system of claim 22, wherein the control valve is a damper operated by a motorized modulating actuator.

25. The cooling regulation system of claim 21, further including:

WO 2018/076056

PCT/AU2017/051172 means for guiding the cooling fluid from the one or more outlets of the space to an inlet of the cooling system, and means for guiding the cooling fluid from an outlet of the cooling system to the one or more inlets of the space.

26. A cooling fluid circulation system for a plurality of computing units accommodated in a space by a cooling fluid, said cooling fluid being introduced into the space via one or more inlets and being arranged to leave the space via one or more outlets, said system comprises:

a cooling system for generating the cooling fluid required for cooling the space, said cooling system being configured to:

receive an adjustable differential pressure dPs set-point of the cooling fluid within and outside of the space, obtain an actual differential pressure dPa measurement of the cooling fluid within the space and outside of the space, control the flow of cooling fluid entering the space via the one or more inlets;

27. The system of claim 26, wherein the step of controlling the flow of cooling fluid includes controlling the speed of one or more supply fans.

WO 2018/076056

PCT/AU2017/051172

28. The method of claim 14, wherein the cooling system includes a computer room air conditioning unit or air handling unit (herein: CRAC unit or AHU) for cooling the cooling fluid from Tr to Ts, and an associated cooling controller for controlling the operation of the CRAC or AHU unit.

5

29. A method as claimed in any one of claims 1 to 20 or 28 wherein the cooling fluid is air.

30. A system as claimed in any one of claims 21 to 27 wherein the cooling fluid is air.