WO2018207306A1

WO2018207306A1 - Electronic device for immersion cooling, and processor module for immersion cooling

Info

Publication number: WO2018207306A1
Application number: PCT/JP2017/017837
Authority: WO
Inventors: 齊藤　元章
Original assignee: 株式会社ＥｘａＳｃａｌｅｒ
Priority date: 2017-05-11
Filing date: 2017-05-11
Publication date: 2018-11-15
Also published as: JP6494773B1; JPWO2018207306A1

Abstract

Provided are an electronic device and a processor module that, even if a relatively tall memory module is mounted, can experience no reduction in packing density, or can further increase packing density. This processor module includes a first circuit board and a second circuit board that each have, on one surface of the board, a processor mounting region and a memory mounting region. At least one processor is mounted to the processor mounting region, and a plurality of memory modules arranged in a comb shape are mounted to the memory mounting region. The one surface of the first circuit board and the one surface of the second surface board are brought together in a facing manner. The first circuit board and the second circuit board are positioned such that the processor mounting region and the memory mounting region of the first circuit board respectively face the processor mounting region and the memory mounting region of the second circuit board, and such that tip end sections of the plurality of memory modules of the first circuit board, said memory modules being arranged in a comb shape, and tip end sections of the plurality of memory modules of the second circuit board, said memory modules being arranged in a comb shape, are alternatingly arranged, with gaps created between neighboring memory modules. A third circuit board is positioned in a space that is formed by upper surfaces of the processors of the first circuit board and the second circuit board facing one another.

Description

Immersion cooling electronic device and immersion cooling processor module

The present invention relates to an electronic device, and more particularly to an electronic device that is immersed in a cooling liquid in a cooling device and directly cooled. The present invention also relates to an immersion cooling processor module that is preferably used in the electronic apparatus. In this specification, an electronic device generally refers to an electronic device that requires ultra-high performance operation or stable operation such as a supercomputer or a data center and generates a large amount of heat from itself, but is not limited thereto. It is not a thing.

One of the biggest issues that determine the limits of the performance of supercomputers in recent years is power consumption, and the importance of research on power saving performance of supercomputers has already been widely recognized. That is, the speed performance (Flops / W) per power consumption is one index for evaluating a supercomputer. Further, in the data center, it is said that about 45% of the power consumption of the entire data center is spent for cooling, and there is a growing demand for reduction of power consumption by improving cooling efficiency.

Conventionally, air cooling and liquid cooling have been used to cool supercomputers and data centers. The liquid cooling type is generally considered to have good cooling efficiency because it uses a liquid that has a heat transfer performance far superior to that of air. In particular, an immersion cooling system that uses a fluorocarbon-based coolant is superior to electronic device maintenance (specifically, for example, adjustment, inspection, repair, replacement, expansion, etc.) as compared with a system that uses synthetic oil. In recent years, it has attracted attention.

The present inventor has already developed a small-sized immersion cooling device with excellent cooling efficiency for a small-scale immersion cooling supercomputer. This apparatus is applied to and operated in a small supercomputer “Suiren” installed in the High Energy Accelerator Research Organization (Non-patent Document 1).

In addition, the present inventor has proposed various improved immersion cooling devices that can significantly increase the mounting density in an electronic device that is immersion-cooled (Patent Document 1, Non-Patent Document 2, Non-Patent Document 2, Patent Document 3, Non-Patent Document 4).

Japanese Patent No. 6042587

Among the above prior art documents, Non-Patent Document 3 describes an immersion cooling electronic device “Multi-Xeon Server Brick” capable of mounting 16 Xeon processors (“Xeon” is a trademark of Intel Corporation) at high density. Is disclosed. According to this technique, two Xeon processors are connected according to QPI (Quick Path Interconnect of Intel Corporation), so that one electronic device includes eight Multi-Xeon server nodes. Eight general-purpose 16GB or 32GB DDR4 (Double-Data-Rate4) VLP DIMM (Very Low Profile Dual Dual Inline Memory Module) is installed in the memory per processor. Can do. Patent Document 1 discloses a configuration example of a substrate group that realizes the immersion cooling electronic device disclosed in Non-Patent Document 3.

As described above, the immersion cooling electronic device disclosed in Non-Patent Document 3 is mounted with a 32 GB DDR4 VLP DIMM, which is a very low-profile memory module. However, it may be requested to further increase the memory capacity per processor. As a method for responding to such a request, it is conceivable to increase the number of memory modules of 32 GB DIMM or to install a memory module having a larger capacity, for example, 64 GB DIMM. However, when increasing the number of memory modules, it is necessary to increase the mounting area on the board for mounting the memory modules, the board becomes larger and the volume of the combined group of boards, that is, the brick, increases, and the mounting density is increased. There is a problem of being lowered. On the other hand, when installing a larger capacity 64GB DIMM, the general-purpose 64GB DIMM has a higher Standard Height than the VLP, so the distance between the boards should be reduced so that the memory module does not interfere with the surface of another board. Need to be separated. For this reason, there is a problem that the volume of the brick increases and the mounting density decreases.

Therefore, in the electronic equipment applied to the immersion cooling device, the mounting density does not decrease even if Standard Height or other relatively tall memory modules are mounted, or the mounting density can be further increased. It is desired to develop an electronic device and a processor module having a new configuration that is suitably used for the electronic device.

In addition, it is desired to develop a technology that utilizes the space that can be created inside the processor module.

In order to solve the above problems, according to one aspect of the present invention, there is provided a processor module for immersion cooling applied to an electronic device that is immersed in a cooling liquid in a cooling device and directly cooled.
A first circuit board and a second circuit board, each of which has a processor mounting area and a memory mounting area on one surface thereof, wherein at least one processor is mounted on the processor mounting area; A plurality of memory modules arranged in a comb shape are mounted in a mounting area, and the one surface of the first circuit board and the one surface of the second circuit board are combined face to face. A first circuit board and a second circuit board;
A connector for electrically connecting the first circuit board and the second circuit board;
Including
The first circuit board and the second circuit board are the processor mounting area and the memory mounting area of the first circuit board, and the processor mounting area and the memory mounting area of the second circuit board, respectively. The front end portions of the plurality of memory modules arranged in a comb shape on the first circuit board and the front end portions of the plurality of memory modules arranged in a comb shape on the second circuit board, respectively, facing each other, The memory modules are aligned so that gaps are formed between adjacent memory modules.

In a preferred embodiment of the processor module for immersion cooling according to one aspect of the present invention, the memory mounting area includes a plurality of memory sockets for fixing individual memory modules, and the one of the first circuit boards. The distance H between the surface and the one surface of the second circuit board is (h ₁ + h ₂ ) with respect to the height h _{1 of} the memory module and the height h ₂ of the memory socket from the one surface. <H <2h ₁ should be satisfied.

Further, in a preferred embodiment of the immersion cooling processor module according to one aspect of the present invention, the memory module is a Standard Height memory module, and the module has a board surface of the memory module relative to the one surface. The memory socket may be inserted vertically or inclined.

Further, in a preferred embodiment of the immersion cooling processor module according to one aspect of the present invention, the processor of the first circuit board and the processor of the second circuit board are connected to each other. It is good to be connected through.

According to another aspect of the present invention, an electronic device that is immersed in a cooling liquid in a cooling device and directly cooled is a carrier substrate having a voltage input terminal that supplies a DC voltage for the electronic device. The voltage input terminal is electrically connected to the voltage output terminal of the power supply unit; and a carrier substrate;
A plurality of module connectors arranged on one surface of the carrier substrate;
A plurality of processor modules, each of the plurality of processor modules having a module connector plug electrically coupled to each of the plurality of module connectors;
When the electronic device is electrically connected to the power supply unit, a support member that supports the carrier substrate so as to be positioned above the power supply unit installed at the bottom of a cooling tank included in the cooling device;
Including
The processor module is
A first circuit board and a second circuit board, each of which has a processor mounting area and a memory mounting area on one surface thereof, wherein at least one processor is mounted on the processor mounting area; A plurality of memory modules arranged in a comb shape are mounted in a mounting area, and the one surface of the first circuit board and the one surface of the second circuit board are combined face to face. A first circuit board and a second circuit board;
A connector for electrically connecting the first circuit board and the second circuit board;
Including
The first circuit board and the second circuit board are the processor mounting area and the memory mounting area of the first circuit board, and the processor mounting area and the memory mounting area of the second circuit board, respectively. The front end portions of the plurality of memory modules arranged in a comb shape on the first circuit board and the front end portions of the plurality of memory modules arranged in a comb shape on the second circuit board, respectively, facing each other, The memory modules are aligned so that gaps are formed between adjacent memory modules.

In a preferred embodiment of an electronic device according to another aspect of the present invention, the support member may include a backboard or a frame structure in which the carrier substrate is fixed to one surface.

Also, in a preferred embodiment of the electronic device according to another aspect of the present invention, the backboard or frame structure is slidably supported by a plurality of support pillars that are vertically erected and fixed in the cooling tank. It is good to be done.

Further, according to still another aspect of the present invention, there is provided a processor module for immersion cooling applied to an electronic device that is immersed in a cooling liquid in a cooling device and directly cooled.
A first circuit board and a second circuit board, each of which has a processor mounting area and a memory mounting area on one surface thereof, wherein at least one processor is mounted on the processor mounting area; A plurality of memory modules arranged in a comb shape are mounted in a mounting area, and the one surface of the first circuit board and the one surface of the second circuit board are combined face to face. A first circuit board and a second circuit board;
A first connector for electrically connecting the first circuit board and the second circuit board;
A third circuit board disposed between the first circuit board and the second circuit board;
A second connector for electrically connecting the third circuit board and the first circuit board or the second circuit board;
Including
The first circuit board and the second circuit board are the processor mounting area and the memory mounting area of the first circuit board, and the processor mounting area and the memory mounting area of the second circuit board, respectively. The front end portions of the plurality of memory modules arranged in a comb shape on the first circuit board and the front end portions of the plurality of memory modules arranged in a comb shape on the second circuit board, respectively, facing each other, It is aligned so that there is a gap between adjacent memory modules,
The third circuit board is disposed in a space formed by the upper surfaces of the processors of the first circuit board and the second circuit board facing each other.

In a preferred embodiment of the immersion cooling processor module according to still another aspect of the present invention, the memory mounting area includes a plurality of memory sockets for fixing individual memory modules, and the first circuit board includes: distance H between the one surface of the said one face a second circuit board, the height h ₁ of the memory module from the one _surface, with respect to the height h ₂ of the memory socket, (h ₁ + H ₂ ) <H <2h ₁ is satisfied.

Further, in a preferred embodiment of the immersion cooling processor module according to still another aspect of the present invention, the memory module is a Standard Height memory module, and the module surface of the memory module is the one in which the substrate surface of the memory module is the one. The memory socket may be inserted perpendicularly or inclined with respect to the surface.

In a preferred embodiment of the immersion cooling processor module according to still another aspect of the present invention, the processor of the first circuit board and the processor of the second circuit board are connected to each other. It may be connected via a connection interface.

In a preferred embodiment of the immersion cooling processor module according to still another aspect of the present invention, the third circuit board is a circuit board on which a storage device and / or an I / O control chip set is mounted. It is good to be.

According to yet another aspect of the present invention, an electronic device that is immersed in a cooling liquid in a cooling device and directly cooled is
A carrier substrate having a voltage input terminal for supplying a DC voltage for the electronic device, wherein the voltage input terminal is electrically connected to a voltage output terminal of a power supply unit; and
A plurality of module connectors arranged on one surface of the carrier substrate;
A plurality of processor modules, each of the plurality of processor modules having a module connector plug electrically coupled to each of the plurality of module connectors;
When the electronic device is electrically connected to the power supply unit, a support member that supports the carrier substrate so as to be positioned above the power supply unit installed at the bottom of a cooling tank included in the cooling device;
Including
The processor module is
A first circuit board and a second circuit board, each of which has a processor mounting area and a memory mounting area on one surface thereof, wherein at least one processor is mounted on the processor mounting area; A plurality of memory modules arranged in a comb shape are mounted in a mounting area, and the one surface of the first circuit board and the one surface of the second circuit board are combined face to face. A first circuit board and a second circuit board;
A first connector for electrically connecting the first circuit board and the second circuit board;
A third circuit board disposed between the first circuit board and the second circuit board;
A second connector for electrically connecting the third circuit board and the first circuit board or the second circuit board;
Including
The first circuit board and the second circuit board are the processor mounting area and the memory mounting area of the first circuit board, and the processor mounting area and the memory mounting area of the second circuit board, respectively. The front end portions of the plurality of memory modules arranged in a comb shape on the first circuit board and the front end portions of the plurality of memory modules arranged in a comb shape on the second circuit board, respectively, facing each other, It is aligned so that there is a gap between adjacent memory modules,
The third circuit board is disposed in a space formed by the upper surfaces of the processors of the first circuit board and the second circuit board facing each other.

To prevent the memory modules from interfering with each other when one surface of the first circuit board and one surface of the second circuit board are combined face to face, the distance between the substrates is: It had to be longer than twice the height h ₁ of the memory module from the surface of the one. According to the present invention, the first circuit board and the second circuit board are such that the processor mounting area and the memory mounting area of the first circuit board face the processor mounting area and the memory mounting area of the second circuit board, respectively. In addition, the tip portions of the plurality of memory modules arranged in a comb shape on the first circuit board and the tip portions of the plurality of memory modules arranged in a comb shape on the second circuit board are arranged between adjacent memory modules. Since they are aligned so as to be arranged in a staggered manner by creating a gap, the problem that the memory modules interfere with each other can be solved, and the processor and the memory module can be mounted in a smaller volume in an extremely high density. For example, when the memory mounting area includes a plurality of memory sockets into which individual memory modules are inserted, the distance H between one surface of the first circuit board and one surface of the second circuit board is determined from one surface. memory height _{h 1} of the modules, of the memory socket with respect to the height _{h _2,} so as to satisfy the _{(h 1 + h 2) <} H <2h 1, can be shortened. Note that the tip portions of the plurality of memory modules are arranged in a staggered manner with gaps between adjacent memory modules, so that the cooling liquid removes heat from the surface of the memory modules through the gaps. Since the heat generation amount of the memory module is much smaller than the heat generation amount of the processor, even if the gap between adjacent memory modules is relatively narrow, the heat removal action by the cooling liquid is not impaired.

In a processor mounted on a module for immersion cooling, generally, a heat spreader (a processor in which a heat spreader is integrated is also commercially available) and a heat sink are thermally connected to the upper surface of the processor, and the processor is mounted on the heat spreader. The circulating coolant is configured to efficiently take away heat generated from the processor. In the case of air cooling, the heat sink fins need to be raised to obtain a relatively large heat radiation area. May be low. Accordingly, the height from one surface of the first or second circuit board to the upper surface of the processor including the heat spreader and the heat sink can be lower than the height h ₁ of the memory module from the one surface. According to the present invention, by utilizing this height difference, when the one surface of the first circuit board and the one surface of the second circuit board are combined to face each other, the upper surfaces of the processors face each other. The third circuit board is arranged in a relatively large space. This makes it possible to expand or enhance the function of the processor module by utilizing the space that can be created inside the processor module. In addition, since another circuit board is arranged in the surplus space in the processor module, it is possible to realize much higher density mounting.

In addition, the cooling tank having an “open space” in this specification includes a cooling tank having a simple sealed structure that does not impair maintainability of the electronic device. For example, a structure in which a top plate for closing the open space of the cooling tank can be placed in the opening of the cooling tank, or a structure in which the top plate can be detachably attached via packing etc. is a simple sealed structure It can be said.

The above objects and advantages of the present invention, as well as other objects and advantages, will be understood more clearly through the following description of embodiments. However, the embodiment described below is an exemplification, and the present invention is not limited to this.

It is a perspective view of the electronic device which concerns on one Embodiment of this invention. It is a partial assembly drawing of the electronic device which concerns on one Embodiment of this invention, and is an assembly drawing of a backboard or frame structure, a carrier board, and a processor module. It is a top view which shows an example of the 1st circuit board among the processor modules contained in the electronic device which concerns on one Embodiment of this invention. It is a top view which shows an example of the 2nd circuit board among the processor modules contained in the electronic device which concerns on one Embodiment of this invention. An example of a 3rd circuit board is shown among processor modules contained in electronic equipment concerning one embodiment of the present invention, (A) is a top view and (B) is a side view. It is an assembly drawing which shows an example of a processor module, and is an assembly drawing of a 1st circuit board, a 2nd circuit board, and a 3rd circuit board. It is a perspective view which shows an example of a processor module. FIG. 5B is a cross-sectional view taken along the line AA of the processor module shown in FIG. 5A. It is a perspective view which shows an example of the stage in a power supply unit. It is a perspective view which shows the state which attached the unit board | substrate of the power supply unit on the stage. It is a perspective view of a cooling tub with which a cooling device is provided in a cooling system concerning one embodiment of the present invention. It is a top view of the cooling tank with which a cooling device is provided in the cooling system concerning one embodiment of the present invention. It is a partial sectional view of a cooling device in a cooling system concerning one embodiment of the present invention. It is a top view which shows the example which installs and arranges four stages with a unit board | substrate attached to the bottom part of a cooling tank.

Hereinafter, preferred embodiments of an electronic device according to the present invention and an immersion cooling processor module suitably used for the electronic device will be described in detail with reference to the drawings.
First, with reference to FIG.1 and FIG.2, the electronic device 100 which concerns on one Embodiment of this invention is demonstrated. FIG. 1 is a perspective view of an electronic device 100 according to an embodiment of the present invention, and FIG. 2 is a partial assembly view. The electronic device 100 is an electronic device that is immersed and cooled directly in a cooling liquid in a cooling device described later. The electronic device 100 includes a backboard or frame structure 110 (hereinafter simply referred to as “backboard 110”), a plurality of processor modules 120, a carrier board 121, and a network card 122. As illustrated, the power supply unit is abolished from the electronic device 100. As will be described later, the power supply unit is installed at the bottom of the cooling tank provided in the cooling device. In FIG. 2, only one processor module 120 is depicted, and no network card is depicted.

The backboard 110 constitutes a support member that supports the carrier substrate 121. As shown in FIG. 2, the backboard 110 includes an outer frame portion 110b and a beam portion 110c that traverses the inside of the outer frame portion 110b in the width direction. A suspension fitting hole (not shown) through which a suspension tool is passed when the electronic device 100 is put into or taken out of the cooling bath may be formed in the upper portion of the outer frame 110b. The backboard 110 may include a pair of support pins or guide pins (hereinafter simply referred to as “support pins”) extending downward from the lower portion of the outer frame portion 110b. A plurality of support plates (not shown) may be attached to the outer frame portion 110b and the beam portion 110c via fasteners such as screws that penetrate the carrier substrate 121. As a result, the carrier substrate 121 is fixed to one surface of the backboard 110, and a plurality of support plates (not shown) are attached at predetermined intervals in the longitudinal direction of the carrier substrate 121. Each of the support plates (not shown) is formed with a plurality of grooves, and each of the end portions of the plurality of processor modules 120 may be inserted into each of the plurality of grooves. In this way, adjacent support plates (not shown) may hold the upper and lower ends of the plurality of processor modules 120.

A DC voltage input connector 131 for supplying a DC voltage for electronic equipment is provided at the bottom of the carrier substrate 121. The DC voltage input connector 131 corresponds to a voltage input end provided on the carrier substrate 121. A plurality of module connectors 128 are arranged on one surface of the carrier substrate 121. As described below with reference to FIG. 3A and below, each of the plurality of processor modules 120 has a pair of module connector plugs 129 that are electrically coupled to each pair of module connectors 128. Therefore, each processor module 120 can be inserted into the carrier substrate 121 and pulled out from the carrier substrate 121. In the illustrated example, four network cards 122 are attached to the carrier substrate 121. In addition, although 16 processor modules 120 are attached to the carrier substrate 121, the number of the processor modules 120 is arbitrary and is not particularly limited.

Next, the processor module 120 will be described in detail with reference to FIGS. 3A-5B.
The processor module 120 includes a first circuit board 123A, a second circuit board 123B, and

connectors

125A and 125B that electrically connect the first circuit board 123A and the second circuit board 123B. . The processor module 120 includes a third circuit board 123C and

connectors

125C and 125D that electrically connect the third circuit board 123C and the first circuit board 123A. It may be omitted if necessary.

As shown in FIG. 3A, the first circuit board 123A includes a processor mounting area PA located near the center of the board and two memory mounting areas MA located on both sides of the processor mounting area PA. And have. At least one processor 124A is mounted in the processor mounting area PA. The processor 124A is not particularly limited, but may be, for example, an Intel® Corporation Xeon processor (“Xeon” is a trademark of Intel® Corporation). The processor 124A may include a back plate, a socket, a heat spreader (which may be integrated with the processor 124A) thermally connected to the upper surface of the processor 124A, and a heat sink (see FIG. 5B). In FIG. 3A, the fins of the heat sink are not drawn, and the heat spreader under the heat sink is not drawn. Here, it is preferable that the arrangement direction of the fins of the heat sink is the same as the arrangement direction of the memory modules described later so as to coincide with the flow direction of the coolant.

In each of the two memory mounting areas MA, two memory modules 127A arranged in a comb shape are mounted. The number of memory modules 127A mounted in the memory mounting area MA may be two or more. The memory module 127A may be a general-purpose Standard Height 64GB DIMM instead of a VLP module in accordance with the object of the present invention. As shown in the drawing, a necessary number of general-purpose memory sockets 126A are mounted on the substrate at a predetermined interval, and the memory module 127A is inserted into each of the memory sockets 126A. Further, on one surface of the first circuit board 123A, a connector 125A for electrical connection with the second circuit board 123B, a connector 125D for electrical connection with the third circuit board 123B, and And one of the pair of module connector plugs 129 coupled to one of the pair of module connectors 128 of the carrier substrate 121. In addition to the above-described components, various electronic components are mounted on one surface of the first circuit board 123A, but the electronic components are not drawn. The first circuit board 123 configured as described above and the entire components mounted thereon may be hereinafter referred to as “A board 120A”.

As shown in FIG. 3B, the second circuit board 123B has a configuration similar to that of the first circuit board 123A. Specifically, the second circuit board 123B has a processor mounting area PB located near the center of the board and two memory mounting areas MB located on both sides of the processor mounting area PB on one surface. Have. At least one processor 124B is mounted in the processor mounting area PB. Since the processor 124B is the same as the processor 124A, description thereof is omitted here.

Two memory modules 127B arranged in a comb shape are mounted in each of the two memory mounting areas MB. The number of memory modules 127B mounted in the memory mounting area MB may be two or more. Note that the memory module 127B and the memory socket 126B are the same as the memory module 127A and the memory socket 126A, and thus description thereof is omitted here. On one surface of the second circuit board 123B, a connector 125B for electrical connection with the first circuit board 123A, and a module connector coupled to the other of the pair of module connectors 128 of the carrier board 121 It has the other of the pair of plugs 129. In addition to the above-described components, various electronic components are mounted on one surface of the second circuit board 123B, but the electronic components are not drawn. The second circuit board 123B configured as described above and the entire components mounted thereon may be hereinafter referred to as “B board 120B”.

Here, as can be seen by comparing FIG. 3A and FIG. 3B, the size of the first circuit board 123A and the size of the second circuit board 123B are the same, and the processor mounting area PA of the first circuit board 123A. The positions of the memory mounting area MA and the module connector plug 129 and the positions of the processor mounting area PB, the memory mounting area MB, and the module connector plug 129 of the second circuit board 123B are substantially vertically symmetrical. However, the entire processor mounting area PB and memory mounting area MB of the second circuit board 123B are slightly shifted toward the center of the board (downward). This is because when one surface of the first circuit board 123A and one surface of the second circuit board 123B are combined face to face, the four corners of the first circuit board 123A and the second circuit board 123B are combined. When the processor area and the memory mounting area are brought close to each other so that the four corners of the first and second memory boards face each other, the front ends of the plurality of memory modules 127A arranged in a comb shape on the first circuit board 123A and the second circuit board 123B This is because the front ends of the plurality of memory modules 127B arranged in a comb shape are arranged in a staggered manner by creating a gap between adjacent memory modules (see FIG. 5B). You can avoid fitting.

Next, as shown in FIG. 3C, the third circuit board 123 </ b> C has four storage devices 130 mounted on one surface of the board, and electrical connection with the first circuit board 123 </ b> A on the other surface of the board. A connector 125D for connection is provided. The third circuit board 123C may alternatively or additionally be mounted with an I / O control chip set. In addition, the third circuit board 123C may alternatively or additionally be mounted with a programmable logic device. The storage device 130 may use flash storage such as M.2 SSD (Solid State Drive) or mSATA (mini Serial ATA) SSD. Further, an FPGA (Field-ProgrammablemGate Array) may be used as the programmable logic device. In addition to the above-described components, various electronic components are mounted on one surface of the third circuit board 123C, but the electronic components are not drawn. The third circuit board 123C configured as described above and the entire components mounted thereon may be hereinafter referred to as “C board 120C”.

4 is an assembly view when the processor board 120 is configured by combining the A board 120A, the B board 120B, and the C board 120C, FIG. 5A is a perspective view of the assembled processor module 120, and FIG. FIG. 5B is a cross-sectional view taken along line AA of the processor module 120 shown in FIG. 5A. In the drawing, the fins of the heat sink that are thermally connected to the upper surfaces of the

processors

124A and 124B are not drawn. Here, it is preferable that the arrangement direction of the fins of the heat sink is the same as the arrangement direction of the memory modules described later so as to coincide with the flow direction of the coolant. As shown in FIG. 4, the processor module 120 is assembled by first coupling the connector 125C of the C board 120C to the connector 125D of the A board 120A, and then making one surface of the B board 120B one of the A boards 120A. The B board 120B is turned over so as to face the surface, and the connector 125B of the B board 120B is coupled to the connector 125A of the A board 120A. When the

connectors

125A and 125B are coupled to each other, the front ends of the plurality of memory modules 127A arranged in a comb shape on the A substrate 120A and the front ends of the plurality of memory modules 127B arranged in a comb shape on the B substrate 120B are In addition to helping to align the A substrate 120A and the B substrate 120B so that gaps are formed between adjacent memory modules, the one surface of the A substrate 120A and one surface of the B substrate 120B are used. It is useful to keep the distance H to a predetermined length.

Referring to FIG. 5B, as described above, the A substrate 120A and the B substrate 120B include the front ends of the plurality of memory modules 127A arranged in a comb shape of the A substrate 120A and the plurality of memories arranged in a comb shape of the B substrate 120C. The front ends of the modules 127B are aligned so as to be alternately arranged with a gap between adjacent memory modules. For example, the memory sockets 126A can be installed at an interval such that the distance between the center lines is about 8 mm. The standard Standard Height 64GB DIMM is approximately 30mm in height and approximately 3mm in thickness, so when the

memory modules

127A and 127B are arranged in a staggered manner, the gap between adjacent memory modules is approximately 1mm ( = (8-3-1.5 × 2) / 2). Since the heat generation amount of the memory module is much smaller than the heat generation amount of the processor, even if the gap between adjacent memory modules is so narrow, the heat removal action by the coolant is not impaired.

As shown in FIG. 5B, the distance H between one surface of the A substrate 120A and one surface of the B substrate 120B is set such that the height h ₁ of the

memory modules

127A and 127B from the one surface and the

memory sockets

126A and 126B. Regarding the height h ₂ , the height h ₂ can be shortened so as to satisfy (h ₁ + h ₂ ) <H <2h ₁ . For example, since the height of the

memory sockets

126A and 126B for general-purpose Standard Height 64GB DIMMs is about 5 mm, the distance H can be shortened to 36 mm to 40 mm, and higher density mounting can be realized. Can do. For example, when the distance H is designed to be about 40 mm, and the height of the processor (including the heat sink fins) from one side of the substrate is about 15 mm, the space formed by the top surfaces of the

processors

124A and 124B facing each other is about It has a height of 10 mm (= 40-15 × 2). This approximately 10 mm high space created in the processor module 120 is large enough for additional placement of various components to expand or enhance the functionality of the processor module (if any). If the space is about 5 mm in height, the height is too low and it is very difficult to place the components there). The C substrate 120 </ b> C is another circuit substrate for function expansion or expansion arranged in this space. Of course, with such a height, the gap between the upper surface of the processor 124A of the A substrate 120A and the back surface of the C substrate 120C, and the gap between the upper surface of the processor 124B of the B substrate 120B and the surface of the C substrate 120C, Since it can be ensured without any problem, there is no problem in achieving the desired heat removal performance by the coolant passing therethrough.

Next, returning to FIG. 2, the configuration of the electronic device 100 will be described in more detail. On the surface opposite to one surface of the backboard 110, a plurality of the electronic devices 100 are arranged along the longitudinal direction of the outer frame portion 110 b of the backboard 110. A slider (not shown) may be attached. A pair of sliders attached to both the left and right sides of the outer frame portion 110b are engaged with rail grooves provided in adjacent support pillars that are vertically raised and fixed in a cooling tank, which will be described later. The board 110 may be supported so as to be slidable (up and down in the vertical direction).

When the electronic device 100 configured as described above slides the backboard 110 with respect to the plurality of support pillars and is immersed in the cooling liquid in the cooling device and directly cooled, the electronic device 100 circulates in the electronic device. The cooling liquid that passes through and out of the processor module 120 quickly and efficiently removes heat from the processor module 120 and the carrier substrate 121, so that the electronic device 100 is stable even when mounted at a high density. Operation can be secured. At this time, in the processor module 120, a gap between the A board 120A and the C board 120C, a gap between the B board 120B and the C board 120C, and a gap between adjacent memory modules are respectively secured. However, as described above, the flow channel through which the coolant flows is formed. In addition, each processor module 120 can be attached to and removed from the carrier substrate 121. As a result, adjustment, inspection, repair, replacement, expansion, etc. can be performed for each processor module 120, so that maintainability is significantly improved.

Next, the power supply unit 20 installed in the lower part of the cooling device will be described with reference to FIGS. FIG. 6 is a perspective view showing an example of the stage 22 in the power supply unit 20, and FIG. 7 is a perspective view showing a state in which the unit substrate 21 in the power supply unit 20 is mounted on the stage 22.

As described above, the power supply unit 20 is a component not included in the electronic device 100, and is installed at the bottom of the cooling tank provided in the cooling device. The power supply unit 20 includes a unit substrate 21 and a step-down device 215 mounted on the unit substrate 21. The unit board 21 includes a power supply voltage input connector 212 that supplies an external power supply voltage from an external power supply (not shown) via a power supply cable 211, a DC voltage output connector 213 that outputs a DC voltage stepped down by the step-down device 215, and Is provided. The step-down device 215 is, for example, a converter module that steps down an external high-voltage DC voltage of 200V-420V to a DC voltage of 24V-52V, or a single-phase or three-phase external high-voltage AC voltage of 100V-250V, It is preferable to include a converter module that performs AC / DC conversion and step-down to a DC voltage. More specifically, the former converter module may be capable of stepping down DC380V to DC48V, and the latter converter module may be capable of AC / DC conversion and stepping down to DC48V from AC200V. The step-down device 215 may include any one or more peripheral circuits of a power factor correction circuit, a noise filter, an additional rectifier, and a surge circuit, as needed. A heat sink 216 for heat dissipation is preferably thermally connected to the surface of the step-down device 215. Further, the unit substrate 21 may include a plurality of input fuses 217 that protect against failure. As shown in FIG. 7, the unit substrate 21 is fixed to the stage 22 via a plurality of spacers 218. Thus, the unit substrate 21 is arranged away from the bottom so as to form a flow channel 219 through which the coolant flows between one surface of the unit substrate 21 and the bottom of the cooling tank described later. The unit substrate 21 may be configured to have a flow channel through which the coolant passes. For example, the unit substrate 21 may be formed in a hierarchical structure or a hollow structure having an intermediate space, and a cooling liquid may be passed through the structure.

As shown in FIG. 6, the stage 22 includes a flat plate placed at the bottom of a cooling tank described later. Near the center in the width direction of the flat plate, a plurality of holes 23 through which the coolant flowing in from the bottom is passed are formed at intervals in the longitudinal direction. Further, a plurality of notches 24 are formed at intervals in the longitudinal direction at the end in the width direction of the flat plate. The adjacent cutouts 24 have a length and a width necessary for the adjacent cutouts 24 to form the same hole as the hole 23 when the plurality of stages 22 are arranged side by side. A plurality of support pillars 25 are vertically mounted on the stage 22 using an L-shaped bracket 26. Therefore, when the stage 22 is installed in the bottom part of a cooling tank, the some support pillar 25 will stand upright in a cooling tank. A plurality of brackets 27 are fixed on the stage 22. A support pin insertion hole 28 is formed in the bracket 27.

A rail groove 251 is formed in each of the plurality of support pillars 25. A pair of sliders (not shown) included in the backboard 110 of the electronic device 100 engage with rail grooves 251 provided in adjacent support pillars, so that the backboard 110 can be slid (raised and lowered in the vertical direction). Supported by

With respect to the power supply unit 20 configured as described above, the electronic device 100 can raise or lower the backboard 110 by sliding the backboard 110 with respect to the plurality of support pillars 25. When the electronic device 100 is lowered, a pair of support pins (not shown) extending downward from the lower portion of the outer frame portion 110b of the backboard 110 of the electronic device 100 are fixed to the power supply unit 20. 27 is inserted into the support pin insertion hole 27 and the DC voltage output connector 213 of the power supply unit 20 and the DC voltage input connector 131 of the electronic device 100 may be accurately aligned. When the electronic device 100 is further lowered, the DC voltage output connector 213 and the DC voltage input connector 131 are electrically connected. At this time, the pair of support columns 25 and the pair of brackets 27 support the weight of the electronic device 100 of one unit.

The power supply unit 20 may further include a first controller that starts supplying a DC voltage to the electronic device 100 when it detects that the DC voltage output connector 213 is coupled to the DC voltage input connector 131 of the electronic device 100. . The first controller may be mounted on the unit substrate 21 as an additional circuit or an electronic mechanism. As a result, the electronic device 100 can be plugged in immediately by energizing the electronic device 100 by simply lowering the electronic device 100 into the cooling tank and coupling the electronic device 100 to the power supply unit 20.

Further, the power supply unit 20 detects ON / OFF of a switch that can be operated from the upper part of the liquid level of the cooling liquid in the cooling tank, the wall surface structure part of the cooling tank, or a control panel installed in the vicinity of the cooling tank, A second controller that switches start / disconnection of voltage supply to the electronic device 100 may be further included. Thereby, since an operator can switch ON / OFF for every electronic device 100 manually, a maintainability can be improved. The second controller may also be mounted on the unit board 21 as an additional circuit or an electronic mechanism.

A switch for sending a signal for switching start / cut of voltage supply to the electronic device 100 to the second controller may be provided at the upper end or the side surface of each of the plurality of support pillars 25.

Next, based on the drawings, an example of an immersion cooling device for directly cooling the electronic device 100 according to an embodiment of the present invention described above and the power supply unit 20 described above by immersing them in the cooling liquid. explain. In the following description, a configuration of a high-density immersion cooling apparatus that stores and cools a total of 24 electronic devices 100 and power supply units 20 in 6 × 4 sections of a cooling tank will be described. This is merely an example, and the number of units of the electronic device in the high-density immersion cooling apparatus is arbitrary, and does not limit the configuration of the electronic device that can be used in the present invention.

8 to 11, the immersion cooling apparatus 1 according to an embodiment includes a cooling tank 10, and an open space 10 a is formed by the bottom wall 11 and the side wall 12 of the cooling tank 10. In the bottom wall 11, a plurality of inflow openings 150 into which the cooling liquid flows are formed in a 9 × 3 pattern. Further, the side wall 12 is formed with a power cable introduction port 12a, a network cable introduction port 12b, and an outflow opening 170 formed in the vicinity of the liquid surface of the coolant.

The immersion cooling device 1 has a top plate 10b for closing the open space 10a of the cooling bath 10. During maintenance work of the immersion cooling device 1, the top plate 10 b is removed from the opening to open the open space 10 a, and during operation of the immersion cooling device 1, the top plate 10 b is placed in the opening of the cooling tank 10 to open it. The space 10a can be closed.

The cooling tank 10 is filled with a sufficient amount of coolant to immerse the entire electronic device 100 up to the liquid level shown in FIG. 10 (see FIG. 10). As the coolant, trade names of 3M Company “Fluorinert (trademark of 3M Company, hereinafter the same) FC-72” (boiling point 56 ° C.), “Fluorinert FC-770” (boiling point 95 ° C.), “Fluorinert FC-3283” ( Fluorine inert liquid composed of perfluorinated compounds (perfluorocarbon compounds) known as “Fluorinert FC-40” (boiling point 155 ° C.), “Fluorinert FC-43” (boiling point 174 ° C.) Although it can be used, it is not limited to these. Since Fluorinert FC-40 and FC-43 have a boiling point higher than 150 ° C. and are extremely difficult to evaporate, when one of them is used as a cooling liquid, the liquid level in the cooling tank 10 is long. This is advantageous over time.

A plurality of inflow headers 16 having a coolant inlet 15 at one end are provided below the bottom wall 11 of the cooling tank 10. A receiving portion 17 having a coolant outlet 18 is provided outside the side wall 12 of the cooling tank 10. The receiving part 17 covers the outflow opening 170 and receives the coolant flowing out from the outflow opening 170 without leaking.

Referring to FIG. 11, four flat stages 22 are arranged side by side on the bottom wall 11 of the cooling bath 10. Each of the plurality of holes 23 formed in the stage 22 and the notches 24 adjacent to each other, which are substantially the same as the holes 23, are respectively formed in the plurality of inflow openings 150 formed in the bottom wall 11. Match. Therefore, the cooling liquid flowing in from the inflow opening 150 is not blocked by the power supply unit 20. In addition, since a flow channel through which the coolant flows is secured between the unit substrate 21 of the power supply unit 20 and the stage 22 (bottom wall 11), the coolant quickly heats up from both sides of the unit substrate 21, and Take away efficiently. Accordingly, the cooling efficiency of the power supply unit 20 is excellent. Furthermore, since the unit substrate 21 of the power supply unit 20 can be placed parallel to the bottom wall 11 of the cooling tank 10, the height of the power supply unit 20 in the height (depth) direction of the cooling tank 10 is conventionally set. It can be kept low compared to this. Therefore, since the combined length of the electronic device 100 and the power supply unit 20 can be shortened, the cooling tank 10 can be designed to have a low height (a shallow depth).

In addition, the cooling liquid flowing in from the inflow opening 150 passes through the inside and outside of the processor module from the lower side to the upper side of the electronic device 100, and quickly and efficiently heats the processor module 120 and the carrier substrate 121. Steal well. The coolant thus warmed reaches the outlet 18 through the outlet opening 170 and the receiving portion 17. A pipe (not shown) that leads to the inlet 15 through a heat exchanger (not shown) is connected to the outlet 18, and the cooling liquid is cooled in the heat exchanger, and the cooled cooling liquid flows into the inlet 15. To be supplied.

The present invention can be widely applied to an immersion cooling processor module and electronic equipment that are mounted at an ultra-high density.

DESCRIPTION OF SYMBOLS 1 Immersion cooling device 10 Cooling tank 10a Open space 10b Top plate 11 Bottom wall 12 Side wall 12a Power cable inlet 12b Network cable inlet 100 Electronic device 110 Backboard or frame structure 110a Hole 110b Outer frame part 110c Beam part 120 Processor module 121 Carrier board 122

Network card

123A, 123B,

123C Circuit board

124A,

124B Processor

125A, 125B, 125C,

125D Connector

126A,

126B Memory socket

127A, 127B Memory module 128 Module connector 129 Module connector plug 130 Storage device PA, PB Processor mounting Area MA, MB Memory mounting area

Claims

A processor module for immersion cooling applied to an electronic device that is immersed in a cooling liquid in a cooling device and directly cooled,
A first circuit board and a second circuit board, each of which has a processor mounting area and a memory mounting area on one surface thereof, wherein at least one processor is mounted on the processor mounting area; A plurality of memory modules arranged in a comb shape are mounted in a mounting area, and the one surface of the first circuit board and the one surface of the second circuit board are combined face to face. A first circuit board and a second circuit board;
A first connector for electrically connecting the first circuit board and the second circuit board;
A third circuit board disposed between the first circuit board and the second circuit board;
A second connector for electrically connecting the third circuit board and the first circuit board or the second circuit board;
Including
The first circuit board and the second circuit board are the processor mounting area and the memory mounting area of the first circuit board, and the processor mounting area and the memory mounting area of the second circuit board, respectively. The front end portions of the plurality of memory modules arranged in a comb shape on the first circuit board and the front end portions of the plurality of memory modules arranged in a comb shape on the second circuit board, respectively, facing each other, It is aligned so that there is a gap between adjacent memory modules,
The third circuit board is disposed in a space formed by the upper surfaces of the processors of the first circuit board and the second circuit board facing each other.
Processor module for immersion cooling.
The memory mounting area includes a plurality of memory sockets for fixing individual memory modules,
The distance H between the one surface of the first circuit board and the one surface of the second circuit board is the height h 1 of the memory module from the one surface, and the height of the memory socket. 2. The immersion cooling processor module according to claim 1 , wherein (h 1 + h 2 ) <H <2h 1 is satisfied with respect to h 2 .
The liquid immersion according to claim 2, wherein the memory module is a Standard メモリ Height memory module, and the module is inserted into the memory socket such that a substrate surface of the memory module is perpendicular or inclined with respect to the one surface. Cooling processor module.
The processor module for immersion cooling according to claim 1, wherein the processor of the first circuit board and the processor of the second circuit board are connected via an inter-processor interconnection interface.
The immersion cooling processor module according to claim 1, wherein the third circuit board is a circuit board on which a storage device and / or an I / O control chipset is mounted.
An electronic device that is directly cooled by being immersed in a cooling liquid in a cooling device,
A carrier substrate having a voltage input terminal for supplying a DC voltage for the electronic device, wherein the voltage input terminal is electrically connected to a voltage output terminal of a power supply unit; and
A plurality of module connectors arranged on one surface of the carrier substrate;
A plurality of processor modules, each of the plurality of processor modules having a module connector plug electrically coupled to each of the plurality of module connectors;
When the electronic device is electrically connected to the power supply unit, a support member that supports the carrier substrate so as to be positioned above the power supply unit installed at the bottom of a cooling tank included in the cooling device;
Including
The processor module is
A first circuit board and a second circuit board, each of which has a processor mounting area and a memory mounting area on one surface thereof, wherein at least one processor is mounted on the processor mounting area; A plurality of memory modules arranged in a comb shape are mounted in a mounting area, and the one surface of the first circuit board and the one surface of the second circuit board are combined face to face. A first circuit board and a second circuit board;
A first connector for electrically connecting the first circuit board and the second circuit board;
A third circuit board disposed between the first circuit board and the second circuit board;
A second connector for electrically connecting the third circuit board and the first circuit board or the second circuit board;
Including
The first circuit board and the second circuit board are the processor mounting area and the memory mounting area of the first circuit board, and the processor mounting area and the memory mounting area of the second circuit board, respectively. The front end portions of the plurality of memory modules arranged in a comb shape on the first circuit board and the front end portions of the plurality of memory modules arranged in a comb shape on the second circuit board, respectively, facing each other, It is aligned so that there is a gap between adjacent memory modules,
The third circuit board is disposed in a space formed by the upper surfaces of the processors of the first circuit board and the second circuit board facing each other.
Electronics.
The electronic device according to claim 6, wherein the support member includes a backboard or a frame structure in which the carrier substrate is fixed to one surface.
The electronic device according to claim 7, wherein the backboard or the frame structure is slidably supported by a plurality of support pillars which are vertically fixed in the cooling tank.
The memory mounting area includes a plurality of memory sockets for fixing individual memory modules,
The distance H between the one surface of the first circuit board and the one surface of the second circuit board is the height h 1 of the memory module from the one surface, and the height of the memory socket. respect h 2, satisfy (h 1 + h 2) < H <2h 1, the electronic device according to claim 6.
The electronic device according to claim 6, wherein the memory module is a Standard Height memory module, and the module module is inserted into the memory socket with a substrate surface of the memory module being perpendicular or inclined with respect to the one surface. .
The electronic device according to claim 6, wherein the processor of the first circuit board and the processor of the second circuit board are connected via an inter-processor interconnection interface.