JP4985750B2 - Data storage system - Google Patents

Description

  The present invention relates to a configuration of a data storage system used as an external storage device of a computer, and in particular, a data storage system having a combination of units and connections that can flexibly configure a data storage system to which a large number of disk devices are connected. About.

  In recent years, as various data has been digitized and handled on computers, data storage that can store large amounts of data efficiently and with high reliability, independent of the host computer that executes data processing The importance of devices (external storage devices) is increasing.

  As this data storage device, a disk array device is used which is composed of a large number of disk devices (for example, magnetic disks and optical disks) and a disk controller which controls the large number of disk devices. This disk array apparatus can simultaneously receive disk access requests from a plurality of host computers and control a large number of disks.

  In recent years, there has also been provided a disk array apparatus that can control a disk device group of several thousand or more disk devices and a storage capacity of several hundred terabytes or more.

  Such a disk array device incorporates a memory that serves as a disk cache. This shortens the data access time when receiving a read request and a write request from the host computer, thereby realizing high performance.

  In general, a disk array device includes a plurality of main units, that is, a channel adapter that is a connection part with a host computer, a disk adapter that is a connection part with a disk drive, a cache memory, a cache control unit in charge of controlling the cache memory, And a large number of disk drives.

  FIG. 11 is an explanatory diagram of the first prior art. The disk array device 102 shown in FIG. 11 includes two cache managers (cache memory and cache control unit) 10, and a channel adapter 11 and a disk adapter 13 are connected to each cache manager 10.

  The two cache managers 10 and 10 are directly connected to each other via a bus 10c so as to communicate with each other. The two cache managers 10 and 10, the cache manager 10 and the channel adapter 11, and the cache manager 10 and the disk adapter 13 are connected by a PCI bus because low latency is required. Yes.

  Further, the channel adapter 11 is connected to a host computer (not shown) by, for example, fiber channel or Ethernet (registered trademark), and the disk adapter 13 is connected to each disk drive of the disk enclosure 12 by, for example, a fiber channel cable. Has been.

  The disk enclosure 12 has two ports (for example, fiber channel ports), and these two ports are connected to different disk adapters 13. This provides redundancy and increases fault tolerance.

  FIG. 12 is a configuration diagram of the disk array device 100 of the second conventional example. As shown in FIG. 12, the conventional disk array device 100 is an interface with a cache manager (indicated as CM in the figure) 10 having a cache memory and a cache control unit as main units, and a host computer (not shown). A channel adapter (denoted as CA in the figure) 11, a disk enclosure 12 having a plurality of disk drives, and a disk adapter (denoted as DA in the figure) 13 as an interface with the disk device 12 are provided.

  Further, a router (Router; denoted as RT in the figure) 14 for connecting the cache manager 10, the channel adapter 11, and the disk adapter 13 to each other and performing data transfer and communication between these main units is provided.

  In this disk array device 100, four cache managers 10 are provided, and four routers 14 are provided corresponding to these cache managers 10. The cache manager 10 and the router 14 are connected to each other in a one-to-one relationship, thereby making the connection between the plurality of cache managers 10 redundant and increasing the availability (for example, Patent Document 1). reference).

  That is, even when one router 14 fails, the connection between the plurality of cache managers 10 is ensured through another router 14. In such a case, the disk array device 100 operates normally. Can continue.

  In this disk array device 100, two channel adapters 11 and two disk adapters 13 are connected to each router 14, and the disk array device 100 includes a total of eight channel adapters 11 and a total of eight disk adapters 13. I have it.

  These channel adapters 11 and disk adapters 13 can communicate with all the cache managers 10 by the interconnection of the cache manager 10 and the router 14.

  The channel adapter 11 is connected to a host computer (not shown) for processing data by, for example, fiber channel or Ethernet (registered trademark), and the disk adapter 13 is, for example, by a fiber channel cable. It is connected to a disk enclosure 12 (specifically, a disk drive).

  Between the channel adapter 11 and the cache manager 10 and between the disk adapter 13 and the cache manager 10, not only user data from the host computer but also the internal operation of the disk array device 100 is maintained. Exchange of various information (for example, data mirroring processing between a plurality of cache memories) is performed.

  The interface between the cache manager 10, the channel adapter 11 and the disk adapter 13, and the router 14 can realize a lower latency (fast response speed) than between the disk array device 100 and the host computer or between the disk drives. Connected through. For example, a cache manager 10, a channel adapter 11, a disk adapter 13, and a router are connected by a bus designed to connect between LSI (Large Scale Integration) and a printed circuit board, such as a PCI (Peripheral Component Interconnect) bus. 14 is connected.

  Further, the disk enclosure 12 that accommodates the disk drive has two Fiber Channel ports, and a disk adapter 13 belonging to a different router 14 is connected to each port. Thereby, it is possible to prevent the connection from the cache manager 10 from being disconnected even when the disk adapter 13 or the router 14 fails.

JP 2001-256003 A (FIG. 1)

  With the recent advance of computerization, there is a demand for a data storage system with a larger capacity and a higher speed. In the first conventional disk array device described above, when the cache manager 10, the channel adapter 11, and the disk adapter 13 are further increased in order to increase the capacity and speed, the number of ports of the disk enclosure 12 is increased. In addition, it is necessary to increase the number of connection cables between the disk adapter 13 and the disk enclosure 12.

  Increasing the number of ports of the disk enclosure 12 increases the number of cables corresponding to the number of disk adapters connected to one disk enclosure, and increases the mounting space. That is, the size of the apparatus is increased. In addition, if there are two paths in a single disk enclosure, a sufficient redundant configuration can be obtained, so it is not a good idea to increase the number of ports. In addition, the number of connected disk adapters is not constant and changes according to the user's request. Therefore, if a large number of ports are added, a small number of disk adapters will be wasted, and if a small number of ports are added, a large number Cannot support other disk adapters. That is, the versatility is lost.

  On the other hand, in the second conventional disk array device, it is possible to add the cache manager 10, the channel adapter 11, and the disk adapter 13, but all communicate via the router 14, and therefore communicate with the router 14. Since data concentrates, it becomes a bottleneck for throughput and high throughput cannot be expected.

  In addition, in the disk array device 100, when a large-scale disk array device having a large number of main units is configured, the number of connection lines between the cache manager 10 and the router 14 increases rapidly, so that the connection relationship is complicated. It becomes difficult to implement physically.

  For example, in the configuration of FIG. 12, as shown in FIG. 13, four (four) cache managers 10 and four (four) routers 14 are connected via the back panel 15. Take the structure. In this case, as described above, the number of signals is 4 × 4 × (number of signals per path) as shown in FIG. For example, as described above, when one path is connected by 64-bit PCI (parallel bus), the number of signal lines is 100 × 16 = 1600 on the back panel 15 including the control lines. In order to wire this signal line, the printed circuit board of the back panel 15 requires six signal layers.

  Furthermore, in the case of a large-scale configuration, for example, in a configuration in which eight (four) cache managers 10 and eight (four) routers 14 are connected via the back panel 15, As for the number, 100 × 8 × 8 = about 6400 is required. For this purpose, the printed circuit board of the back panel 15 requires four times 24 layers, which is difficult to realize.

  If, instead of the 64-bit PCI bus, the connection is made with 4-lane PCI-Express with few signal lines, the number of signal lines is 16 × 8 × 8 = 1024. However, while the PCI bus is 66 MHz, the PCI-Express is a 2.5 Gbps high-speed bus, and it is necessary to use an expensive substrate material in order to maintain the signal quality of the high-speed bus.

  Furthermore, in the case of a low-speed bus, the wiring layer can be replaced by using a via (Via). However, in the high-speed bus, Via causes a decrease in signal quality and must be avoided. For this reason, in a high-speed bus, it is necessary to arrange so that all signal lines do not cross, and about twice as many signal layers as the same number of low-speed buses are required. For example, the substrate requires 12 signal layers and needs to be made of an expensive material, which is also difficult to realize.

  Moreover, in the second conventional disk array device 100, when one of the routers 14 fails, the channel adapter 11 and the disk adapter 13 connected under the router 14 cannot be used simultaneously with the failure of the router 14. End up.

  Accordingly, an object of the present invention is to provide a data storage system for performing data transfer between units with high throughput and easily realizing a configuration from a small scale to a large scale without any problems in mounting. It is in.

  Another object of the present invention is the flexibility to easily realize a configuration from a small scale to a large scale by combining the same units while maintaining the redundancy that can operate even when a unit fails. Is to provide a data storage system.

  Furthermore, still another object of the present invention is to provide a data storage system for easily realizing a configuration from a small scale to a large scale without a problem in mounting while ensuring high throughput and redundancy. It is in.

In order to achieve this object, the data storage system of the present invention includes a plurality of disk enclosures configured by connecting a plurality of disk storage devices for storing data in parallel to a pair of signal lines , and an access instruction from a host. And a plurality of control modules for controlling access to the disk storage device, each of the control modules including a first interface unit for interface control with the host and an interface with the disk storage device A second interface unit that performs control; and a control unit that controls access to the disk storage device from the second interface unit in response to the access instruction, and the second interface of each of the control modules. Connected to the front of each control module A plurality of first substrates on which a plurality of first switch units for selectively switching between a second interface unit and the plurality of disk enclosures are mounted; at least the control unit; and the second interface unit. a plurality of second substrates, wherein a plurality of first substrate, said plurality of second substrate is mounted, individually each said second interface unit to each of the plurality of first switch units and a back panel for connecting a serial bus is provided for each of said plurality of disk enclosure is constructed by connecting the unit disk enclosures connected in parallel a plurality of disk storage devices to a pair of signal lines in a plurality in series, Each of the pair of signal lines of the head unit disk enclosure of each of the plurality of disk enclosures , It is connected to different first switch units.

  In the present invention, it is preferable that the control module has a communication unit for communicating with another control module, and further, a second switch unit for selectively connecting the communication units of the control modules. Have

  Furthermore, in the present invention, preferably, the communication unit of each control module and the second switch unit are connected by the back panel.

  Furthermore, in the present invention, preferably, the first switch unit and the plurality of storage devices are connected by a cable.

  In the present invention, it is preferable that the storage device has a plurality of access ports, and the plurality of different first switch units are connected to the plurality of access ports.

  In the present invention, it is preferable that the control module connects the cache control unit and the second interface unit with a high-speed serial bus having a plurality of lanes, and also connects the second interface unit and the plurality of second interfaces. 1 switch unit was connected to the back panel by a serial bus.

  Furthermore, in the present invention, preferably, the high-speed serial bus is a PCI-Express bus.

  Furthermore, in the present invention, preferably, the serial bus is a fiber channel.

  In the present invention, it is preferable that the control module connect the cache control unit and the first interface unit with a low-latency high-speed serial bus.

In the present invention, to connect each second interface of the plurality of control modules to the plurality of first switch units, each of the pair of signal lines of the head unit disk enclosures of each of the plurality of disk enclosures, different first because connected to the switch unit, all the control modules can maintain redundancy to access all the storage devices, and the number of the number and the disk enclosure of the control module in such a connection type is increased, the unit Since the disk enclosures are connected in series and the control module and the first switch unit can be connected by a back panel and a serial bus with a small number of signals constituting the interface, mounting on a printed circuit board becomes possible.

It is a block diagram of the data storage system of one embodiment of this invention. It is a block diagram of the control module of FIG. FIG. 3 is a configuration diagram of the back-end router and the disk enclosure of FIGS. 1 and 2. FIG. 4 is a configuration diagram of the disk enclosure of FIGS. 1 and 3. FIG. 3 is an explanatory diagram of a read process having the configuration of FIGS. 1 and 2. FIG. 3 is an explanatory diagram of a write process with the configuration of FIGS. 1 and 2. It is a figure which shows the mounting structure of the control module of one embodiment of this invention. It is a figure which shows the example of mounting structure of the data storage system of one embodiment of this invention. 1 is a block diagram of a large-scale storage system according to an embodiment of the present invention. It is a block diagram of the medium-scale storage system of other embodiment of this invention. 1 is a configuration diagram of a first conventional storage system. FIG. It is a block diagram of the 2nd conventional storage system. It is a figure which shows the mounting structure of the 2nd conventional storage system of FIG.

  Hereinafter, embodiments of the present invention will be described in the order of a data storage system, read / write processing, mounting structure, and other embodiments.

[Data storage system]
1 is a configuration diagram of a data storage system according to an embodiment of the present invention, FIG. 2 is a configuration diagram of a control module in FIG. 1, FIG. 3 is a configuration diagram of a back-end router and a disk enclosure in FIG. 4 is a configuration diagram of the disk enclosure of FIGS. 1 and 3. FIG.

  FIG. 1 shows an example of a large-scale storage system having eight control modules. As shown in FIG. 1, the storage system 1 includes a plurality of disk enclosures 2-0 to 2-25 for holding data, a host computer (data processing device) (not shown), and a plurality of disk enclosures 2-0 to 2-. 25, a plurality of (in this case, eight) control modules 4-0 to 4-7, a plurality of control modules 4-0 to 4-7, and a plurality of disk enclosures 2-0 to 2 A plurality of (here, eight) Back-end Routers (first switch unit; referred to as BRT in the figure, hereinafter referred to as BRT) 5-0 to 5-7, Here, there are two front-end routers (second switch units; denoted as FRT in the figure, hereinafter referred to as FRT) 6-0 and 6-1.

  Each of the control modules 4-0 to 4-7 includes a cache manager 40, a channel adapter (first interface unit; denoted as CA in the figure) 41a to 41d, and a disk adapter (second interface part; denoted as DA in the figure). ) 42a and 42b, and a DMA (Direct Memory Access) engine (communication unit; denoted as DMA in the figure) 43.

  In FIG. 1, for simplification of the drawing, these cache manager codes “40”, channel adapter codes “41a”, “41b”, “41c”, “41d”, disk adapter codes “42a”, “42b” and DMA code “43” are given only to the control module 4-0, and the symbols of these components in the other control modules 4-1 to 4-7 are omitted.

  The control modules 4-0 to 4-7 will be described with reference to FIG. The cache manager 40 performs read / write processing based on a processing request (read request or write request) from the host computer, and includes a cache memory 40b and a cache control unit 40a.

  The cache memory 40b serves as a so-called cache for a plurality of disks that holds a part of the data held in the plurality of disks of the disk enclosures 2-0 to 2-25.

  The cache control unit 40a controls the cache memory 40b, the channel adapter 41, the device adapter 42, and the DMA 43. For this reason, one or a plurality (two in the figure) of CPUs 400 and 410 and a memory controller 420 are provided. The memory controller 420 controls reading / writing of each memory and performs path switching.

  The memory controller 420 is connected to the cache memory 40b via the memory bus 434, and is connected to the CPUs 400 and 410 via the CPU buses 430 and 432. Further, the memory controller 420 is a 4-lane high-speed serial bus (for example, PCI) described later. -Express) Connect to disk adapters 42a, 42b via 440,442. Similarly, the memory controller 420 is connected to the channel adapters 41a, 41b, 41c, and 41d via a 4-lane high-speed serial bus (for example, PCI-Express) 443, 444, 445, and 446, and connected to the 4-lane high-speed serial bus ( For example, it connects to DMAs 43-a and 43-b via PCI-Express) 447 and 448.

  As will be described later, this high-speed serial bus such as PCI-Express communicates with packets, and by providing a plurality of lanes for the serial bus, even if the signal line main line is reduced, the delay is low and the response speed is high, so-called It is possible to communicate with low latency.

  The channel adapters 41a to 41d are interfaces to the host computer, and the channel adapters 41a to 41d are connected to different host computers. Each of the channel adapters 41a to 41d is preferably connected to an interface unit of a corresponding host computer by a bus, for example, Fiber Channel or Ethernet (registered trademark). In this case, as the bus, An optical fiber or a coaxial cable is used.

  Further, each of these channel adapters 41a to 41d is configured as a part of each control module 4-0 to 4-7, but serves as an interface between the corresponding host computer and the control modules 4-0 to 4-7. Need to support multiple protocols. Since the protocols to be implemented by the corresponding host computers are not the same, the cache manager 40, which is the main unit of the control modules 4-0 to 4-7, can be easily replaced as necessary. Is mounted on another printed circuit board as will be described later with reference to FIG.

  For example, as described above, the protocol between the host computers to be supported by the channel adapters 41a to 41d includes Fiber Channel, iSCSI (Internet Small Computer System Interface) corresponding to Ethernet (registered trademark), and the like.

  Further, as described above, each of the channel adapters 41a to 41d is directly connected to the cache manager 40 by a bus designed to connect between an LSI (Large Scale Integration) and a printed circuit board, such as a PCI-Express bus. Are combined. Thereby, the high throughput requested | required between each channel adapter 41a-41d and the cache manager 40 is realizable.

  The disk adapters 42a and 42b are interfaces to the disk drives of the disk enclosures 2-0 to 2-25, and are connected to the BRTs 5-0 to 5-7 connected to the disk enclosures 2-0 to 2-25. It has 4 FC (Fiber Channel) ports.

  Further, as described above, each of the disk adapters 42a and 42b is directly connected to the cache manager 40 by a bus designed to connect between an LSI (Large Scale Integration) and a printed circuit board, such as a PCI-Express bus. Are combined. As a result, a high throughput required between the disk adapters 42a and 42b and the cache manager 40 can be realized.

  As shown in FIGS. 1 and 3, the BRTs 5-0 to 5-7 selectively select the disk adapters 42a and 42b of the control modules 4-0 to 4-7 and the disk enclosures 2-0 to 2-25. It is a multi-port switch that is connected to be communicable.

  As shown in FIG. 3, each of the disk disk enclosures 2-0 to 2-7 is connected with a plurality (two in this case) of BRTs 5-0 and 5-1. As shown in FIG. 4, each disk enclosure 2-0 is equipped with a plurality of disk drives 200 each having two ports, and this disk enclosure 2-0 has four connection ports 210, 212, and 214. , 216, unit disk enclosures 20-0 to 23-0. These are connected in series to realize an increase in capacity.

  In the disk enclosures 20-0 to 23-0, each port of each disk drive 200 is connected to the two ports 210 and 212 by a pair of FC cables from the two ports 210 and 212. These two ports 210 and 212 are connected to different BRTs 5-0 and 5-1, as described in FIG.

  As shown in FIG. 1, the disk adapters 42a and 42b of the control modules 4-0 to 4-7 are connected to all the disk enclosures 2-0 to 2-25, respectively. That is, the disk adapter 42a of each control module 4-0 to 4-7 includes the BRT 5-0 (see FIG. 3) connected to the disk enclosures 2-0 to 2-7, and the disk enclosures 2-8 and 2-9. To BRT5-2 connected to, BRT5-4 connected to disk enclosures 2-16 and 2-17, and BRT5-6 connected to disk enclosures 2-24 and 2-25, respectively. Is done.

  Similarly, the disk adapter 42b of each control module 4-0 to 4-7 includes a BRT5-1 (see FIG. 3) connected to the disk enclosures 2-0 to 2-7, a disk enclosure 2-8, BRT5-3 connected to 2-9 to, BRT5-5 connected to the disk enclosures 2-16 and 2-17, and BRT5-7 connected to the disk enclosures 2-24 and 2-25 to Are connected to each other.

  As described above, each of the disk enclosures 2-0 to 2-31 is connected to a plurality of (two in this case) BRTs, and two BRTs connected to the same disk enclosure 2-0 to 2-31. Different disk adapters 42a and 42b in the same control module 4-0 to 4-7 are connected to each.

  With such a configuration, the control modules 4-0 to 4-7 can access all the disk enclosures (disk drives) 2-0 to 2-31 through any of the disk adapters 42a and 42b.

  Each of these disk adapters 42a and 42b is configured as a part of the control modules 4-0 to 4-7, and is on the board of the cache manager 40 which is a main unit of the control modules 4-0 to 4-7. The disk adapters 42a and 42b are directly coupled to the cache manager 40 by, for example, a PCI (Peripheral Component Interconnect) -Express bus, thereby requesting between the disk adapters 42a and 42b and the cache manager 40. High throughput can be achieved.

  Further, as shown in FIG. 2, each of the disk adapters 42a and 42b is connected to the corresponding BRT 5-0 to 5-7 by a bus, for example, Fiber Channel or Ethernet (registered trademark). In this case, the bus is provided by electrical wiring on the printed circuit board of the back panel, as will be described later.

  Between the disk adapters 42a and 42b of the control modules 4-0 to 4-7 and the BRTs 5-0 to 5-7, as described above, since all the disk enclosures are connected, one-to-one mesh connection is used. Therefore, as the number of control modules 4-0 to 4-7 (that is, the number of disk adapters 42a and 42b) increases, the number of connections increases and the connection relationship becomes complicated, and physical mounting becomes difficult. Become. However, by adopting a fiber channel with a small number of signals constituting the interface for connection between the disk adapters 42a and 42b and the BRTs 5-0 to 5-7, mounting on a printed circuit board becomes possible.

  When the disk adapters 42a and 42b and the corresponding BRTs 5-0 to 5-7 are connected by a fiber channel, the BRTs 5-0 to 5-7 are fiber channel switches. Also, the BRTs 5-0 to 5-7 and the corresponding disk enclosures 2-0 to 2-31 are connected by, for example, fiber channels. In this case, the modules are different, so that the optical cables 500 and 510 are used. Connected.

  As shown in FIG. 1, the DMA engine 43 communicates with the other control modules 4-0 to 4-7, and communicates with other control modules 4-0 to 4-7 and data. Responsible for transfer processing. Each of the DMA engines 43 of the control modules 4-0 to 4-7 is configured as a part of the control modules 4-0 to 4-7, and is a cache that is a main unit of the control modules 4-0 to 4-7. It is mounted on the substrate of the manager 40. Then, it is directly coupled to the cache manager 40 by the above-described high-speed serial bus, and communicates with the DMA engines 43 of the other control modules 4-0 to 4-7 via the FRTs 6-0 and 6-1.

  The FRTs 6-0 and 6-1 are connected to the DMA engines 43 of a plurality (particularly three or more, here eight) of the control modules 4-0 to 4-7, and the control modules 4-0 to 4-7 are connected to each other. The connection is made selectively so that communication is possible.

  With such a configuration, each of the DMA engines 43 of the control modules 4-0 to 4-7 allows the cache manager 40 and other control modules 4- connected to the DMA engine 43 via the FRTs 6-0 and 6-1. Communication or data transfer processing (for example, mirroring processing) that occurs in response to an access request from the host computer or the like is executed with the cache manager 40 of 0 to 4-7.

  As shown in FIG. 2, the DMA engines 43 of the control modules 4-0 to 4-7 are composed of a plurality of (here, two) DMA engines 43-a and 43-b, and these two DMA engines. Each of 43-a and 43-b uses two FRTs 6-0 and 6-1.

  Further, as described above, the DMA engines 43-a and 43-b are connected to the cache manager 40 by, for example, a PCI-Express bus, and realize low latency.

  Further, in communication and data transfer processing between the control modules 4-0 to 4-7 (that is, between the cache managers 40 of the control modules 4-0 to 4-7), the amount of data transfer is large and the time required for communication is large. It is desirable to shorten the period, and low latency (fast response speed) is required at the same time as high throughput. For this reason, as shown in FIGS. 1 and 2, the DMA engine 43 and the FRTs 6-0 and 6-1 of each control module 4-0 to 4-7 satisfy both requirements of high throughput and low latency. It is connected by a bus (PCI-Express or Rapid-IO) that uses high-speed serial transmission designed for this purpose.

  These PCI-Express and Rapid-IO use 2.5 Gbps high-speed serial transmission, and a low-amplitude differential interface called LVDS (Low Voltage Differential Signaling) is adopted for these bus interfaces.

[Read / write processing]
Next, read processing of the data storage system of FIGS. 1 to 4 will be described. FIG. 5 is an explanatory diagram of the read operation of the configuration of FIGS.

  First, when the cache manager 40 receives a read request from any of the host computers via the corresponding channel adapters 41a to 41d, if the cache memory 40b holds the target data of the read request, the cache memory 40b Is sent to the host computer via the channel adapters 41a to 41d.

  On the other hand, if the target data is not held in the cache memory 40b, the cache control unit 40a reads the target data from the disk drive 200 holding the target data onto the cache memory 40b, and then stores the target data. To the host computer that issued the read request.

  The read process with this disk drive will be described with reference to FIG.

  (1) The control unit 40a (CPU) of the cache manager 40 creates an FC header and a descriptor in the descriptor area of the cache memory 40b. The descriptor is an instruction for requesting data (DMA) transfer to the data transfer circuit (DMA circuit), the address on the cache memory of the FC header, the address on the cache memory of the data to be transferred and the number of data bytes, Contains the logical address of the disk for data transfer.

  (2) The data transfer circuit of the disk adapter 42 is activated.

  (3) The activated data transfer circuit of the disk adapter 42 reads the descriptor from the cache memory 40b.

  (4) The activated data transfer circuit of the disk adapter 42 reads the FC header from the cache memory 40b.

  (5) The activated data transfer circuit of the disk adapter 42 decodes the descriptor, obtains the requested disk, head address, and number of bytes, and transfers the FC header to the target disk drive 200 from the fiber channel 500 (510). To do. The disk drive 200 reads the requested target data and transmits it to the data transfer circuit of the disk adapter 42 via the fiber channel 500 (510).

  (6) The disk drive 200 reads the requested target data, and when the transmission is completed, the disk drive 200 transmits a completion notification to the data transfer circuit of the disk adapter 42 via the fiber channel 500 (510).

  (7) Upon receiving the completion notification, the activated data transfer circuit of the disk adapter 42 reads the read data from the memory of the disk adapter 42 and stores it in the cache memory 40b.

  (8) When the read transfer is completed, the activated data transfer circuit of the disk adapter 42 notifies the cache manager 40 of completion by interruption.

  (9) The control unit 42a of the cache manager 40 obtains the interrupt factor of the disk adapter 42 and confirms the read transfer.

  (10) The control unit 42a of the cache manager 40 checks the end pointer of the disk adapter 42 and confirms the completion of the read transfer.

  As described above, in order to obtain sufficient performance, all connections need to have high throughput, but there are many signal exchanges between the cache control unit 40a and the disk adapter 42 (in the figure, 7). Times), especially low latency buses are required.

  In this example, both PCI-Express (4 lanes) and Fiber Channel (4G) are used as high-throughput connections, but PCI-Express is a low-latency connection, whereas Fiber Channel is a comparison. A connection with a large dynamic latency (data transfer takes time).

  Therefore, in the second conventional technique, the RT 14 (see FIG. 12) between the CM 10 and the DA 13 and the CA 11 cannot employ a high-latency Fiber Channel. In the present invention, the configuration shown in FIG. Fiber Channels can be adopted for BRTs 5-0 to 5-7.

  In order to achieve a low latency, the number of bus signals cannot be reduced to some extent. However, in the present invention, a Fiber Channel with a small number of signal lines is used for the connection between the disk adapter 42 and the BRT 5-0. This reduces the number of signals on the back panel and is effective in mounting.

  Next, the write operation will be described. When a write request is received from any of the host computers via the corresponding channel adapters 41a to 41d, the channel adapters 41a to 41d that have received the write request command and write data write to the cache manager 40. The address of the cache memory 40b to which data is to be written is inquired.

  When the channel adapters 41a to 41d receive a response from the cache manager 40, the write data is written into the cache memory 40b of the cache manager 40, and at least one cache manager 40 different from the cache manager 40 (that is, different) Write data is also written to the cache memory 40b in the cache manager 40) of the control modules 4-0 to 4-7. For this reason, the DMA engine 43 is activated, and write data is written into the cache memory 40b in the cache manager 40 of the other control modules 4-0 to 4-7 via the FRTs 6-0 and 6-1.

  Here, the write data is written to the cache memory 40b of at least two different control modules 4-0 to 4-7 by duplicating the data (mirroring), thereby causing the unexpected control modules 4-0 to 4- This is to prevent data loss even in the case of 7 or a hardware failure of the cache manager 40.

  Finally, when the writing of the write data to the plurality of cache memories 40b ends normally, the channel adapters 41a to 41d notify the host computers 3-0 to 3-31 of completion, and the processing ends.

  Furthermore, it is necessary to write back this write data to the target disk drive (referred to as write back). The cache control unit 40a writes back the write data in the cache memory 40b to the disk drive 200 holding the target data according to the internal schedule. The write process with this disk drive will be described with reference to FIG.

  (1) The control unit 40a (CPU) of the cache manager 40 creates an FC header and a descriptor in the descriptor area of the cache memory 40b. The descriptor is an instruction for requesting data (DMA) transfer to the data transfer (DMA) circuit. The address on the cache memory of the FC header, the address on the cache memory of the data to be transferred, the number of data bytes, the data Contains the logical address of the transfer disk.

  (2) The data transfer circuit of the disk adapter 42 is activated.

  (3) The activated data transfer circuit of the disk adapter 42 reads the descriptor from the cache memory 40b.

  (4) The activated data transfer circuit of the disk adapter 42 reads the FC header from the cache memory 40b.

  (5) The activated data transfer circuit of the disk adapter 42 decodes the descriptor, obtains the requested disk, head address, and number of bytes, and reads the data from the cache memory 40b.

  (6) After the reading is completed, the data transfer circuit of the disk adapter 42 transfers the FC header and data to the target disk drive 200 from the fiber channel 500 (510). The disk drive 200 writes the transferred data to a built-in disk.

  (7) Upon completion of data writing, the disk drive 200 transmits a completion notification to the data transfer circuit of the disk adapter 42 via the fiber channel 500 (510).

  (8) Upon receiving the completion notification, the activated data transfer circuit of the disk adapter 42 notifies the cache manager 40 of completion by interruption.

  (9) The control unit 42a of the cache manager 40 obtains the interrupt factor of the disk adapter 42 and confirms the write operation.

  (10) The control unit 42a of the cache manager 40 checks the end pointer of the disk adapter 42 and confirms the completion of the write operation.

  In both FIG. 6 and FIG. 5, the arrow indicates the transfer of a packet such as data, and the U-shaped arrow indicates the read of data, and data is sent back in response to one data request. Of As described above, since it is necessary to confirm the start and end state of the control circuit in the DA, seven exchanges are performed between the CM 40 and the DA 42 to perform one data transfer. There are two times between the DA 42 and the disk 200.

  Thus, it can be understood that a low latency is required for the connection between the cache control unit 40 and the disk adapter 42, while the disk adapter 42 and the disk device 200 can use an interface with a small number of signals.

[Mounting structure]
FIG. 7 is a diagram showing an example of the mounting configuration of the control module according to the present invention, FIG. 8 is a diagram showing an example of the mounting configuration including the control module and the disk enclosure of FIG. 7, and FIGS. 1 is a block diagram of a data storage system.

  As shown in FIG. 8, four disk enclosures 2-0, 2-1, 2-8, 2-9 are mounted on the upper side of the housing of the storage apparatus. The lower half of the storage device is equipped with a control circuit. As shown in FIG. 7, the lower half is divided back and forth by the back panel 7. Slots are provided in front and rear of the back panel 7, respectively. In the large-scale storage system of FIG. 9, eight (8) CMs 4-0 to 4-7 are on the front side (Front), and two (2) FRTs 6-0, 6-1 and 8 (eight) BRTs 5-0 to 5-7 and a service processor SVC (not shown in FIGS. 1 and 9) in charge of power control and the like are arranged.

  In FIG. 7, eight CMs 4-0 to 4-7 and two FRTs 6-0 and 6-1 are connected via the back panel 7 by 4-lane PCI-Express. PCI-Express is four signal lines (because of differential and bi-directional), and there are 16 signal lines for 4 lanes, so the number of signals is 16 × 16 = 256. Also, eight CMs 4-0 to 4-7 and eight BRTs 5-0 to 5-7 are connected via the back panel 7 via Fiber Channel. Since the Fiber Channel is differential and bidirectional, there are 1 × 2 × 2 = 4 signal lines, and the number of signals is 8 × 8 × 4 = 256.

  In this way, by using a different bus for each connection location, even in a large-scale storage system as shown in FIG. 9, eight CM4-0 to 4-7, two FRT6-0 and 6-1, BRT5 Eight connections of −0 to 5-7 can be realized by 512 signal lines. The number of signal lines is the number of signals that can be sufficiently mounted on the back panel substrate 7, and the number of signal layers of the substrate is sufficient with six layers, and is in a range that can be realized in terms of cost.

  In FIG. 8, four disk enclosures 2-0, 2-1, 2-8, and 2-9 (see FIG. 9) are mounted, but the other disk enclosures 2-3 to 2-7, 2- 10-2-31 are provided in another housing.

  Furthermore, even a medium-scale storage system as shown in FIG. 10 can be realized with the same configuration. That is, even in the configuration of four CM4-0 to 4-3, four BRT5-0 to 5-3, two FRT6-0 to 6-1 and 16 module disk enclosures 2-0 to 2-15. Can be realized with the same architecture.

  Moreover, each of the disk adapters 42a and 42b of each control module 4-0 to 4-7 is connected to all the disk drives 200 by BRT, and each control module 4-0 to 4-7 is connected to any disk adapter 42a. , 42b, all the disk drives can be accessed.

  Each of these disk adapters 42a and 42b is mounted on the board of the cache manager 40 which is a main unit of the control modules 4-0 to 4-7, and each of the disk adapters 42a and 42b is a low level such as PCI-Express. The latency bus can be directly coupled to the cache manager 40 to achieve high throughput.

  Further, since the one-to-one mesh connection is made between the disk adapters 42a and 42b of each control module 4-0 to 4-7 and the BRT 5-0 to 5-7, the control module 4-0 provided in the system. Even if the number of 4-7 (that is, the number of disk adapters 42a and 42b) increases, the number of signals constituting the interface is small in the connection between the disk adapters 42a and 42b and the BRTs 5-0 to 5-7. A fiber channel can be employed and the mounting problem can be solved.

  Further, in communication and data transfer processing between the control modules 4-0 to 4-7 (that is, between the cache managers 40 of the control modules 4-0 to 4-7), the amount of data transfer is large and the time required for communication is large. It is desirable to shorten the DMA engine 43 of each of the control modules 4-0 to 4-7 and the FRTs 6-0, 6 as shown in FIG. -1 is connected by a bus PCI-Express using high-speed serial transmission, which is designed to satisfy the requirements of both high throughput and low latency.

[Other embodiments]
In the above-described embodiment, the signal lines in the control module have been described by PCI-Express, but other high-speed serial buses such as Rapid-IO can be used. The number of channel adapters and disk adapters in the control module can be increased or decreased as necessary.

  As the disk drive, a storage device such as a hard disk drive, an optical disk drive, or a magneto-optical disk drive can be applied.

  As mentioned above, although this invention was demonstrated by embodiment, in the range of the meaning of this invention, this invention can be variously deformed, These are not excluded from the scope of the present invention.

  (Additional remark 1) It has a some storage device which memorize | stores data, and a some control module which carries out access control of the said storage device according to the access instruction from a high-order, The said control module is memorize | stored in the said storage device Interface control with a plurality of storage devices, a cache memory that stores a part of the data, a cache control unit that controls the cache memory, a first interface unit that performs interface control with the host, and the storage device A second interface unit, and is provided between the plurality of control modules and the plurality of storage devices, and selectively selects the second interface unit and the plurality of storage devices of each control module. A plurality of first switch units for switching to the plurality of control modules. Data storage system is characterized in that connecting the first switch unit le and the plurality in the back panel.

  (Supplementary Note 2) The control module connects the cache control unit and the second interface unit with a low-latency high-speed serial bus, and connects the second interface unit and the plurality of first switch units. The data storage system according to appendix 1, wherein the back panel is connected by a serial bus.

  (Additional remark 3) The said control module has a communication unit for communicating with the said other control module, and also has a 2nd switch unit which selectively connects the communication unit of each said control module. The data storage system of appendix 1.

  (Additional remark 4) The data storage system of Additional remark 3 characterized by connecting the communication unit of each control module, and the 2nd switch unit with the said back panel.

  (Supplementary note 5) The data storage system according to supplementary note 1, wherein the first switch unit and the plurality of storage devices are connected by a cable.

  (Supplementary note 6) The data storage system according to supplementary note 1, wherein the storage device has a plurality of access ports, and a plurality of different first switch units are connected to the plurality of access ports.

  (Supplementary Note 7) The control module connects the cache control unit and the second interface unit with a high-speed serial bus having a plurality of lanes, and includes the second interface unit and the plurality of first switch units. The data storage system according to appendix 2, wherein the back panel is connected by a serial bus.

  (Supplementary note 8) The data storage system according to supplementary note 2, wherein the high-speed serial bus is a PCI-Express bus.

  (Supplementary note 9) The data storage system according to supplementary note 2, wherein the serial bus is a fiber channel.

  (Supplementary note 10) The data storage system according to supplementary note 2, wherein the control module connects the cache control unit and the first interface unit with a low-latency high-speed serial bus.

  (Supplementary Note 11) In a data storage control apparatus that controls access to a plurality of storage devices that store data in response to an access instruction from a host, a cache memory that stores a part of the data stored in the storage device, A plurality of control modules having a cache control unit that controls the cache memory, a first interface unit that performs interface control with the host, and a second interface unit that performs interface control with the plurality of storage devices; A plurality of first switch units that are provided between the plurality of control modules and the plurality of storage devices and selectively switch between the second interface unit of each control module and the plurality of storage devices. And the plurality of control modules and the plurality of first modules. Data storage control apparatus according to claim and a switch unit that is connected with the back panel.

  (Supplementary Note 12) The control module connects the cache control unit and the second interface unit with a low-latency high-speed serial bus, and connects the second interface unit and the plurality of first switch units. The data storage control device according to appendix 11, wherein the back panel is connected by a serial bus.

  (Additional remark 13) The said control module has a communication unit for communicating with the said other control module, and also has a 2nd switch unit which selectively connects the communication unit of each said control module. The data storage control device according to appendix 11.

  (Supplementary note 14) The data storage control device according to supplementary note 13, wherein a communication unit and a second switch unit of each control module are connected by the back panel.

  (Supplementary note 15) The data storage control device according to supplementary note 1, wherein the first switch unit and the plurality of storage devices are connected by a cable.

  (Supplementary note 16) The data storage control device according to supplementary note 11, wherein a plurality of different first switch units are connected to each of the storage devices having a plurality of access ports.

  (Supplementary Note 17) The control module connects the cache control unit and the second interface unit with a high-speed serial bus having a plurality of lanes, and also includes the second interface unit and the plurality of first switch units. The data storage control device according to appendix 12, wherein the back panel is connected by a serial bus.

  (Supplementary note 18) The data storage control device according to supplementary note 12, wherein the high-speed serial bus is a PCI-Express bus.

  (Supplementary note 19) The data storage control device according to supplementary note 12, wherein the serial bus is a fiber channel.

  (Supplementary note 20) The data storage control device according to supplementary note 12, wherein the control module connects the cache control unit and the first interface unit with a low-latency high-speed serial bus.

Connect each second interface of the plurality of control modules to the plurality of first switch units, each of the pair of signal lines of the head unit disk enclosures of each of the plurality of disk enclosures, the different first switch units As a result, all control modules can maintain redundancy to access all storage devices, and even if the number of control modules and disk enclosures increases in this way, unit disk enclosures are connected in series. Since the control module and the first switch unit can be connected by a back panel and connected by a serial bus with a small number of signals constituting the interface, mounting on a printed circuit board is possible. For this reason, it is effective in unifying the architecture from a large scale to a small scale, and can contribute to the cost reduction of the apparatus.

1 Storage system 2-0 to 2-35 Disk enclosure 4-0 to 4-7 Control unit 5-0 to 5-7 Backend router 6-0 to 6-1 Frontend router 7 Back panel 40 Control module 40a Cache control Unit 40b Cache memory 41 Channel adapter 42 Device adapter 43 Communication unit (DMA engine)

Claims (2)

A plurality of disk enclosures configured by connecting a plurality of disk storage devices for storing data in parallel to a pair of signal lines ;
A plurality of control modules for controlling access to the disk storage device in response to an access instruction from a host;
Each of the control modules
A first interface unit that performs interface control with the host;
A second interface unit for performing interface control with the disk storage device;
A control unit for controlling access to the disk storage device from the second interface unit in response to the access instruction;
A plurality of first switch units that are connected to the second interface unit of each of the control modules and selectively switch between the second interface unit and the plurality of disk enclosures of each of the control modules. A first substrate;
A plurality of second substrates provided with at least the control unit and the second interface unit;
Wherein a plurality of first substrate, said plurality of second substrate is mounted, and a back panel for connecting each said second interface in a separate serial bus to each of the plurality of first switch units Provided ,
Each of the plurality of disk enclosures is configured by connecting a plurality of unit disk enclosures in which a plurality of disk storage devices are connected in parallel to a pair of signal lines, and the first unit of each of the plurality of disk enclosures A data storage system , wherein each of a pair of signal lines of a disk enclosure is connected to a different first switch unit . The first substrate is
A communication unit connected to the control unit by a bus and communicating with the other control module;
Furthermore, a second switch unit for selectively connecting the communication units of the respective control modules is mounted on the back panel,
The data storage system according to claim 1 , wherein the communication unit of each control module is electrically connected to the second switch unit by the back panel .

Images