JP2016194928A - Storage system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for calculating a redundant code.SOLUTION: A storage system 30 includes: a first controller 20; and multiple storage devices 50 constituting RAID. Each one of the storage devices includes non-volatile memory chips 55 and a controller 60. When the controller 20 receives a request to update first data to second data, the controller 60 of the first storage device stores the second data in an area different from the area where the first data is stored, and generates an intermediate parity from the first and second data. The controller 60 of the second storage device storing a first parity corresponding to the first data receives the intermediate parity, generates a second parity from the intermediate parity and the first parity, and stores the second parity in the second storage device. The controller 60 of the first storage device defines the area where the first data is stored as a deletion target area after the storage of the second parity.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for calculating a redundant code stored in a storage apparatus.

  A technology is known in which a storage system controller configures a Redundant Array of Independent Disks (RAID) by controlling a plurality of storage devices and prevents data loss. For example, in a storage system that employs RAID 5, a storage system controller, which is a host device of a storage device, generates parity from a plurality of data blocks.

  In order to reduce the processing of the storage system controller associated with parity generation as described above, there is a storage system in which each storage device corresponding to a lower level device constitutes a RAID. Each storage device is, for example, an HDD (Hard Disk Drive), and a controller of each HDD (hereinafter referred to as an HDD controller) generates a parity.

  For example, in Patent Document 1, when there is a data update request for pre-update data (old data), the HDD controller generates an intermediate value for generating parity from the update data (new data) and pre-update data (old data). After generating (intermediate parity), the old data is updated with the new data. The HDD controller transfers the generated intermediate parity to the HDD in which the old parity corresponding to the old data is stored. In the HDD that has received the intermediate parity, the HDD controller generates a new parity from the intermediate parity and the old parity, and updates the old parity with the generated new parity.

  With the recent price reduction of flash memory, Flash SSD (Flash Solid State Drive) is being used instead of HDD as a storage device. In the HDD, when there is an update request for data specified by a certain range of logical addresses, the update data (new data) is overwritten in the physical area in which the data (old data) is stored. However, in Flash SSD, data cannot be overwritten. Therefore, when there is an update request for data specified by a logical address, update data (new data) is stored in a physical area different from the physical area in which the data is stored. The stored and specified logical address is mapped to the new physical area. As a result, the physical area in which the old data of the Flash SSD is stored is invalidated.

US 5,191,584

  In Patent Document 1, since the old data is overwritten with the new data after the intermediate parity is generated, the old data does not exist after the generation of the intermediate parity. Therefore, if a failure occurs before the parity is updated (for example, before the intermediate parity is transferred to the HDD storing the old parity) and the intermediate parity is lost, the old data does not exist. Cannot be generated again.

  On the other hand, when the storage device is replaced with Flash SSD, the flash memory does not overwrite the data, so there is a possibility that the old data remains after the intermediate parity is generated. However, in the Flash SSD, when new data is written, the physical area storing the old data is invalidated and the old data is erased at a predetermined timing. That is, even in Flash SSD, there is a possibility that old data does not exist when a failure occurs, as in Cited Document 1, in which case intermediate parity cannot be generated.

  A storage system that is one embodiment of the present invention includes a first controller and a plurality of storage devices. Each of the plurality of storage devices constituting the RAID includes one or more nonvolatile memory chips that provide a storage space for storing data from the host computer, and a second controller connected to the nonvolatile memory chips. Yes. When the first controller receives an update request for updating the first data to the second data, the second controller in the first storage device of the plurality of storage devices stores the second data. , Storing information in a storage space of the first storage device that is different from an area in which the first data is stored, generating information for associating the first and second data, and the first And an intermediate parity is generated from the second data. The second controller in the second storage device storing the first parity corresponding to the first data among the plurality of storage devices receives the intermediate parity, and receives the intermediate parity and the first parity. The second parity is generated from the second storage device, and the second parity is stored in the storage space area of the second storage device. The second controller of the first storage device erases the information and stores the first data after the second parity is stored in the storage space area of the second storage device. An area is set as an erasure target area.

FIG. 1 shows an example of the configuration of a storage system. FIG. 2 shows an example of the configuration of the system controller 20. FIG. 3 illustrates an example of a configuration of the computer system according to the first embodiment. FIG. 4 shows an example of the configuration of the RG and LU. FIG. 5 shows an example of the RG management table 600. FIG. 6 shows an example of the LU management table 700. FIG. 7 shows an example of the FMPK management table 800. FIG. 8 shows an example of an address space on the RG. FIG. 9 shows an example of an address space on the FMPK 50 belonging to the RG. FIG. 10 shows an example of page mapping in the FMPK50. FIG. 11 shows an example of the page mapping management table 1100. FIG. 12 shows an example of the parity operation function registration process. FIG. 13 shows an example of a method 1300 for determining the parity operation execution apparatus. FIG. 14 shows an example of the write process of the system controller 20. FIG. 15 shows an example of the first parity calculation selection process. FIG. 16 shows an example of the second parity calculation selection process. FIG. 17 shows an example of the third parity calculation selection process. FIG. 18 shows an example of the fourth parity calculation selection process. FIG. 19 shows an example of the write method selection processing 1403. FIG. 20 shows an example of the first read-modify-write process. FIG. 21 shows an example of the operation of the system controller 20 in the first read-modify-write process. FIG. 22 shows an example of the second read-modify-write process. FIG. 23 shows an example of the operation of the system controller 20 in the second read-modify-write process. FIG. 24 shows an example of the third read modify write process. FIG. 25 shows an example of the operation of the system controller 20 in the third read-modify-write process. FIG. 26 shows an example of the first full stripe write process. FIG. 27 shows an example of the operation of the system controller 20 in the first full stripe write process. FIG. 28 shows an example of the second full stripe write process. FIG. 29 shows an example of the operation of the system controller 20 in the second full stripe write process. FIG. 30 shows an example of the first data recovery process. FIG. 31 shows an example of the operation of the system controller 20 in the first data recovery process. FIG. 32 shows an example of the second data recovery process. FIG. 33 shows an example of the operation of the system controller 20 in the second data recovery process. FIG. 34 shows an example of processing of the device controller 60 by a normal write command. FIG. 35 shows an example of processing of the device controller 60 by the old data holding write command. FIG. 36 shows an example of processing of the device controller 60 by the intermediate parity read command. FIG. 37 shows an example of processing of the device controller 60 by the parity update write command. FIG. 38 shows an example of processing of the device controller 60 by the parity generation write command. FIG. 39 shows an example of old data discard processing by the device controller 60. FIG. 40 shows an example of a Q parity generation method. FIG. 41 shows an example of a recovery method for D0 and D1. FIG. 42 shows an example of the fourth read-modify-write process. FIG. 43 shows an example of the operation of the system controller 20 in the fourth read-modify-write process. FIG. 44 illustrates an example of a configuration of a computer system according to the third embodiment. FIG. 45 illustrates an example of a configuration of a computer system according to the fourth embodiment.

  Several examples will be described. The technical scope of the present invention is not limited to each example.

  In the following description, various types of information may be described using the expression “*** table”, but the various types of information may be expressed using a data structure other than a table. In order to show that various types of information do not depend on the data structure, the “*** table” can be called “*** information”.

  In the following description, the process may be described using “program” as a subject. However, the program is executed by a processor (for example, a CPU (Central Processing Unit)), so that a predetermined process is appropriately performed. Since the processing is performed using a storage resource (for example, a memory) and a communication control device (for example, a communication port), the subject of processing may be a processor. Further, the processing described with the program as the subject may be processing performed by the management system. Further, part or all of the program may be realized by dedicated hardware. For this reason, the processing described with the program as the subject may be processing performed by the controller. The controller may include a processor and a storage resource that stores a computer program executed by the processor, or may include the dedicated hardware described above. Further, the computer program may be installed in each computer from a program source. The program source may be, for example, a program distribution server or a storage medium.

  In the following description, the management system is one or more computers, for example, a management computer, or a combination of a management computer and a display computer. Specifically, for example, when the management computer displays display information, the management computer is a management system. In addition, in order to increase the processing speed and reliability, a function equivalent to that of the management computer may be realized by a plurality of computers. In this case, the plurality of computers (in the case where the display computer performs display, the display is displayed). A management system).

  First, the configuration of a storage system that is an application example of the present invention will be described.

  FIG. 1 shows an example of the configuration of the storage system 30. The storage system 30 includes a plurality of storage devices 50 that store data, and a system controller 20 that is connected to the plurality of storage devices. The storage device is, for example, an FMPK (Flash Memory Package) 50.

  The FMPK 50 includes a nonvolatile memory that stores data, and a device controller 60 that is connected to the nonvolatile memory and the system controller 20. The nonvolatile memory is, for example, a NAND type FM (Flash Memory) 55. The nonvolatile memory is not limited to the flash memory, and may be a write-once memory (for example, a phase change memory). The device controller 60 includes a communication interface device, a storage device, and a control device connected to them. The communication interface devices are, for example, the system interface 53 and the FM interface 54. In the following description, the interface may be indicated as I / F. The system I / F 53 is connected to the system controller 20. The FM I / F 54 is connected to the FM 55.

  The storage device is, for example, the memory 52 and the buffer 66. The control device is, for example, the CPU 51. The control device may include a dedicated hardware circuit that performs predetermined processing (for example, compression, expansion, encryption, or decryption) in addition to a processor such as the CPU 51. The dedicated hardware circuit is, for example, a parity calculation circuit 65 that calculates parity or intermediate parity. The memory 52 stores a program for controlling the FM 55 and various information. The CPU 51 implements various functions by executing programs stored in the memory 52. The buffer 66 is, for example, a volatile memory such as a DRAM (Dynamic Random Access Memory). The buffer 66 temporarily stores data to be written to the FM 55, data read from the FM 55, data being subjected to parity calculation, and the like.

  The FMPK50 may have a plurality of FM55. The plurality of FMs 55 may include different types of storage media or the same type of storage media. For example, the FM 55 is composed of a plurality of physical blocks. Each physical block is composed of a plurality of physical pages. Each cell in the FM 55 may be an SLC (Single Level Cell) or an MLC (Multiple Level Cell). The nonvolatile memory may be another nonvolatile memory or a recordable memory such as a phase change memory.

  FIG. 2 shows the configuration of the system controller 20. The system controller 20 includes a communication interface device, a storage device, and a control device connected to them. The communication interface device is, for example, a communication I / F 18 for connecting to a communication network and a disk I / F 19 for connecting to the FMPK 50. The storage device is, for example, the memory 12 and the buffer 26. The control device is, for example, the CPU 11. The control device may include a dedicated hardware circuit that performs predetermined processing (for example, compression, expansion, encryption, or decryption) in addition to a processor such as the CPU 11. The dedicated hardware circuit is, for example, a parity operation circuit 25 that calculates either parity or intermediate parity. The memory 12 stores a program for controlling the FMPK 50 and various information. The CPU 11 realizes various functions by executing programs based on information stored in the memory 12. The buffer 26 is a volatile memory such as a DRAM. The buffer 26 temporarily stores data to be written to the FMPK 50, data read from the FMPK 50, data being subjected to parity calculation, and the like. When all the FMPKs 50 connected to the system controller 20 have the parity calculation circuit 65, the system controller 20 may not have the parity calculation circuit 25.

  Functions provided by a conventional system controller corresponding to the system controller 20 include parity calculation, snapshot, data compression, deduplication, and the like. A storage medium such as the FMPK 50 can realize these functions. In this embodiment, a parity calculation function will be described.

  Since the system controller 20 and the device controller 60 form a hierarchy, the system controller 20 can be defined as an upper controller and the device controller 60 as a lower controller.

  Next, a computer system, which is an application example of the storage system 30 when the system controller 20 is a controller that supports a RAID (Redundant Array of Independent Disks) function, will be described.

  FIG. 3 illustrates an example of a configuration of the computer system according to the first embodiment. This computer system has a host computer 10 and a storage system 30. The host computer 10 and the storage system 30 are connected via a communication network such as a SAN (Storage Area Network) 1. Note that the computer system may include a plurality of host computers 10. In this case, the storage system 30 is connected to a plurality of host computers 10 via the SAN 1.

  The storage system 30 includes a plurality of FMPKs 50 and a system controller 20 that controls these FMPKs 50. The storage device is, for example, FMPK50. In this example, the system controller 20 is, for example, a RAID controller. In this example, the storage system 30 includes a plurality of system controllers 20. Each system controller 20 is connected to the host computer 10 via the SAN 1. Each of the plurality of FMPKs 50 is connected to a plurality of system controllers 20. Note that the storage system 30 may have only one system controller 20.

  The configuration of the system controller 20 is the same as the configuration shown in FIG. The memory 12 further stores a program for a RAID function using a plurality of FMPKs 50 and various information. In this example, the system controller 20 includes the parity calculation circuit 25, but may not include the parity calculation circuit 25.

  The host computer 10 may be a management system.

  Hereinafter, a case where the system controller 20 performs RAID5 control will be described.

  The system controller 20 is associated with an RG (RAID Group), an LU (Logical Unit, sometimes called a logical volume), and an FMPK 50. FIG. 4 shows an example of the configuration of the RG and LU. The system controller 20 allocates some of the FMPKs 50 to the RG and allocates a part or all of the storage area of the RG to the LU. The logical volume may be a virtual volume in which the volume capacity is virtualized by thin provisioning technology. A physical storage area for storing data is not allocated in advance to the virtual volume. In response to a write request to the virtual volume, a physical storage area is allocated to the virtual volume in a predetermined unit.

  FIG. 5 shows an example of the RG management table 600. FIG. 6 shows an example of the LU management table 700. FIG. 7 shows an example of the FMPK management table 800. The system controller 20 writes the relationship among the RG (RAID Group), LU (Logical Unit), and FMPK 50 to the RG management table 600, LU management table 700, and FMPK management table 800 in the memory 12.

  The RG management table 600 has a record for each RG. A record of an RG indicates an RG number 601 indicating the RG, an FMPK number 602 indicating the FMPK 50 allocated to the RG, and a RAID level 603 of the RG.

  The LU management table 700 has a record for each LU. A record of an LU includes an LU number 701 indicating the LU, an RG number 702 indicating the RG allocated to the LU, a stripe size 703 which is the size of the stripe block allocated to the LU, and the LU An LU start address 704 that is a start logical address, an LU size 705 that is the size of this LU, and an IO characteristic 706 of this LU are shown. The IO characteristic 706 is the tendency of the IO pattern performed on this LU, and indicates whether the sequential tendency is strong or the random tendency is strong.

  The IO characteristic 706 may be specified in advance by the user or may be determined by the system controller 20. Further, by combining these, the initial value of the IO characteristic 706 may be designated in advance by the user, and the system controller 20 may optimize the initial value by determination after a predetermined time has elapsed. Further, as a method for determining the IO characteristic 706, for example, the system controller 20 takes a statistics of a command for a certain LU for each unit time, and selects the higher one of the random I / O pattern and the sequential I / O pattern. The IO characteristic 706 may be determined. Command types include random write and sequential write. The comparison of the ratio of the random write and the ratio of the sequential write may be made by comparing the frequency of these commands or by comparing the amount of data written by these commands. In addition, the system controller 20 may monitor the IO characteristics 706 at a constant cycle by the above-described determination method, and update the LU management table 700 if necessary.

  The FMPK management table 800 has a record for each FMPK 50. A record of a certain FMPK 50 indicates an FMPK number 801 indicating the FMPK 50, an RG number 802 indicating an RG to which the FMPK 50 belongs, and parity calculation function support information 803 indicating a parity calculation function of the FMPK 50. The parity calculation function support information 803 is, for example, a flag indicating whether or not the FMPK 50 has a parity calculation function such as the parity calculation circuit 65.

  FIG. 8 shows an example of an address space on the RG. FIG. 9 shows an example of an address space on the FMPK 50 belonging to the RG. In this example, the system controller 20 allocates FMPK # 0 to FMPK # 3, which are four FMPK50, for RG # 0 of RAID5. Further, the system controller 20 assigns a continuous area to LU # 0 from the address space on RG # 0, and assigns another continuous area to LU # 1. The system controller 20 assigns stripe lines over the address spaces on FMPK # 0 to FMPK # 3, and assigns stripe blocks and parity in the order of stripe lines and FMPK numbers. Here, the system controller 20 shifts the FMPK number to which the stripe block and the parity are assigned for each stripe line. At this time, the system controller 20 writes information on RG # 0, LU # 0, and LU # 1 to the RG management table 600, LU management table 700, and FMPK management table 800.

  FIG. 10 shows an example of page mapping in the FMPK50. The logical address space on the FMPK 50 is divided into a plurality of logical pages. On the other hand, the physical address space on the FMPK 50 is divided into a plurality of physical blocks, and each physical block has a predetermined block size and is divided into a plurality of physical pages. Each logical page and each physical page has a predetermined page size.

  Instead of physical pages, other physical areas such as physical blocks may be used. Further, instead of a logical page, another logical area such as a logical unit may be used.

  FIG. 11 shows an example of the page mapping management table 1100. The device controller 60 associates the logical page with the physical page and writes the association to the page mapping management table 1100 in the memory 52. The page mapping management table 1100 has a record for each logical page. A record of a certain logical page includes a logical page number 1101 indicating this logical page, a physical page number 1102 indicating the current physical page that is the physical page currently allocated to this logical page, and the record immediately before the current physical page. An old physical page number 1103 indicating an old physical page that is a physical page allocated to the logical page is shown. That is, the old physical page associated with a certain logical page indicates the current physical page associated with this logical page one generation before. The old physical page associated with the logical page is a valid page, and the relationship is maintained until the association with the logical page is released. Since the old physical page whose association has been canceled becomes an invalid page, data in the old physical page is erased at a predetermined timing by a reclamation process or the like. Reclamation is a state in which invalid data (data stored in an invalid page) can be erased in a flash memory where data cannot be overwritten due to memory characteristics, and the page where the invalid data is stored can be rewritten. Say the process. Since data is erased in units of blocks in the flash memory, the device controller 60 moves the data in the valid page to another block and stores it in the target block when both the valid page and invalid page exist in the target block. Delete invalid data. In the present embodiment, the device controller 60 determines that the physical page 1102 and the old physical page 1103 associated with the logical page are valid pages, and executes reclamation processing.

  A specific example of the old physical page management method will be described. The device controller 60 determines whether or not the old physical page shown in the page mapping management table 1100 is unnecessary. If it is determined that a certain old physical page is not required, the device controller 60 associates the old physical page with the logical page. You may cancel. For example, the upper limit of the number of old physical pages managed by the device controller 60 is set in advance, and the device controller 60 deletes old physical pages from the page mapping management table 1100 in order of oldness. Further, the device controller 60 may delete the old physical page from the page mapping management table 1100 according to an instruction from the system controller 20. As an example of the determination criterion, the parity update accompanying the data update is completed. Further, the system controller 20 determines whether or not the data of the old physical page is necessary, and if it is determined that the old physical page is not necessary, the page mapping is performed to the device controller 60 corresponding to the old physical page. It may be instructed to delete the old physical page from the management table 1100. Further, the system controller 20 or the device controller 60 may delete the old physical page from the page mapping management table 1100 when the used capacity of the FM 55 becomes a certain value or more. Further, when the reading of the old physical page data is requested and the old physical page remains in the page mapping management table 1100, the device controller 60 can read the old physical page data.

  Next, the operation of the system controller 20 will be described.

  First, a writing method at the time of data update will be described.

  The write method is, for example, read modify write and full stripe write. The read-modify-write in this example is a process for updating the data of one stripe block designated in one stripe line. Alternatively, the full stripe write in this example is a process for updating all data in one stripe line. In the read modify write, intermediate parity is calculated from old data and new data of a specified stripe block, and new parity is calculated from the intermediate parity and old parity. Full stripe write calculates new parity from all new data. Here, old data indicates data before update, new data indicates data after update, old parity indicates parity before update, and new parity indicates parity after update. The intermediate parity is a parity in the middle of the parity calculation, and indicates a difference between the old parity and the new parity.

  Next, the parity calculation function will be described.

  The parity calculation function is, for example, the parity calculation circuit 25 and the parity calculation circuit 65, and calculates an exclusive OR between the data of two stripe blocks in one stripe line as a parity. The parity calculation function calculates an exclusive OR between the old data and the new data as an intermediate parity. Further, the parity calculation function calculates an exclusive OR between the old parity and the intermediate parity as a new parity. If the old data is Di, the new data is Di_new, and the old parity is P, the new parity P_new is expressed by the following equation.

  P_new = P + (Di + Di_new)

  Here, the operator “+” represents exclusive OR, and (Di + Di_new) represents intermediate parity. The intermediate parity is equal to an exclusive OR between all data other than Di.

  Instead of parity, other redundant codes such as a Hamming code may be used.

  Next, a parity calculation function registration process in which the system controller 20 registers the parity calculation function support information 803 in the FMPK management table 800 will be described.

  FIG. 12 shows an example of the parity operation function registration process. The system controller 20 performs a parity operation function registration process when installing a new FMPK 50 or starting up the system controller 20.

  First, the system controller 20 issues a parity operation function support confirmation command to the device controller 60 of the FMPK 50 and receives a response (1201). At this time, the device controller 60 that has received the parity operation function support confirmation command sends a response indicating whether or not the device controller 60 itself has a parity operation function to the system controller 20. Next, based on the response, the system controller 20 registers information indicating whether or not the device controller 60 has a parity calculation function in the parity calculation function support information 803 of the FMPK management table 800 (1202). The process flow ends.

  Instead of performing the parity operation function registration process, the user may register the parity operation function support information 803 in advance.

  Next, a method of determining a parity calculation execution device that performs parity calculation will be described.

  FIG. 13 shows an example of a method 1300 for determining the parity operation execution apparatus. The system controller 20 determines the system controller 20 or the device controller 60 as the parity operation execution device according to the determination method 1300. Based on the information indicating whether or not the system controller 20 has a parity calculation function such as the parity calculation circuit 25 and the parity calculation function support information 803 of the FMPK management table 800, the system controller 20 and the device controller Any one of 60 is determined as the parity operation execution device. Four cases are defined for determining the parity operation execution device. Each case has a case number 1301, system function information 1302 indicating whether the system controller 20 has a parity calculation function, device function information 1303 indicating whether the device controller 60 has a parity calculation function, and a parity calculation. Parity calculation execution information 1304 indicating an execution apparatus and an option 1305 are shown.

  Case # 1 is a case where the system controller 20 has a parity operation function, and the device controller 60 also has a parity operation function. According to the parity calculation execution information 1304, in this case, the system controller 20 can select either the system controller 20 or the device controller 60 as the parity calculation execution device. In this case, the system controller 20 performs a later-described write process based on the option 1305.

  Case # 2 is a case where the system controller 20 has a parity operation function and the device controller 60 does not have a parity operation function. According to the parity calculation execution information 1304, in this case, the system controller 20 selects the system controller 20 as the parity calculation execution device.

  Case # 3 is a case where the system controller 20 does not have a parity calculation function and the device controller 60 has a parity calculation function. According to the parity operation execution information 1304, in this case, the system controller 20 selects the device controller 60 as the parity operation execution device.

  Case # 4 is a case where the system controller 20 does not have a parity operation function, and the device controller 60 does not have a parity operation function. According to the parity calculation execution information 1304, in this case, the system controller 20 cannot select either the system controller 20 or the device controller 60 as the parity calculation execution device.

  The parity calculation function registration process and the determination of the parity calculation execution device may be omitted. For example, when the system controller 20 and all the device controllers 60 have a parity calculation function, the system controller 20 omits the parity calculation function registration process and the determination of the parity calculation execution device, and the write process in case # 1 described later I do. Also, when the parity calculation execution device is set in advance by an administrator or the like, these processes can be omitted.

  Next, the write process of the system controller 20 in cases # 1 to # 3 will be described.

  FIG. 14 shows an example of the write process of the system controller 20. First, the system controller 20 receives a write request as an IO request from an external host (1401). The external host is, for example, the host computer 10 connected to the system controller 20 via the SAN 1. Next, the system controller 20 performs a parity calculation selection process for selecting one of the system controller 20 and the device controller 60 as a parity calculation execution apparatus that performs the parity calculation (1402). In case # 2 and case # 3, since the parity operation execution device has been selected, the processing in step 1402 is omitted. Next, the system controller 20 performs a write method selection process for selecting a write method based on the received command (1403). The write method is one of a read modify write and a full stripe write. Next, the system controller 20 performs a data update process in which the device controller 60 updates data and parity according to the selected write method (1404).

  Next, some specific examples of the parity operation selection processing 1402 will be described.

  First, a first parity calculation selection process, which is a specific example of a process in which the system controller 20 selects a parity calculation execution device based on the IO characteristic 706 for each LU, will be described.

  FIG. 15 shows an example of the first parity calculation selection process. First, based on the LU management table 700, the system controller 20 determines whether or not the IO characteristic 706 of the write target LU indicates random (1501). If the IO characteristic 706 of the LU indicates random (1501, Yes), the system controller 20 determines to instruct the device controller 60 to perform parity calculation by selecting the device controller 60 as the parity calculation execution device ( 1502), the flow of this processing is terminated. On the other hand, when the IO characteristic 706 of the LU does not indicate randomness (1501, No), that is, in the case of a sequential write described later, the system controller 20 selects the system controller 20 as the parity calculation execution device, thereby 20 decides to execute the parity operation (1503), and ends this processing flow.

  In other words, the system controller 20 refers to the IO characteristic 706 of the LU management table 700 at the IO opportunity, and if the IO characteristic 706 indicates that there are more random writes than sequential writes in writing to the LU, the system controller 20 Instructs parity generation. On the other hand, when the IO characteristic 706 indicates that there are fewer random writes than sequential writes in writing to the LU, the system controller 20 performs parity generation.

  Next, a second parity calculation selection process, which is a specific example of a process in which the system controller 20 detects a hardware load of the system controller 20 and selects a parity calculation execution apparatus based on the load, will be described.

  FIG. 16 shows an example of the second parity calculation selection process. The hardware is, for example, the CPU 11, the memory 12, and the parity calculation circuit 25. First, the system controller 20 determines whether or not its hardware load is higher than the reference (1601). When it is determined that the hardware load is higher than the reference (1601, Yes), the system controller 20 selects the device controller 60 as the parity calculation execution device, and instructs the device controller 60 to perform the parity calculation (1602). The process flow ends. On the other hand, when it is determined that the hardware load is not higher than the reference (No in 1601), the system controller 20 selects the system controller 20 as the parity operation execution device (1603), and ends this processing flow.

  Here, for example, the system controller 20 measures the hardware load of the system controller 20, and determines that the hardware load is high when the measurement result exceeds a predetermined threshold. The measurement results are, for example, the usage rate of the CPU 11, the usage amount of the memory 12, the data amount input to the parity calculation circuit 25, and the like.

  Next, a third parity calculation selection process, which is a specific example of a process in which the system controller 20 selects a parity calculation execution device based on the write IO pattern, will be described.

  FIG. 17 shows an example of the third parity calculation selection process. First, the system controller 20 determines whether or not the IO pattern of the IO request is a random write (1701). If it is determined that the IO pattern is a random write (1701, Yes), the system controller 20 selects the device controller 60 as the parity operation execution device, and instructs the device controller 60 to perform the parity operation (1702). End the flow. On the other hand, when it is determined that the IO pattern is not a random write, that is, when it is determined that the IO pattern is a sequential write (1701, No), the system controller 20 selects the system controller 20 as a parity operation execution device ( 1703), the process flow ends.

  Next, a fourth parity calculation selection process, which is a specific example of a process in which the system controller 20 selects a parity calculation execution device based on the write IO pattern and the hardware load of the system controller 20, will be described.

  FIG. 18 shows an example of the fourth parity calculation selection process. First, the system controller 20 determines whether the IO pattern of the write is a random write (1801). When it is determined that the IO pattern is a random write (1801, Yes), the system controller 20 selects the device controller 60 as the parity operation execution device, and instructs the device controller 60 to perform the parity operation (1802). End the flow. On the other hand, if it is determined that the IO pattern is not a random write (1801, No), the system controller 20 determines whether the hardware load is high (1803). When it is determined that the hardware load is high (1803, Yes), the system controller 20 shifts the processing flow to 1802. On the other hand, if it is determined that the hardware load is not high (1803, No), the system controller 20 selects the system controller 20 as the parity operation execution device (1804), and ends this processing flow.

  Note that the system controller 20 may select a parity calculation execution device by combining the plurality of types of parity calculation selection processing 1402 described above.

  When the system controller 20 performs a parity operation at the time of sequential write or full stripe write, an increase in the amount of data transfer from the system controller 20 to the device controller 60 can be prevented. This can prevent a decrease in write speed.

  Next, a specific example of the write method selection processing 1403 will be described.

  FIG. 19 shows an example of the write method selection processing 1403. Options for the write method are, for example, read modify write and full stripe write. First, the system controller 20 determines whether or not the IO pattern of the write that is the IO request is a random write (1901). If it is determined that the IO pattern is random write (1901, Yes), the system controller 20 selects read-modify-write as the write method (1902), and ends this processing flow. On the other hand, if it is determined that the IO pattern is not a random write (1901, No), that is, if it is determined that the IO pattern is a sequential write, the system controller 20 selects full stripe write as the write method (1903). Then, the process flow ends.

  Next, some specific examples of the data update processing 1404 according to the selection result of the parity calculation execution device and the selection result of the write method will be described.

  First, the first read modify write process, which is a specific example of the data update process 1404 when the device controller 60 is selected as the parity calculation execution apparatus and the read modify write is selected as the write method, will be described.

  FIG. 20 shows an example of the first read-modify-write process. FIG. 21 shows an example of the operation of the system controller 20 in the first read-modify-write process. In this example, FMPKs # 0, # 1, # 2, and # 3, which are four FMPKs 50, store data D0, D1, D2, and parity P in one stripe line, respectively. Here, the state in which the system controller 20 receives new data for updating the old data of D0 by the write command is set as the initial state. In the following description, among FMPK # 0, # 1, # 2, and # 3, FMPK # 0 that stores data to be updated may be referred to as data FMPK. Of FMPK # 0, # 1, # 2, and # 3, FMPK # 3 storing parity may be referred to as parity FMPK.

  First, the system controller 20 selects FMPK # 0 (data FMPK) corresponding to D0 from FMPK # 0, # 1, # 2, and # 3, and designates old data holding and new data writing. By issuing a data holding write command to FMPK # 0, new data is transferred to FMPK # 0 (2101). Upon receiving this command, the device controller 60 of FMPK # 0 writes the new data from the system controller 20 into a physical page different from the physical page of the old data in FMPK # 0. In the normal write command, when a new physical page is assigned to a logical page, the old physical page becomes invalid, and the old data stored in the old physical page is to be erased. Therefore, the system controller 20 uses the old data holding write command instead of the normal write command in order to hold the old data until the parity update is completed. In the following description, the write command indicates a normal write command.

  Next, the system controller 20 obtains the intermediate parity from FMPK # 0 by issuing an intermediate parity read command for requesting the intermediate parity to FMPK # 0 (2102). The intermediate parity is an intermediate calculation result for generating a new parity by the parity calculation of the old parity. Upon receiving this command, the FMPK # 0 device controller 60 calculates the intermediate parity from the old data stored in the FMPK # 0 and the new data stored in the FMPK # 0, and uses the intermediate parity as a response to the system controller. Send to 20. Here, the device controller 60 of the FMPK # 0 may write the calculated intermediate parity into the buffer 66, or is different from both the physical page of the old data and the physical page of the new data in the FM55 in the FMPK # 0. You may write to a physical page.

  Next, the system controller 20 selects FMPK # 3 (parity FMPK) corresponding to P from FMPK # 0, # 1, # 2, and # 3, and issues a parity update write command for instructing parity update to FMPK #. The intermediate parity received from FMPK # 0 is transferred to FMPK # 3 (2103). Upon receiving this command, FMPK # 3 calculates a new parity from the old parity in FMPK # 3 and the intermediate parity from the system controller 20, and writes the new parity to FMPK # 3. When the device controller 60 of FMPK # 3 writes the new parity to FMPK # 3, it notifies the system controller 20 of a parity update write completion response. Here, the device controller 60 of the FMPK # 3 may write the intermediate parity from the system controller 20 into the buffer 66, or both the physical page of the old parity and the physical page of the new parity in the FM 55 in the FMPK # 3. You may write to a different physical page.

  Next, when receiving a completion response from FMPK # 3, the system controller 20 issues an old data discard command instructing invalidation of old data to FMPK # 0 (2104), and ends this flow. Upon receiving this command, the device controller 60 of FMPK # 0 deletes the physical page mapped to the logical page specified by the old data discard command from the mapping management table 1100 in order to invalidate the retained old data. .

  If the system controller 20 does not receive a completion response from the FMPK # 3 within a predetermined time after issuing the parity update write command, the system controller 20 determines that the parity update has failed, and starts from the FMPK # 0. The parity update process may be re-executed by re-acquiring the intermediate parity and retransferring the intermediate parity to FMPK # 3.

  Specifically, after issuing the parity update write command, the system controller 20 reissues the intermediate parity read command to FMPK # 0 if the completion response from FMPK # 3 is not received within a predetermined time. The intermediate parity recalculated in response to this command is received. Then, the system controller 20 reissues the parity update write command to FMPK # 3, thereby writing the recalculated intermediate parity in FMPK # 3.

  According to this processing, the data transfer between the system controller 20 and the device controller 60 in updating the data of one stripe block is as follows.

(1) The updated data is transferred from the system controller 20 to the device controller 60 of the data FMPK.
(2) The intermediate parity is transferred from the device controller 60 of the data FMPK to the system controller 20.
(3) The intermediate parity is transferred from the system controller 20 to the device controller 60 of the parity FMPK.

  As a result, data transfer between the system controller 20 and the device controller 60 can be suppressed. Alternatively, when the system controller 20 performs a parity operation, data transfer between the system controller 20 and the device controller 60 is as follows.

(1) Data before update is transferred from the device controller 60 of the data FMPK to the system controller 20.
(2) The parity before update is transferred from the device controller 60 of the parity FMPK to the system controller 20.
(3) The updated data is transferred from the system controller 20 to the device controller 60 of the data FMPK.
(4) The updated parity is transferred from the system controller 20 to the device controller 60 of the parity FMPK.

  As described above, data transfer in the first read-modify-write process is less than when the system controller 20 performs the parity calculation.

  Further, the data FMPK holds both the data before the update and the data after the update in the FM 55, so that the buffer 66 used by the device controller 60 of the data FMPK to calculate the intermediate parity can be reduced.

  Furthermore, by holding the old data until the parity is updated even after writing the new data, data recovery is possible even if a failure occurs before the parity update. In addition, after the parity update, the old data is invalidated and the old data is targeted for erasure, so that the amount of storage area used by the old data can be reduced.

  Next, a second read-modify-write process, which is another specific example of another data update process 1404 when the device controller 60 is selected as the parity calculation execution apparatus and the read-modify-write is selected as the write method, will be described. .

  FIG. 22 shows an example of the second read-modify-write process. FIG. 23 shows an example of the operation of the system controller 20 in the second read-modify-write process. The initial state in this example is the same as the initial state of the first read modify write process. First, the system controller 20 obtains old data from FMPK # 0 by issuing a read command to FMPK # 0 (data FMPK) corresponding to D0 (2301). Upon receiving this command, the device controller 60 of FMPK # 0 reads out the old data in FMPK # 0 and sends the old data to the system controller 20 as a response.

  Next, the system controller 20 issues a parity update write command to FMPK # 3 (parity FMPK) corresponding to P, thereby transferring new data and old data to FMPK # 3 (2302). Upon receiving this command, the FMPK # 3 device controller 60 calculates a new parity from the new data from the system controller 20, the old data from the system controller 20, and the old parity in FMPK # 3. Write to # 3.

  Next, the system controller 20 issues a write command to FMPK # 0 to transfer new data to FMPK # 0 (2303). Upon receiving this command, the device controller 60 of FMPK # 0 writes new data into FMPK # 0.

  According to this processing, the data transfer between the system controller 20 and the device controller 60 in updating the data of one stripe block is as follows.

(1) Data before update is transferred from the device controller 60 of data FMPK to the system controller 20, and data after update is transferred from the system controller 20 to the device controller 60 of data FMPK.
(2) Data before update and data after update are transferred from the system controller 20 to the device controller 60 of the parity FMPK.

  As a result, data transfer between the system controller 20 and the device controller 60 can be suppressed.

  In 2302, the system controller 20 transfers new data and old data to the parity FMPK by one parity update write command. The transfer data amount at this time is twice as large as the transfer data amount when the first read-modify-write process transfers the intermediate parity from the system controller 20 to the parity FMPK. However, since the number of transfers is one, an increase in data transfer overhead can be prevented. When data transfer is performed by a high-speed interface such as, for example, SAS (Serial Attached Small Computer System Interface) -6 GB, the merit of reducing the number of transfers is greater than the demerit of increasing the amount of transfer data.

  Furthermore, by retaining the old data until the parity is updated even after writing the new data, data recovery is possible even if a failure occurs before the parity update. Further, after the parity update, the old data is invalidated and the old data is targeted for erasure, so that the amount of storage area used by the old data can be reduced.

  Note that when the device controller 60 is selected as the parity operation execution device and the read modify write is selected as the write method, either the first read modify write process or the second read modify write process may be performed.

  Next, a third read modify write process, which is a specific example of the data update process 1404 when the system controller 20 is selected as the parity calculation execution apparatus and the read modify write is selected as the write method, will be described.

FIG. 24 shows an example of the third read modify write process. FIG. 25 shows an example of the operation of the system controller 20 in the third read-modify-write process. The initial state in this example is the same as the initial state of the first read modify write process. First, the system controller 20 obtains old data from FMPK # 0 by issuing a read command requesting D0 to FMPK # 0 corresponding to D0, and issues a read command to FMPK # 3 corresponding to P. The old parity is acquired from FMPK # 3 (2501). Receiving this read command, FMPK # 0 reads the old data in FMPK # 0 and sends the old data to the system controller 20 as a response. Upon receiving this read command, the device controller 60 of FMPK # 3 reads out the old parity in FMPK # 3 and sends the old parity to the system controller 20 as a response.

  Next, the system controller 20 calculates a new parity from the old data, the old parity, and the new data (2502). Next, the system controller 20 transfers new data to FMPK # 0 by issuing a write command to FMPK # 0, and issues a new parity to FMPK # 3 by issuing a write command to FMPK # 3 corresponding to P. Transfer (2503), and the flow of this process is terminated. Upon receiving this write command, the device controller 60 of FMPK # 0 writes new data to FMPK # 0. In response to this write command, FMPK # 3 writes a new parity to FMPK # 3.

  According to this processing, the data transfer between the system controller 20 and the device controller 60 in updating the data of one stripe block is as follows.

(1) Data before update is transferred from the device controller 60 of data FMPK to the system controller 20, and data after update is transferred from the system controller 20 to the device controller 60 of data FMPK.
(2) The parity before update is transferred from the device controller 60 of the parity FMPK to the system controller 20.
(3) The updated parity is transferred from the system controller 20 to the device controller 60 of the parity FMPK.

  As a result, data transfer between the system controller 20 and the device controller 60 can be suppressed.

  Next, some specific examples of the full stripe light will be described.

  First, the first full stripe write process, which is a specific example of another data update process 1404 when the device controller 60 is selected as the parity calculation execution apparatus and the full stripe write is selected as the write method, will be described.

  FIG. 26 shows an example of the first full stripe write process. FIG. 27 shows an example of the operation of the system controller 20 in the first full stripe write process. In this example, FMPK # 0, # 1, # 2, and # 3 store D0, D1, D2, and P, respectively. Here, the state in which the system controller 20 has received new data for updating the old data of D0, D1, and D2 in accordance with a write command is set as an initial state. In the following description, among FMPKs # 0, # 1, # 2, and # 3, FMPKs # 0, # 1, and # 2 storing updated data may be referred to as data FMPK.

  First, the system controller 20 divides the received data into new data of D0, D1, and D2 by issuing a write command to FMPK # 0, # 1, and # 2 (data FMPK) corresponding to D0, D1, and D2, respectively. Then, the new data of D0, D1, and D2 are transferred to FMPK # 0, # 1, and # 2, respectively (2701). Upon receiving this write command, the device controllers 60 of FMPK # 0, # 1, and # 2 write new data from the system controller 20 to FMPK # 0, # 1, and # 2, respectively.

  Next, the system controller 20 issues a parity generation write command instructing the generation and writing of parity to FMPK # 3 (parity FMPK) corresponding to P, whereby the new data of D0, D1, and D2 is sent to FMPK # 3. Transfer (2702), the flow of this processing is terminated. Upon receiving this write command, the device controller 60 of FMPK # 3 calculates a new parity from the new data of D0, D1, and D2, and writes the new parity to FMPK # 3.

  According to this processing, data transfer between the system controller 20 and the device controller 60 in updating data of one stripe line is as follows.

(1) The updated data is transferred from the system controller 20 to the device controller 60 of the data FMPK.
(2) The updated parity data is transferred from the system controller 20 to the device controller 60 of the parity FMPK.

  As a result, data transfer between the system controller 20 and the device controller 60 can be suppressed.

  Next, a second full stripe write process, which is a specific example of another data update process 1404 when the system controller 20 is selected as the parity calculation execution apparatus and the full stripe write is selected as the write method, will be described.

  FIG. 28 shows an example of the second full stripe write process. FIG. 29 shows an example of the operation of the system controller 20 in the second full stripe write process. The initial state in this example is the same as the initial state of the first full stripe write process. First, the system controller 20 calculates a new parity from the new data of D0, D1, and D2 (2901). Next, the system controller 20 issues new commands of D0, D1, and D2 to FMPK # 0, # 1, and # 2 by issuing a write command to FMPK # 0, # 1, and # 2 corresponding to D0, D1, and D2, respectively. Transfer to # 2 and issue a write command to FMPK # 3 corresponding to P, thereby transferring the calculated new parity to FMPK # 3 (2902) and end this processing flow. Upon receiving the write command, the device controllers 60 of FMPK # 0, # 1, and # 2 write new data from the system controller 20 to FMPK # 0, # 1, and # 2, respectively. In response to the write command, the device controller 60 of FMPK # 3 writes the new parity from the system controller 20 to FMPK # 3.

  According to this processing, data transfer between the system controller 20 and the device controller 60 in updating data of one stripe line is as follows.

(1) The updated data is transferred from the system controller 20 to the device controller 60 of the data FMPK.
(2) The updated parity is transferred from the system controller 20 to the device controller 60 of the parity FMPK.

  As a result, data transfer between the system controller 20 and the device controller 60 can be suppressed.

  Next, some specific examples of the data restoration process according to the selection result of the parity operation execution device will be described.

  First, the first data recovery process, which is a specific example of the data recovery process when the device controller 60 is selected as the parity calculation execution apparatus, will be described.

  FIG. 30 shows an example of the first data recovery process. FIG. 31 shows an example of the operation of the system controller 20 in the first data recovery process. In this example, FMPK # 0, # 1, # 2, and # 3 store D0, D1, D2, and P, respectively. Here, a failure occurs in the FM 55 in the FMPK # 1, and the state in which the FM 55 is replaced with a new FM 55 is defined as an initial state. In the following description, among FMPKs # 0, # 1, # 2, and # 3, FMPK # 1 storing data to be recovered may be referred to as a recovering FMPK.

  First, the system controller 20 issues a read command to the surviving FMPKs # 0, # 2, and # 3 in the RG to which the FMPKs # 0, # 1, # 2, and # 3 belong, whereby the FMPK # 0, D0, D2, and P stored in # 2 and # 3, respectively, are acquired (3101). Upon receiving this read command, the device controllers 60 of FMPK # 0, # 2, and # 3 read D0, D2, and P, respectively, and transfer the read D0, D2, and P to the system controller 20. Data read here or parity may be used.

  Next, the system controller 20 transfers the read D0, D2, and P to FMPK # 1 by issuing a write command indicating parity generation to the FMPK # 1 being restored (FMPK being restored) (3102). Then, the process flow ends. Upon receiving this write command, the device controller 60 of FMPK # 1 generates D1 recovery data by calculating D1 from D0, D2, and P, and writes the recovery data of D1 into FMPK # 1.

  According to this processing, data transfer between the system controller 20 and the device controller 60 in the restoration of data of one stripe block is as follows.

(1) Data of the FMPK 50 other than the FMPK being restored is transferred from the corresponding device controller 60 to the system controller 20.
(2) The transferred data is transferred from the system controller 20 to the device controller 60 of the FMPK being restored.

  As a result, data transfer between the system controller 20 and the device controller 60 can be suppressed.

  Next, the second data recovery process, which is a specific example of the data recovery process when the system controller 20 is selected as the parity calculation execution device, will be described.

  FIG. 32 shows an example of the second data recovery process. FIG. 33 shows an example of the operation of the system controller 20 in the second data recovery process. The initial state in this example is the same as in the first specific example. First, the system controller 20 issues a read command to the surviving FMPKs # 0, # 2, and # 3 in the RG to which the FMPKs # 0, # 1, # 2, and # 3 belong, whereby the FMPK # 0, D0, D2, and P are read from # 2 and # 3 (3301). Upon receiving this read command, the device controllers 60 of FMPK # 0, # 2, and # 3 read D0, D2, and P, respectively, and transfer the read D0, D2, and P to the system controller 20. Data read here or parity may be used.

  Next, the system controller 20 generates D1 recovery data by calculating D1 from D0, D2, and P (3302). Next, the system controller 20 issues a write command to FMPK # 1, writes the restored D1 into FMPK # 1 (3303), and ends this processing flow. Upon receiving this write command, the device controller 60 of FMPK # 1 writes the received D1 to FMPK # 1.

  According to this processing, data transfer between the system controller 20 and the device controller 60 in the restoration of data of one stripe block is as follows.

(1) Data of the FMPK 50 other than the FMPK being restored is transferred from the corresponding device controller 60 to the system controller 20.
(2) The transferred data is transferred from the system controller 20 to the device controller 60 of the FMPK being restored.

  As a result, data transfer between the system controller 20 and the device controller 60 can be suppressed.

  Next, the operation of the device controller 60 will be described.

  First, normal write processing, which is a specific example of processing performed by the device controller 60 when a normal write command is received, will be described.

  In this normal write process, the device controller 60 does not need to hold old data.

  FIG. 34 shows an example of normal write processing. First, the device controller 60 receives a normal write command (3401). Next, the device controller 60 obtains a logical page number from the logical address designated by this write command, and sets the logical page of this logical page number as the write destination logical page (3402). Next, the device controller 60 newly acquires a free page and allocates the acquired physical page as a write destination physical page (3403). Next, the device controller 60 writes the write data received by the write command to the write destination physical page (3404).

  Next, the device controller 60 associates the write destination physical page with the write destination logical page in the page mapping management table 1100, and registers the number of the write destination physical page as the physical page number (3405). Next, the device controller 60 sends a response indicating the completion of the command to the system controller 20 (3406), and ends this processing flow.

  In 3403, the device controller 60 acquires a free physical page by performing processing such as securing an unused physical page, erasing a physical block, and securing a physical page in the physical block, and the like. Also good. At this time, the device controller 60 may cancel the association between another logical page and the old physical page in the page mapping management table 1100 in order to erase the physical block.

  Next, an old data holding write command process, which is a specific example of processing of the device controller 60 when an old data holding write command is received, will be described.

  This old data holding write process is used, for example, for the first read-modify-write process described above.

  FIG. 35 shows an example of the old data holding write process. First, the device controller 60 receives an old data holding write command (3501). Next, the device controller 60 obtains a logical page number from the logical address designated by this write command, and sets the logical page of this logical page number as the write destination logical page (3502). Next, the device controller 60 refers to the page mapping management table 1100 to determine whether a physical page is assigned to the write destination logical page (3503).

  If a physical page is assigned to the write destination logical page (3503, Yes), the device controller 60 registers the physical page number associated with the write destination logical page in the page mapping management table 1100 as the old physical page number. (3504).

  When a physical page is not allocated to the write destination logical page (3503, No) or after 3504, the device controller 60 newly acquires a free page and allocates the acquired physical page as a write destination physical page (3505). ). Next, the device controller 60 writes the write data received by the write command to the write destination physical page (3506).

  Next, the device controller 60 associates the write destination physical page with the write destination logical page in the page mapping management table 1100, and registers the number of the write destination physical page as the physical page number (3507). Next, the device controller 60 sends a response indicating completion of the command to the system controller 20 (3508), and ends this processing flow.

  According to this process, the data FMPK can hold the data before update when the data is updated. Furthermore, the device controller 60 of the data FMPK can associate the updated logical page, the old physical page that stores the old data, and the physical page that stores the new data. Thereby, the data FMPK can hold both the data before update and the data after update in the nonvolatile memory at the time of data update. Further, the device controller 60 of the data FMPK can read both the data before update and the data after update without using a volatile memory. Furthermore, the system controller 20 can cause the device controller 60 of the data FMPK to hold both the data before update and the data after update.

  Next, intermediate parity read processing, which is a specific example of processing performed by the device controller 60 when an intermediate parity read command is received, will be described.

  This intermediate parity read process generates and returns an intermediate parity by performing a parity operation on old data and new data at a specified address. This intermediate parity read process is used, for example, in the first read-modify-write process described above.

  FIG. 36 shows an example of intermediate parity read processing. First, the device controller 60 receives an intermediate parity read command (3601). Next, the device controller 60 obtains a logical page number from the logical address designated by this intermediate parity read command, and sets the logical page of this logical page number as the read destination logical page (3602). Next, the device controller 60 refers to the page mapping management table 1100 to determine whether or not the old physical page corresponding to the read destination logical page is registered (3603).

  When the old physical page corresponding to the read destination logical page is registered (3603, Yes), the device controller 60 reads the data of the old physical page and the current physical page corresponding to the read destination logical page, An intermediate parity is calculated from the read data (3604). Next, the device controller 60 transfers the intermediate parity to the system controller 20 (3606), and ends this processing flow.

  In 3603, when the old physical page corresponding to the read destination logical page is not registered (3603, No), the device controller 60 reports to the system controller 20 a command result indicating that there is no old data of the read destination logical page. (3605), and the flow of this process is terminated.

  Upon receiving the result indicating that there is no old data of the read destination logical page in 3605, for example, the system controller 20 reads all the data in the same stripe line as the new data, calculates the new parity from the read data, A write command for writing a new parity to the device controller 60 corresponding to the parity may be issued. Alternatively, in this case, for example, the system controller 20 reads all data in the same stripe line as the new data, and writes a parity generation write command for calculating the new parity from the read data to the device controller 60 corresponding to the parity. May be issued.

  According to this process, when the device controller 60 of the data FMPK receives an instruction to calculate an intermediate parity from the system controller 20, the device controller 60 reads the data before update and the data after update, and reads the intermediate parity from the read data Can be generated. Further, the system controller 20 can cause the device controller 60 of the data FMPK to generate an intermediate parity and acquire the intermediate parity from the device controller 60.

  Suppose that the device controller 60 of the data FMPK generates an intermediate parity asynchronously with the operation of the system controller 20. In this case, the performance of the device controller 60 is deteriorated because the calculation of the intermediate parity places a burden on the buffer 66 of the device controller 60. In addition, when a power interruption occurs while data necessary for the intermediate parity calculation is stored in the buffer 66, this data may be lost and data recovery may be impossible. On the other hand, according to this embodiment, it is possible to prevent the performance degradation of the storage system 30 by generating the intermediate parity at the timing when the device controller 60 of the data FMPK receives the intermediate parity read command. Further, loss of data necessary for the intermediate parity calculation can be prevented and the reliability of the storage system 30 can be improved.

  Next, a parity update write process, which is a specific example of the process of the device controller 60 when a parity update write command is received, will be described.

  In the parity update write processing, a new parity is generated by performing a parity operation between the transferred intermediate parity and the designated old parity, and the new parity is written to the FM 55. This parity update write command is used, for example, in the first read-modify-write process described above.

  FIG. 37 shows an example of parity update write processing. First, the device controller 60 receives a parity update write command (3701). Next, the device controller 60 obtains a logical page number from the logical address designated by this write command, and sets the logical page of this logical page number as the read destination logical page (3702). Next, the device controller 60 obtains a physical page number corresponding to the read destination logical page by referring to the page mapping management table 1100, and reads data from the physical page indicated by the physical page number (3703). The read data is, for example, old parity. Next, the device controller 60 newly acquires a free page and allocates the acquired physical page as a result storage destination physical page (3704). Next, the device controller 60 performs a parity operation using the read data and the received data, and writes the operation result to the result storage destination physical page (3705).

  Next, the device controller 60 associates the result storage destination physical page with the write destination logical page in the page mapping management table 1100, and registers the number of the result storage destination physical page as the physical page number (3706). Next, the device controller 60 sends a response indicating the completion of the command to the system controller 20 (3707), and ends this processing flow.

  According to this processing, the parity FMPK device controller 60 calculates the updated parity from the updated data from the system controller 20 and the updated parity stored in the parity FMPK, and the updated parity. Can be transmitted to FM55. Furthermore, the system controller 20 can cause the parity FMPK device controller 60 to generate and store updated parity.

  Next, a parity generation write process, which is a specific example of the process of the device controller 60 when a parity generation write command is received, will be described.

  This parity generation write processing is a command for generating parity using data transferred for a plurality of stripe blocks and writing the parity to a specified address. This parity generation write processing is used, for example, in the first full stripe write processing and data recovery processing described above.

  FIG. 38 shows an example of processing of the device controller 60 by the parity generation write command. First, the device controller 60 receives a parity generation write command (3801). Next, the device controller 60 obtains the logical page number from the logical address designated by this write command, and sets the logical page of this logical page number as the write destination logical page (3802). Next, the device controller 60 newly acquires a free page and allocates the acquired physical page as a write destination physical page (3803). Next, the device controller 60 writes the write data received by the write command to the write destination physical page (3804).

  Next, the device controller 60 associates the write destination physical page with the write destination logical page in the page mapping management table 1100, and registers the number of the write destination physical page as the physical page number (3805). Next, the device controller 60 sends a response indicating the completion of the command to the system controller 20 (3806), and ends this processing flow.

  According to this processing, the device controller 60 of the parity FMPK can calculate the updated parity from the updated data from the system controller 20 and write the updated parity to the parity FMPK. Further, the system controller 20 can cause the parity FMPK device controller 60 to generate and store the updated parity.

  FIG. 39 shows an example of old data discard processing by the device controller 60. First, the device controller 60 receives an old data discard command from the system controller 20 (3901). Next, the device controller 60 obtains a logical page number from the logical address specified by this old data discard command, and deletes the old physical page number corresponding to this logical page number from the page mapping management table 1100 (3902). The device controller 60 does not invalidate the old data until the old data discard command is received. That is, even after writing new data, the device controller 60 can reliably hold the old data until the parity is updated (the old data discard command is received). In this way, by maintaining the old data until the parity is updated, data can be recovered even if a failure occurs before the parity update. In addition, after the parity update, the old data is invalidated to be erased, so that the amount of storage area used by the old data can be reduced.

  In this embodiment, a case where the system controller 20 performs RAID 6 control will be described.

  The configuration of the computer system in this embodiment is the same as the configuration of the computer system in the first embodiment. Therefore, differences from the first embodiment will be described below.

  The system controller 20 controls RAID6. First, the difference between RAID 5 and RAID 6 will be described.

  In order to extend the above-described RAID5 to RAID6, the parity calculation circuit 25 or the parity calculation circuit 65 further performs a parity calculation with a coefficient. Here, it is assumed that five FMPKs 50 store data D0, D1, D2, and parities P, Q, respectively. When A0, A1, and A2 are used as coefficients for generating Q, P and Q are generated by the following equations.

P = D0 + D1 + D2
Q = A0 · D0 + A1 · D1 + A2 · D2

  FIG. 40 shows an example of a Q parity generation method. The parity calculation execution apparatus stores A0, A1, and A2 in a memory in advance. Q is calculated from D0, D1, D2, A0, A1, and A2. An arithmetic unit indicated by a black diamond in the figure multiplies the coefficient attached thereto.

Further, by solving the P generation equation and the Q generation equation as simultaneous equations, it is possible to restore any data or parity at the time of data lost. For example, when D0 and P are restored, they can be generated by the following equation.

D0 = A1 / A0 · D1 + A2 / A0 · D2 + 1 / A0 · Q
P = D0 + D1 + D2

  For example, when D0 and D1 are restored, these can be generated by the following equation.

D0 = (A1 + A2) / (A0 + A1) · D2 + A1 / (A0 + A1) · P
+ 1 / (A0 + A1) · Q
D1 = (A0 + A2) / (A0 + A1) · D2 + A0 / (A0 + A1) · P
+ 1 / (A0 + A1) · Q

FIG. 41 shows an example of a recovery method for D0 and D1. As shown above, D0 and D1 can be expressed as D2, P, and Q linear forms αD2 + βP + γQ, respectively.
Here, the coefficients α, β, and γ are values based on A0, A1, and A2.
The parity calculation execution device may store α, β, and γ based on A0, A1, and A2 in a memory in advance. D0 is calculated from α, β, and γ, which are values based on D2, P, Q, and A0, A1, and A2.
This example is merely an example to show the concept, and in actuality, the design is made so as to increase the processing speed and reduce the memory usage.

  Next, a fourth read-modify-write process, which is a specific example of the data update process 1404 when the device controller 60 is selected as the parity calculation execution apparatus and the read-modify-write is selected as the write method, will be described.

  FIG. 42 shows an example of the fourth read-modify-write process. FIG. 42 shows an example of the operation of the system controller 20 in the fourth read modify write process. In this example, FMPK # 0, # 1, # 2, # 3, and # 4 store D0, D1, D2, P, and Q, respectively. Here, the state in which the system controller 20 receives new data for updating the old data of D0 by the write command is set as the initial state. In the following description, among FMPK # 0, # 1, # 2, # 3, and # 4, FMPK # 0 that stores D0 that is data to be updated may be referred to as data FMPK. Of FMPK # 0, # 1, # 2, # 3, and # 4, FMPK # 3 storing P may be referred to as P parity FMPK. Of FMPK # 0, # 1, # 2, # 3, and # 4, FMPK # 4 storing Q may be referred to as Q parity FMPK.

  First, the system controller 20 writes new data to FMPK # 0 by issuing an old data holding write command to FMPK # 0 (data FMPK) corresponding to D0 (4101). Upon receiving this command, the device controller 60 of FMPK # 0 writes the new data from the system controller 20 into a physical page different from the physical page of the old data in FMPK # 0.

  Next, the system controller 20 obtains the intermediate parity from FMPK # 0 by issuing an intermediate parity read command to FMPK # 0 (4102). Upon receiving this command, the device controller 60 of FMPK # 0 calculates an intermediate parity from the old data and the new data stored in FMPK # 0, and sends the intermediate parity to the system controller 20 as a response.

  Next, the system controller 20 issues a parity update write command similar to that in RAID 5 to FMPK # 3 (P parity FMPK) corresponding to P, thereby transmitting intermediate parity to FMPK # 3 (4103). Upon receiving this command, the device controller 60 of FMPK # 3 calculates a new parity from the old parity in FMPK # 3 and the intermediate parity from the system controller 20, and writes the new parity to FMPK # 3.

  The calculation of the new parity P_new when i is any one of 0, 1, and 2 and the data Di in the RG is updated to Di_new will be described. Similar to RAID 5, the device controller 60 that has received the parity update write command calculates P_new from the intermediate parity (Di + Di_new) and the old parity P by the following equation.

  P_new = P + (Di + Di_new)

  The device controller 60 of the data FMPK calculates (Di + Di_new) according to the intermediate parity read command. The device controller 60 of P parity FMPK calculates P + (Di + Di_new) according to the parity update write command.

  Next, the system controller 20 transfers the intermediate parity to FMPK # 4 (Q parity FMPK) by issuing a Q parity update write command, which is a parity update command for updating Q, to FMPK # 4 corresponding to Q. (4104), and the flow of this process is terminated. Upon receiving this command, the device controller 60 of FMPK # 4 calculates a new parity from the old parity in FMPK # 4, the intermediate parity and coefficient from the system controller 20, and writes the new parity to FMPK # 3.

  According to this processing, the data transfer between the system controller 20 and the device controller 60 in updating the data of one stripe block is as follows.

(1) The updated data is transferred from the system controller 20 to the device controller 60 of the data FMPK.
(2) The intermediate parity is transferred from the device controller 60 of the data FMPK to the system controller 20.
(3) The intermediate parity is transferred from the system controller 20 to the device controller 60 of P parity FMPK and the device controller 60 of Q parity FMPK.

  As a result, data transfer between the system controller 20 and the device controller 60 can be suppressed.

  The process of the device controller 60 that has received the Q parity update write command is different from the case of the parity update write command in that the coefficient Ai preset in the device controller 60 is used. When updating the data Di in the RG to Di_new, the device controller 60 that has received the Q parity update write command calculates a new parity Q_new from the intermediate parity (Di + Di_new), the old parity Q, and Ai by the following equation.

Q_new = Q + Ai · (Di + Di_new)

  The device controller 60 of the data FMPK calculates (Di + Di_new) according to the intermediate parity read command. The device controller 60 of the Q parity FMPK calculates Q + Ai · (Di + Di_new) according to the Q parity update write command.

  According to the parity update write command, the device controller 60 of the P parity FMPK calculates the updated P parity from the updated data from the system controller 20 and the pre-updated P parity stored in the P parity FMPK. Then, the updated P parity can be written to the P parity FMPK. Further, the system controller 20 can cause the device controller 60 of the P parity FMPK to generate and store the updated P parity.

  According to the Q parity update write command, the device controller 60 of the Q parity FMPK updates the updated data from the system controller 20, the pre-update Q parity stored in the Q parity FMPK, and a predetermined coefficient. Q parity can be calculated, and the updated Q parity can be written to the Q parity FMPK. Further, the system controller 20 can cause the device controller 60 of the Q parity FMPK to generate and store the updated Q parity.

  In this embodiment, a computer system, which is an application example of the storage system 30 when the system controller 20 does not have a parity operation function, will be described.

  FIG. 44 illustrates an example of a configuration of a computer system according to the third embodiment. In the computer system of this embodiment, the elements denoted by the same reference numerals as the elements of the storage system 30 indicate the same or equivalent elements as the elements of the storage system 30. The computer system of this embodiment has a host computer 41a, a host computer 41b, and a plurality of FMPKs 50 connected to the host computer 41b. The host computer 41a and the host computer 41b are connected via a LAN (Local Area Network) 2, for example.

  The host computer 41a has a host controller 42a and a plurality of FMPKs 50 connected to the host controller 42a. The host controller 42 a includes a NIC (Network Interface Card) 13 for connecting to a communication network such as LAN 2, a memory 12, a CPU 11, and a buffer 26.

  The host computer 41 b includes a NIC 13 for connecting to a communication network such as LAN 2, an HBA (Host Bus Adapter) 15 for connecting to the FMPK 50, a memory 12, a CPU 11, and a buffer 26.

  In the host controller 42a and the host computer 41b, the memory 12 stores a program for controlling the FMPK 50 and various types of information. The CPU 11 realizes various functions by executing programs based on information stored in the memory 12. Each of the host controller 42a and the host computer 41b may perform RAID control using a plurality of FMPKs 50 connected to itself.

  One of the host controller 42a and the host computer 41b may make an IO request to the other via the LAN 2, or may make an IO request to its own FMPK 50.

  In this embodiment, the system controller 20 may be in any form of the host controller 42a and the host computer 41b.

  Each of the host controller 42a and the host computer 41b corresponds to case # 3 or case # 4 in the determination method 1300 of the parity operation execution device.

  Next, a third computer system, which is an application example of the storage system 30 when the system controller 20 has a parity calculation function, will be described.

  FIG. 45 illustrates an example of a configuration of a computer system according to the fourth embodiment. In the computer system of this embodiment, the elements denoted by the same reference numerals as those of the computer system of the third embodiment are the same as or equivalent to the elements of the computer system of the third embodiment. The computer system of this embodiment includes a host computer 41c, a host computer 41d, and a plurality of FMPKs 50 connected to the host computer 41d. The host computer 41c and the host computer 41d are connected via a communication network, for example, LAN2.

  The host computer 41c has a host controller 42c and a plurality of FMPKs 50 connected to the host controller 42c. The host controller 42c has a parity calculation circuit 25 in addition to the elements of the host controller 42a.

  The host computer 41d has a parity calculation circuit 25 in addition to the elements of the host computer 41b.

  In the host controller 42c and the host computer 41d, the memory 12 stores a program for controlling the FMPK 50 and various types of information. The CPU 11 realizes various functions by executing programs based on information stored in the memory 12. Each of the host controller 42c and the host computer 41d may perform RAID control using a plurality of FMPKs 50 connected to itself.

  One of the host controller 42c and the host computer 41d may make an IO request to the other via the LAN 2, or may make an IO request to the device controller 60 of its own FMPK 50.

  Each of the host controller 42a and the host computer 41b corresponds to case # 1 or case # 2 in the determination method 1300 of the parity operation execution device.

  In this embodiment, the system controller 20 may be in any form of the host controller 42c and the host computer 41d.

  Note that the computer system may be a combination of any of the above-described embodiments. Further, a system controller 20 having a parity calculation function and a system controller 20 not having a parity calculation function may be mixed in one computer system.

  According to each embodiment described above, data transfer between the system controller 20 and the device controller 60 can be suppressed. Thereby, the speed of the storage system 30 can be improved. Data transfer is represented by, for example, the number of data transfers or the transfer amount.

  Further, according to each embodiment described above, data transfer between the system controller 20 and the device controller 60 can be suppressed, and the amount of volatile memory used in the device controller 60 can be reduced.

  In addition, the order of each process in the operation of the system controller 20 may be exchanged. For example, 1402 and 1403 are interchangeable. In addition, the order of each step in the operation of the device controller 60 may be exchanged. For example, 3703 and 3704 are interchangeable.

10: Host computer, 20: System controller, 25: Parity operation circuit, 26: Buffer, 30: Storage system, 41a, 41b, 41c, 41d: Host computer 42a, 42c: Host controller, 50: FMPK (flash memory package) 55: FM (flash memory), 60: device controller, 65: parity operation circuit, 66: buffer

Claims (18)

A plurality of storage devices each including a plurality of nonvolatile memory chips that are write-once memories and a device controller connected to the plurality of nonvolatile memory chips;
A system controller connected to the plurality of storage devices and configured to control the plurality of storage devices as a RAID group, the first storage device storing old data and the second storage device storing old parity When,
Have
The first device controller of the first storage device validates the old data stored in the physical storage area of the plurality of nonvolatile memory chips of the first storage device and new data for updating the old data A first intermediate parity is generated based on the old data and the new data,
The system controller transmits a parity update command and the first intermediate parity to instruct the second storage device to generate a first new parity;
The second device controller of the second storage device, based on the first intermediate parity and the old parity stored in the second storage device, in accordance with the parity update command, Produces
When the system controller receives a completion response to the parity update command from the second storage device, the system controller transmits an invalidation command for invalidating the old data to the first storage device,
The first device controller invalidates the old data upon receipt of the invalidation command.
A system characterized by that. The physical storage area is a physical page of the nonvolatile memory chip, and the first device controller has a page mapping management table that associates the physical page with a logical page,
The first device controller associates a first physical page storing the old data with a first logical page on the page mapping management table,
The first device controller is
When the new data is stored in the nonvolatile memory chip of the first storage device, the first physical data is stored on the page mapping management table in order to maintain the old data and the new data in a valid state. While maintaining the association of the page with the first logical page, the second physical page stored with the new data and different from the first physical page is associated with the first logical page;
Deleting the information of the first physical page from the page mapping management table in order to invalidate the old data;
The system according to claim 1, wherein: The first device controller is
Deleting the old data after deleting the information of the first physical page from the page mapping management table;
The system according to claim 2, wherein: If the system controller does not receive the completion response from the second storage device within a predetermined time, the system
Generating a second intermediate parity by the first device controller based on the old data and the new data;
Transferring the second intermediate parity from the first device controller to the second device controller;
Generating a second new parity by the second device controller based on the second intermediate parity and the old parity;
The system of claim 1, wherein: A plurality of storage devices each including a plurality of nonvolatile memory chips that are write-once memories and a device controller connected to the plurality of nonvolatile memory chips;
A system controller connected to the plurality of storage devices and configured to control the plurality of storage devices as a RAID group, the first storage device storing old data and the second storage device storing old parity When,
Have
The first device controller of the first storage device is
Storing new data received from the system controller while maintaining the old data stored in the physical storage areas of the plurality of nonvolatile memory chips of the first storage device in a valid state;
Generating a first intermediate parity to be transferred to the second storage device based on the new data received from the system controller and the old data stored in the physical storage area;
The old data transmitted from the system controller after recognizing that the first new parity has been generated based on the old parity and the first intermediate parity stored in the second storage device are invalidated. Invalidating the old data in response to receiving an invalidation command for
A system characterized by that. The second storage device transmits a completion response to the system controller after the generation of the first new parity,
The system controller transmits the invalidation command to the first storage device in response to receiving the completion response.
The system according to claim 5, wherein: The physical storage area is a physical page of the nonvolatile memory chip, and the first device controller has a page mapping management table that associates the physical page with a logical page,
The first device controller associates a first physical page storing the old data with a first logical page on the page mapping management table,
The first device controller is
When the new data is stored in the nonvolatile memory chip of the first storage device, the first physical data is stored on the page mapping management table in order to maintain the old data and the new data in a valid state. While maintaining the association of the page with the first logical page, the second physical page stored with the new data and different from the first physical page is associated with the first logical page;
Deleting the information of the first physical page from the page mapping management table in order to invalidate the old data;
The system according to claim 6, wherein: The first device controller is
Deleting the old data after deleting the information of the first physical page from the page mapping management table;
The system according to claim 7, wherein: If the system controller does not receive the completion response from the second storage device within a predetermined time, the system
Generating a second intermediate parity by the first device controller based on the old data and the new data;
Transferring the second intermediate parity from the first device controller to a second device controller of the second storage device;
Generating a second new parity by the second device controller based on the second intermediate parity and the old parity;
The system according to claim 6, wherein: A plurality of storage devices each including a plurality of nonvolatile memory chips that are write-once memories and a device controller connected to the plurality of nonvolatile memory chips;
A system controller connected to the plurality of storage devices and configured to control the plurality of storage devices as a RAID group, the first storage device storing old data and the second storage device storing old parity When,
Have
The first device controller of the first storage device validates the old data stored in the physical storage area of the plurality of nonvolatile memory chips of the first storage device and new data for updating the old data A first intermediate parity is generated based on the old data and the new data,
The second device controller of the second storage device is
Based on the first intermediate parity generated by the first device controller and the old parity stored in the second storage device, a first new parity is generated,
Sending a completion response from the second storage device to the system controller;
A system characterized by that. In response to receiving the completion response, the system controller transmits an invalidation command for invalidating the old data to the first storage device,
The first storage device invalidates the old data in response to receipt of the invalidation command;
The system according to claim 10, wherein: The physical storage area is a physical page of the nonvolatile memory chip, and the first device controller has a page mapping management table that associates the physical page with a logical page,
The first device controller associates a first physical page storing the old data with a first logical page on the page mapping management table,
The first device controller is
When the new data is stored in the nonvolatile memory chip of the first storage device, the first physical is stored on the page mapping management table in order to maintain the old data and the new data in a valid state. While maintaining the association of the page with the first logical page, the second physical page stored with the new data and different from the first physical page is associated with the first logical page;
Deleting the information of the first physical page from the page mapping management table in order to invalidate the old data;
The system according to claim 11, wherein: The first device controller is
Deleting the old data after deleting the information of the first physical page from the page mapping management table;
The system according to claim 12, wherein: If the system controller does not receive the completion response from the second storage device within a predetermined time, the system
Generating a second intermediate parity by the first device controller based on the old data and the new data;
Transferring the second intermediate parity from the first device controller to the second device controller;
Generating a second new parity by the second device controller based on the second intermediate parity and the old parity;
The system according to claim 11, wherein: A plurality of storage devices each including a plurality of nonvolatile memory chips that are write-once memories and a device controller connected to the plurality of nonvolatile memory chips;
A system controller connected to the plurality of storage devices and configured to control the plurality of storage devices as a RAID group, the first storage device storing old data and the second storage device storing old parity When,
Have
The system controller or the plurality of storage devices have a parity calculation unit that generates parity,
When the first storage device and the second storage device have a parity calculation unit,
The first device controller of the first storage device validates the old data stored in the physical storage area of the plurality of nonvolatile memory chips of the first storage device and new data for updating the old data A first intermediate parity is generated based on the old data and the new data,
The system controller transmits a parity update command and the first intermediate parity to instruct the second storage device to generate a first new parity;
The second device controller of the second storage device, based on the first intermediate parity and the old parity stored in the second storage device, in accordance with the parity update command, Produces
When the system controller receives a completion response to the parity update command from the second storage device, the system controller transmits an invalidation command for invalidating the old data to the first storage device,
In response to receiving the invalidation command, the first device controller invalidates the old data,
When the first device controller and the second device controller do not have the parity operation unit, and the system controller has the parity operation unit,
The system controller generates a third new parity;
A system characterized by that. The physical storage area is a physical page of the nonvolatile memory chip, and the first device controller has a page mapping management table that associates the physical page with a logical page,
The first device controller associates a first physical page storing the old data with a first logical page on the page mapping management table,
The first device controller is
When the new data is stored in the nonvolatile memory chip of the first storage device, the first physical data is stored on the page mapping management table in order to maintain the old data and the new data in a valid state. While maintaining the association of the page with the first logical page, the second physical page stored with the new data and different from the first physical page is associated with the first logical page;
Deleting the information of the first physical page from the page mapping management table in order to invalidate the old data;
The system according to claim 15, wherein: The first device controller is
Deleting the old data after deleting the information of the first physical page from the page mapping management table;
The system of claim 16, wherein: If the system controller does not receive the completion response from the second storage device within a predetermined time, the system
The first device controller generates a second intermediate parity from the old data and the new data,
Transferring the second intermediate parity from the first device controller to the second device controller;
A second new parity is generated from the second intermediate parity and the old parity by the second device controller;
The system according to claim 15, wherein:

Images