JP4968078B2 - Failure diagnosis apparatus and failure diagnosis method - Google Patents

Description

  The present invention relates to a failure diagnosis device and a failure diagnosis method suitable for application to a data recording / reproducing device using a large number of hard disk drives such as a RAID (Redundant Array of Inexpensive Disks) device.

  A RAID device, which is a device for recording AV data (video data and audio data) on a plurality of hard disk drives using RAID technology, is widely used as a data recording device for business use and professional use. Such a RAID apparatus normally performs writing / reading of AV data based on a command transmitted from a host control apparatus.

  A hard disk drive is a device that uses a head to write and read data on the disk, and writing and reading can be unstable due to various factors such as writing and reading by the head and seek state of the head. This is a data recording apparatus that has a possibility of causing some errors in recording and reproduction.

  As described above, by configuring as a RAID device using a plurality of hard disk drives, data is distributed and recorded on a plurality of hard disks, and even if there is an error in one hard disk drive. If data can be read correctly from other hard disk drives, the missing data can be compensated, and the reliability of the data recording apparatus can be ensured. In addition, as an error, in addition to the case where the data is not read and the corresponding data is lost, for example, the time from when the read command is sent until the corresponding data is read from the disk is slow. There is also an error such as

  By the way, a hard disk drive is a device with a long life and needs to be replaced with a new one after it has been used to some extent. For example, the reliability of data recording is very important for the above-mentioned data recording device for business use or professional use, and maintenance is required to replace the hard disk drive every time it is used as a data recording device to some extent.

  Considering such replacement of the hard disk drive, it is easy in terms of management to replace all at once every certain year, for example, every year. However, hard disk drive malfunctions vary from one hard disk drive to another, and some hard disk drives can be used with little trouble even after a long period of time, while others may generate many errors in a relatively early period of use. It is preferable to perform management by checking the error occurrence status for each unit.

As a conventional technique for diagnosing a failure of this type of hard disk drive, for example, the frequency of error occurrence when accessing a hard disk drive is determined, and when the determined frequency exceeds a preset threshold, the corresponding hard disk drive It was decided that the drive was defective and was replaced.
Japanese Patent Application Laid-Open No. 2004-228561 describes a point of diagnosing whether or not there is a high possibility of a hard disk drive failure based on such a determination.

JP-A-10-11701

By the way, there are cases where it is not possible to accurately grasp the failure status of each hard disk drive simply by looking at the frequency of error occurrence when accessing the hard disk drive.
That is, for example, as shown in FIG. 8A, when the cumulative number of accesses is on the horizontal axis and the error occurrence frequency for each fixed access (for example, 100 accesses) is on the vertical axis, the error occurrence frequency ER0 is the threshold for failure determination. When TH0 is exceeded, it is determined that the corresponding hard disk drive needs to be replaced. The error occurrence frequency ER0 is generally assumed to gradually increase with the use of a hard disk drive.

  However, as an error occurrence state of the hard disk drive in practice, for example, as shown in the example of FIG. 8A, there is a case where the error occurrence frequency ER0 hardly changes even if it is used for a long time. In the example of FIG. 8A, when the integrated value of the number of error occurrences is plotted on the vertical axis and the horizontal axis is viewed as the number of accesses as shown in FIG. 8B, the cumulative error occurrence number ET0 changes. Is an almost linear change.

  For example, even if the error occurrence frequency ER0 shown in FIG. 8 (a) is constant, there is no problem as long as the error occurrence frequency ER0 is constant at a very low frequency, but the frequency ER0 is constant to a degree slightly lower than the failure determination threshold TH0. In such a case, the hard disk drive is continuously used in a situation where errors occur to some extent, which is not preferable.

  The present invention has been made in view of the above-described points, and an object of the present invention is to make it possible to more accurately diagnose a failure of a hard disk drive.

In order to solve this problem, according to the present invention, when performing a failure diagnosis for diagnosing a failure of a hard disk drive, the processing based on the write and / or read command of the hard disk drive is performed within the time specified from the issue of the command The error that could not be completed normally is determined, and the number of times the error is determined in the command error determination is accumulated. When the frequency at which an error is determined exceeds a threshold, it is determined that the hard disk drive is defective, and the threshold is variably set according to the cumulative value of the number of errors.

  According to the present invention, since the threshold value for comparing the frequency at which an error is determined is variably set, the error occurrence frequency varies so much by appropriately setting the threshold value based on the cumulative value of the number of errors. Even if there is no error, it can be judged as defective.

  According to the present invention, by appropriately setting the threshold value based on the cumulative value of the number of errors, even if there is not much change in the frequency of error occurrence, it can be determined as defective, and the error is appropriately Defect management based on the number of occurrences becomes possible. For example, hard disk drives that are stable in situations where there are relatively many errors can be detected as defective products in a relatively short period of use, which improves the reliability of data recording devices that use hard disk drives. To contribute.

  Embodiments of the present invention will be specifically described below with reference to the drawings. In this embodiment, an example in which the present invention is applied to a digital cinema screening system will be described. As an assumption, an outline of the digital cinema screening system will be described first.

  FIG. 1 is an overall configuration diagram of a digital cinema screening system to which the present invention is applied. This digital cinema screening system includes a data recording device (so-called RAID device) 1 using a RAID technology, a digital cinema server 2, a projector (projection display device) 3, and a personal computer 4.

  The data recording device 1, the digital cinema server 2, and the projector 3 are housed in the same casing. The digital cinema server 2 and the personal computer 4 are connected by a high-speed network 5 such as 1000BASE-T.

  This digital cinema screening system records AV data of a movie to be screened (video data is compressed by an image compression method such as JPEG2000) from a personal computer 4 to a data recording device 1 via a digital cinema server 2. The AV data reproduced from the data recording device 1 is sent from the digital cinema server 2 to the projector 3 and projected onto the screen.

  In the personal computer 4, on the GUI screen, recording of AV data, movie screening, data other than AV data (log information of operation of the data recording device 1, file system relating to AV data in the data recording device 1) And a program for managing movie subtitle data). Based on the operation on the GUI screen, the personal computer 4 sends AV data write / read requests, log information, file system information, and subtitle data write / read requests to the digital cinema server 2 via the high-speed network 5. Send.

  FIG. 2 is a diagram showing main circuits provided in the digital cinema server 2. In the digital cinema server 2, a CPU 11 and a decoder 20 that decompresses compressed AV data are provided. The CPU 11 is connected to the high-speed network 5 shown in FIG. 1 and is connected to the data recording apparatus 1 shown in FIG. 1 via a PCI-X standard bus. The decoder 20 is connected to the data recording device 1 and to the projector 3 in FIG.

The CPU 11 operates a general-purpose OS (for example, Linux), and manages the hard disk in the data recording apparatus 1 by a general-purpose file system (for example, XFS) on the general-purpose OS. Further, the CPU 11 stores the following three programs (a) to (c) that operate on the general-purpose OS.
(A) “Internal control application” which is a program for performing internal control of the data recording apparatus 1
(B) “Digital cinema application” which is a program for transmitting AV data write / read commands to the “internal control application” in accordance with AV data write / read requests from the personal computer 4
(C) A program that transmits log information, file system information, and subtitle data write / read commands to the “internal control application” in accordance with log information, file system information, and subtitle data write / read requests from the personal computer 4. A management application

  The CPU 11 functions as a higher-level control device (device that transmits a data write / read command to the data recording device 1) by executing the “digital cinema application” or the “management application”. In addition, it also has a function of performing internal control of the recording apparatus 1 by executing an “internal control application”.

  Therefore, in the following, in order to make the explanation easy to understand, a portion of the CPU 11 that functions as a higher-level control device is referred to as a higher-level control unit 11-1, and a function portion that performs internal control of the data recording device 1 is referred to as an internal control unit. The description will be made by dividing the CPU 11 into two parts according to functions, such as 11-2.

  FIG. 3 is a diagram showing the configuration of the data recording apparatus 1 together with the upper control unit 11-1, the internal control unit 11-2, and the decoder 20. Here, the internal control unit 11-2 is shown as a part of the data recording apparatus 1 because of its function. In addition, the flow of data (AV data, log information, file system information, and caption data) is indicated by double arrows, and commands and responses sent and received by the upper control unit 11-1 and the internal control unit 11-2 are shown. The flow is indicated by a single line arrow.

  The data recording apparatus 1 includes an ECC / DMA unit 12 composed of an FPGA, a cache memory 13, and an HDD controller (for example, SAS controller) 14 (14-1 to 14-) for controlling a total of seven HDDs (hard disks). 7). Out of a total of seven HDDs 15 (15-1 to 15-7) connected to the HDD controller 14, four HDDs 15-1 to 15-4 are for data, and two HDDs 15-5 to 15-6 are errors. For correction, the remaining one HDD 15-7 is for spare. Each of the HDDs 15-1 to 15-7 can be individually replaced at the time of maintenance of the data recording apparatus.

  At the time of recording, the ECC / DMA unit 12 generates an error correction code from the data sent from the upper control unit 11-1 in the digital cinema server 2, and uses the cache memory 13 for the data to which the error correction code is added. Striped and supplied to the HDDs 15-1 to 15-6 via the HDD controllers 14-1 to 14-6.

  During reproduction, the ECC / DMA unit 12 uses the cache memory 13 to destrip data supplied from the HDDs 15-1 to 15-6 via the HDD controllers 14-1 to 14-6, and the destriped data. Correct the error and restore the original data.

  When reproducing the video data, the video data restored by the ECC / DMA unit 12 is sent to the decoder 20. Then, the baseband video data expanded by the decoder 20 is sent to the projector 3 in FIG. 1 and projected from the projector 3 onto the screen.

  At the time of reproduction of audio data, the audio data restored by the ECC / DMA unit 12 is transmitted from an audio output terminal (not shown) provided in a housing housing the data recording device 1, the digital cinema server 2, and the projector 3. Output to the outside.

  At the time of reproduction of log information and file system information, the log information and file system information restored by the ECC / DMA unit 12 is sent to the upper control unit 11-1, and the upper control unit 11-1 performs the personal computer 4 shown in FIG. And displayed on the GUI screen on the display of the personal computer 4.

When reproducing subtitle data, the subtitle data restored by the ECC / DMA unit 12 is sent to a mixing circuit (not shown) provided in the digital cinema server 2 and decompressed by the decoder 20. Synthesized with data.
The internal control unit 11-2 executes the “internal control application” described above, and controls the ECC / DMA unit 12 and the HDD controller 14.

Next, in the configuration of the data recording apparatus 1 shown in FIG. 3, the outline of the configuration related to writing and reading of data to and from the hard disk is shown.
For data to be recorded supplied from the outside via a high-speed network 5 such as FC (Fibre Channel), an error correction code is generated by the ECC / DMA unit 12, and the data with the error correction code added is used by the cache memory 13. Striped and supplied to the HDDs 15-1 to 15-6 via the HDD controllers 14-1 to 14-6. Write and read commands are also supplied via the high-speed network 5.

  During playback, the data supplied from the HDDs 15-1 to 15-6 via the HDD controllers 14-1 to 14-6 is destriped using the cache memory 13, and the destriped data is error-corrected. Restore the original data.

  These recording and reproduction processes are executed under the control of the internal control unit 11-2, and data related to the control state is stored in the SDRAM 11-3, which is a memory included in the internal control unit 11-2. An example of data related to the stored control state will be described later. Data relating to the control state may be stored in a storage means other than the SDRAM. For example, data may be stored using a non-volatile memory. Or you may make it memorize | store in a part of hard disk.

  Next, a process structure, which is a processing structure for writing and reading data performed when a command is supplied to the data recording apparatus 1 of the present embodiment, will be described with reference to FIG. The process structure shown in FIG. 5 shows the process structure when executed by the control of the internal control unit 11-2 shown in FIGS. 3 and 4, and the internal control unit 11-2 has a control function. Is provided.

When a command is supplied from the digital cinema server to the FC driver, which is a high-speed network driver, command standby processing is performed, and the processing content designated by the command is transmitted to the command processing.
The command processing is the part that actually processes the processing contents of the command, and performs table search, resource reservation, etc., and communicates with the high-speed network side.
Also, a rebuild command is issued according to the rebuild setting, and the rebuild is executed according to the setting and the situation.
For example, when a situation in which data cannot be written to the HDD occurs, the LBA manager is notified and performs management.
In the mode page manager, HDD errors are totaled to determine whether each HDD is defective. A defect determination process to be described later is executed here. Specifically, the error occurrence frequency value and the integrated value are tabulated. The frequency value is the number of errors every time a certain number of commands are issued to each HDD, and the integrated value is the number of errors accumulated since the start of use of the corresponding HDD. In addition, the failure determination is made by comparing the current frequency value with a set threshold value and determining that the current frequency value exceeds the threshold value. However, as will be described later, in the case of this example, this threshold value is variably set. Further, when it is determined to be defective, processing for transmitting the corresponding defective HDD to the administrator is executed. For example, a process for notifying the HDD determined to be defective by display or the like is performed.

The resource manager allocates command tables and caches.
The HDD manager monitors the command table, instructs a free HDD controller, and manages the HDD status.
The HDD controller issues a command to the HDD execution unit. Time-out monitoring is also performed to monitor whether the processing based on the command is performed within a specified time. When the process ends normally, it returns an end and waits for the next command. When a time-out occurs in which the process does not end within a specified time, a time-out is returned, the HDD status is returned to the RADIUS controller, and the end of the time-out driver is awaited. If the timeout further occurs, the reset process is started, the status is returned at the end time, and the next wait is entered. In case of further failure, the status is returned as defective and the next wait is entered.
The execution unit executes processing for calling the HDD driver.
The status daemon monitors the status of the HDD. Here, the state of the physical layer is mainly monitored. It also reflects the attachment / detachment of HDDs and the disconnection due to physical layer errors.

Next, HDD failure diagnosis processing according to the present embodiment will be described with reference to the flowchart of FIG.
This failure diagnosis process is performed for each of a plurality of HDDs included in the data recording apparatus 1.
First, when a data write / read command is supplied to the HDD, it is determined whether or not the command has been processed normally. If the command is not processed normally, an error is determined. The state in which processing is not normally performed includes not only the case where data cannot be read or written but also the time-out time when processing is not performed within a specified time.
When an error occurs (step S11), the error occurrence frequency value and integrated value are counted up (step S12). Here, the frequency value is the number of occurrences of errors per 100 accesses. Then, it is determined whether or not the integrated value of the number of error occurrences has reached a multiple of 10,000 (step S13).

When the integrated value of the number of error occurrences reaches a multiple of 10,000, the threshold setting value for comparing the frequency values is set to about half the current setting value (step S14). However, the lower limit value of the threshold is determined in advance. Here, the threshold value in the initial state is set to 10 and is changed from 10 → 5 → 2. This 2 is set as the lower limit value, and thereafter, the state in which the threshold value 2 is set is continued.
Then, in both cases where the integrated value has not reached a multiple of 10000 in step S13 and the threshold value has been changed in step S14, the currently set threshold value is compared with the integrated value of the number of error occurrences. Failure determination is performed (step S15). If the threshold is exceeded, it is determined that the corresponding HDD is out of order. If it is determined that there is a failure, processing for notifying the replacement of the HDD is performed.

  By processing in this way, for example, when one HDD is used, the threshold value initially compared with the error frequency value is 10, and if the number of errors per 100 accesses exceeds 10, it is defective. It is judged that there is. When the cumulative number of occurrences of errors reaches 10,000, the threshold value is changed to 5, and when the number of errors per 100 accesses exceeds 5, it is determined to be defective. Further, when the cumulative number of occurrences of errors reaches 20000, the threshold value is changed to 2, and when the number of errors per 100 accesses exceeds 2, it is determined to be defective. Thereafter, the threshold value remains 2 even if the cumulative number of occurrences of errors reaches 30000.

Note that this change example of the threshold value is an example, and other conditions may be set.
For example, the integrated value ÷ 1000 = n may be set, and the threshold value ÷ 2 ^ n may be set. (2 ^ n is 2 to the power of n)
Alternatively, a threshold setting table is provided, and every time the integrated value exceeds a certain value such as 10,000, a new threshold is set (or a condition value for determining a new threshold) is set by referring to the setting table. You may make it perform.

FIG. 7 shows an example of failure determination processing according to this embodiment.
FIG. 7A is a diagram in which the cumulative number of accesses is plotted on the horizontal axis, and the error frequency for every 100 accesses is plotted on the vertical axis. In this example, it is assumed that the error occurrence frequency ER1 is substantially constant.
In this situation, as shown in FIG. 7B, when the integrated value of the number of error occurrences is taken as the vertical axis and the horizontal axis is viewed as the number of accesses, the change in the cumulative error occurrence number ET1 is almost linear. It is a change.

  In this state, the failure determination threshold value sequentially changes from the initial threshold value TH1 to the threshold values TH2 and TH3 as the use proceeds. Here, when the error occurrence frequency ER1 is a reasonable frequency, even if there is no frequency change, the threshold value is exceeded and a failure is diagnosed. In the example of FIG. 7, it is determined to be defective when the lowest threshold value is reached, but when the frequency is higher, the threshold value is exceeded at an earlier time point, so it is determined to be defective earlier. Will be.

  Therefore, according to the present embodiment, even when the error occurrence frequency of the HDD does not change so much, it can be determined as defective, and defect management based on the number of error occurrences can be appropriately performed. For example, hard disk drives that are stable in situations where there are relatively many errors can be detected as defective products in a relatively short period of use, which improves the reliability of data recording devices that use hard disk drives. To contribute.

  As shown in FIG. 7A, the threshold value is changed stepwise, but the threshold value may be changed more gently (in a curved line).

  In the above embodiment, an example in which the present invention is applied to a digital cinema screening system has been described. However, the present invention is also applicable to hard disk drive failure diagnosis in other RAID devices. Furthermore, the present invention is applicable to failure diagnosis (failure prediction) of hard disk drives for recording various data other than RAID devices.

  In the above-described embodiment, the data recording device is configured to include each unit that performs failure diagnosis. However, for example, a general-purpose information processing device (such as a computer device) that includes a hard disk drive may include each process of the present invention. A program (software) for executing (the process shown in the flowchart of FIG. 6 and the like) may be installed and configured to perform similar failure diagnosis.

  A program recording medium for storing a program that can be executed by a computer device or the like in this case is provided as a package medium by a removable medium. As a removable medium, a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory), a DVD (Digital Versatile Disc), a magneto-optical disk), or a semiconductor memory is applied. Can do. Alternatively, the program recording medium may be configured by a ROM, a hard disk, or the like in which the program is stored (recorded) temporarily or permanently.

  Programs are stored in the program recording medium using a wired or wireless communication medium such as a local area network (LAN) or the Internet via a communication unit that is an interface such as a router or a modem as necessary. Done.

  In the present specification, the processing steps describing the program stored in the program recording medium are not limited to the processing performed in time series in the described order, but are not necessarily performed in time series. This includes processing that is executed manually or individually (for example, parallel processing or object processing).

  Further, the program may be processed by a single computer, or may be processed in a distributed manner by a plurality of computers. Furthermore, the program may be transferred to a remote computer and executed.

1 is an overall configuration diagram of a digital cinema screening system to which an embodiment of the present invention is applied. It is a block diagram which shows the circuit provided in the digital cinema server of the example of FIG. It is a block diagram which shows the structure of the data recording device of the example of FIG. FIG. 4 is a block diagram illustrating a configuration example of a main part of FIG. 3. It is explanatory drawing which shows the example of a process structure in the apparatus of one embodiment of this invention. It is the flowchart which showed the example of a failure diagnosis process by one embodiment of this invention. It is explanatory drawing which showed the example of failure determination by one embodiment of this invention. It is explanatory drawing which showed the conventional failure judgment example.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Data recording device, 2 ... Digital cinema server, 3 ... Projector, 4 ... Personal computer, 5 ... High speed network, 11 ... CPU, 11-1 ... High-order control part, 11-2 ... Internal control part, 12 ... ECC * DMA section, 13 ... cache memory, 14-1 to 14-7 ... HDD controller, 15-1 to 15-7 ... HDD, 20 ... decoder

Claims (6)

In a failure diagnosis device that diagnoses a failure of a hard disk drive,
A command error determination unit for determining an error in which processing based on a command of writing or reading of the hard disk drive has not been normally completed within a time specified from command issue ;
An error number accumulating unit for accumulating the number of times that the command error determining unit determines an error;
A failure determination unit that determines that the hard disk drive is defective when the frequency at which the command error determination unit determines an error exceeds a threshold, and that variably sets the threshold according to the number of times of accumulation in the error number accumulation unit And a fault diagnosis device. The failure diagnosis apparatus according to claim 1,
The error determined by the command error determination unit includes a case where the data read by the command is defective.
Fault diagnosis device. The failure diagnosis apparatus according to claim 2 ,
Wherein each time exceeds a predetermined number of times the number of errors have been determined in advance is accumulated at the error count accumulation portion, wherein the defect determining unit further comprising Ru failure diagnosis apparatus a threshold setting unit to reduce the threshold frequency is variably set in. The failure diagnosis apparatus according to claim 3 ,
The threshold setting unit, said each time exceeds a predetermined number of times, half reduced to fault diagnosis device threshold frequency. In a failure diagnosis method for diagnosing a failure of a hard disk drive,
A command error determination process for determining an error that the process based on the write or read command of the hard disk drive could not be completed normally within a time specified from the command issue ;
An error count accumulation process for accumulating the number of times the command error determination process has determined an error;
A failure determination process in which the frequency at which the command error determination process determines that an error exceeds a threshold value determines that the hard disk drive is defective, and the threshold value is variably set according to the cumulative number in the error count accumulation process A failure diagnosis method characterized by: The failure diagnosis method according to claim 5,
The error determined by the command error determination process includes a case where the data read by the command is defective.
Fault diagnosis method.

Images