JP7367400B2 - Information processing device and information processing program - Google Patents

Description

The present invention relates to an information processing device and an information processing program.

The storage device receives instructions from the connected server through the network and responds with the results. Furthermore, the storage device is capable of managing multiple network tasks, and starts a separate task for each server to process instructions from each server.

Then, when operating a network task, the storage device performs the following processing. For example, a storage device obtains a unique number called a File Descriptor (FD) number for each network task before communicating. After the storage device obtains the FD number, it becomes possible to process network tasks. Then, the storage device releases the FD number when the processing of the network task is completed. The storage device can acquire the released FD number for other network tasks. However, until the FD number is released, the storage device manages the FD number so that it does not overlap for each network task.

For example, the storage device obtains 1 as the FD number of network task A. After the processing of network task A is completed, the storage device releases FD number 1. Thereafter, the storage device acquires 1 as the FD number of another network task B. The time from acquisition to release of an FD number in processing such a network task varies depending on the load condition of the storage device.

Here, while the storage device is in operation, an event may occur in which network task B suffers a network error and communication is temporarily interrupted while both network task A and network task B are operating. One possible cause of this is that the storage device mistakenly releases the FD number acquired for network task B by double releasing the FD number of network task A. In this case, since no FD number is assigned to network task B when network task B is executed, the storage device determines that an abnormality has occurred and ends the process. As a result, communication is interrupted.

However, when double release processing is performed for network task A, if the specified FD number is not used by another network task, the storage device records an error log in order to guarantee the stability of system operation. Leave it to maintain normal condition. In this case, the storage device does not perform panic processing that involves forced shutdown by the OS (Operating System).

If a failure occurs due to such double release, a reproduction test is conducted to identify the cause. A fault investigation regarding the acquisition and release of resources such as FD numbers for processing network tasks is performed, for example, in the following procedure.

First, firmware is created to collect resource management data for investigation. This firmware adds management recording processing to resource acquisition and release processing. Hereinafter, this management recording process will be referred to as tracing. When performing tracing, the firmware secures a certain area in memory and stores information in the form of a table, taking processing performance into consideration. In this case, the firmware uses the table cyclically to reduce the size of the table.

In order to obtain instant data, a storage device that performs fault investigation executes panic processing when an abnormality occurs. The contents of the table created by tracing are stored in dump information that is saved when panic processing occurs. Thereafter, the worker investigating the failure analyzes the dump information generated by the panic processing and identifies the event that occurred when the failure occurred. If the location that is considered to be the cause of the failure is not found at this point, the above process is repeated until the location that is considered to be the cause of the failure is found.

As a technique for fault investigation, there is a conventional technique in which the state at the time of the problem is collected as a log by an application for ascertaining the cause, and the collected log is used to reproduce the state at the time of the problem. In addition, the trace information is retained as investigation data, the investigation data is applied to the test environment, and the state where the failure occurred is faithfully reproduced based on information such as processing timing, startup order, and processing delay. There is conventional technology.

Japanese Patent Application Publication No. 2018-185709 International Publication No. 2014/118897

However, with conventional fault investigation methods, when investigating double releases, there has been a problem in that the reproducibility of fault conditions is low for the following reasons. The reason is that double release does not always occur. Another possible reason is that even if double release occurs, the timing does not match with use by other network tasks, so no failure occurs. If the reproducibility of the failure condition is low and the desired event cannot be obtained, the trace collection range must be changed and trials repeated, which takes time to prepare for the investigation. Furthermore, if the firmware is re-created for each failure event in order to improve reproducibility, the work becomes complicated and the time required to investigate each failure becomes longer.

Here, in order to make it easier to investigate the cause, it may be possible to increase the amount of trace information to be saved, but in this case, the amount of stored traces will increase and there is a risk that the memory of the storage device will be overwhelmed. In order to prevent such an increase in the amount of trace storage, there is a method of cyclically using the memory area, but if a large amount of trace information is written, there is a risk that the events at the time of occurrence will be overwritten and may not remain. . Therefore, it is not preferable to increase trace information in order to identify the cause of the failure.

In addition, with conventional technology that uses logs to reproduce the state at the time of a problem, in order to improve reproducibility in the case of a failure that occurs infrequently, a large number of logs must be retained, resulting in problems such as memory pressure and Because of this, it is difficult to easily investigate failures. In addition, with conventional technology that applies survey data to a test environment and reproduces the state in which a failure occurred based on information such as processing timing, settings are adjusted for each event to reproduce the state, which requires work. This makes it difficult to easily investigate failures.

The disclosed technology has been made in view of the above, and aims to provide an information processing device and an information processing program that facilitate setting and adjustment of processing timing for reproducing a failure state.

In one aspect of the information processing apparatus and information processing program disclosed in the present application, the task execution unit executes processing of a plurality of network tasks. The acquisition unit acquires a processing time of a predetermined network task by the task execution unit. The basic value calculation unit calculates one or more of the maximum value, average value, and mode of the processing time acquired by the time acquisition unit as a basic value of the delay time for each network task. The delay time generation unit selects the basic value from among the basic values calculated by the basic value calculation unit according to a predetermined priority order of the basic values, and based on the selected basic value, A plurality of different individual delay times are generated for each of the basic values as delay times for the predetermined network task. The control unit causes the task execution unit to execute processing of the plurality of network tasks so that the processing time of the predetermined network task is delayed according to the delay time generated by the delay time generation unit. Repeat for each individual delay time .

In one aspect, the present invention can facilitate setting adjustment of processing timing for reproducing a failure state.

FIG. 1 is a system configuration diagram of a storage system. FIG. 2 is a block diagram of the storage device. FIG. 3 is a diagram illustrating an example of an FD number acquisition table according to the first embodiment. FIG. 4 is a diagram showing an example of a test matrix. FIG. 5 is a diagram for explaining an overview of the reproduction test according to Example 1. FIG. 6 is a flowchart of basic value acquisition processing. FIG. 7 is a flowchart of a reproduction test according to Example 1. FIG. 8 is a flowchart of the FD number acquisition process. FIG. 9 is a flowchart of the FD number acquisition process. FIG. 10 is a flowchart of delay control processing in FD number acquisition processing and FD number release processing. FIG. 11 is a diagram illustrating an example of an FD number acquisition table according to the second embodiment. FIG. 12 is a flowchart of the FD number acquisition table registration process in the second embodiment. FIG. 13 is a flowchart of the operation when releasing the FD number in the second embodiment.

Embodiments of the information processing apparatus and information processing program disclosed in the present application will be described in detail below with reference to the drawings. Note that the information processing apparatus and information processing program disclosed in the present application are not limited to the following embodiments.

FIG. 1 is a system configuration diagram of a storage system. The storage system 10 according to this embodiment includes a storage device 1, a switch 2, and servers 31 and 32.

The storage device 1 is network-connected to servers 31 and 32 via a switch 2. The storage device 1 receives data read or write requests from the server 31 or 32 through the network including the switch 2. The storage device 1 starts and processes network tasks corresponding to requests for each of the servers 31 and 32. The storage device 1 then responds to the server 31 or 32 with the results in response to the request obtained by processing each network task. Here, although two servers 31 and 32 are illustrated in FIG. 1, there is no particular limit to the number of servers.

The storage device 1 includes, for example, a CPU (Central Processing Unit) 11, a memory 12, a disk 13, and connection ports 14 and 15. In this embodiment, the storage device 1 has a plurality of disks 13. The CPU 11 is connected to a memory 12, a disk 13, and connection ports 14 and 15 via a bus.

The disk 13 is, for example, an auxiliary storage device such as a hard disk or an SSD (Solid State Drive). The disk 13 stores the OS and various other programs.

The memory 12 is, for example, a main storage device such as a DRAM (Dynamic Random Access Memory). Connection ports 14 and 15 are connected to switch 2. The connection ports 14 and 15 relay communication between the servers 31 and 32 and the CPU 11 via the network.

The CPU 11 reads the OS from the disk 13 and uses the memory 12 to execute and operate the OS. Furthermore, the CPU 11 operates network tasks on the OS.

Next, details of the storage device will be explained with reference to FIG. 2. FIG. 2 is a block diagram of the storage device. In this embodiment, the storage device 1 operates as a test device for fault investigation. As shown in FIG. 2, the storage device 1 includes a task execution unit 101, a history acquisition unit 102, a basic value calculation unit 103, a delay time generation unit 104, a reproduction control unit 105, an abnormality detection unit 106, a notification unit 107, and an abnormality detection unit 106. It has a processing execution unit 108. The functions of the task execution unit 101, history acquisition unit 102, basic value calculation unit 103, delay time generation unit 104, reproduction control unit 105, abnormality detection unit 106, notification unit 107, and abnormality processing execution unit 108 are realized by the CPU 11. Ru.

The task execution unit 101 has an FD number management table for managing whether each predetermined FD number is in use or unused. The FD number is a number assigned to each network and used as an identification number when processing a network task. The task execution unit 101 starts and executes a network task using the FD number management table.

Specifically, the task execution unit 101 starts a network task in response to an input request. Next, the task execution unit 101 acquires an unused FD number from its own FD number management table. Then, the task execution unit 101 assigns the obtained number to the activated network task. Further, the task execution unit 101 changes the information of the acquired FD number in the FD number management table while it is in use. Here, the acquisition of an FD number, the assignment of the acquired FD number to a network task, and the change of information in the FD number management table are collectively referred to as acquisition of an FD number.

Next, the task execution unit 101 executes the network task process using the acquired FD number. When the task execution unit 101 completes the processing of the network task, it releases the assignment of the FD number to the network task. Then, the task execution unit 101 changes the information of the unassigned FD number in the FD number management table to unused. Here, the release of the FD number assignment and the change of information in the FD number management table are collectively referred to as FD number release.

Here, in processing a network task, the task execution unit 101 may perform double release in which an already released FD number is released again from the network task. In this case, if the released FD number has already been acquired by another network task, the assignment of the FD number to the other network task will be cancelled. When the task execution unit 101 attempts to process a network task to which no FD number has been assigned, the FD number cannot be used and a processing error occurs.

The history acquisition unit 102 has a clock and an FD number acquisition table 120 shown in FIG. FIG. 3 is a diagram illustrating an example of an FD number acquisition table according to the first embodiment. As shown in FIG. 3, the FD number acquisition table 120 includes a task ID (Identifier) that is identification information of a network task, an FD number, call information that is a history of a function that called a network task, an FD number acquisition processing time, and an FD number. The number release processing time is registered. In FIG. 3, the state where the FD number is "0x00", the call information is "NULL", the FD number acquisition processing time is "0x00", and the FD number release processing time is "0x00" is a state in which each information is initialized. represents.

The history acquisition unit 102 monitors network task processing by the task execution unit 101. The history acquisition unit 102 acquires a task ID that is identification information of a network task started by the task execution unit 101. Then, the history acquisition unit 102 registers the acquired task ID in the FD number acquisition table 120. Here, task IDs are registered in the FD number acquisition table 120 without duplication.

Next, the history acquisition unit 102 acquires the FD number acquired for the network task by the task execution unit 101. Then, the history acquisition unit 102 registers the acquired FD number in association with the task ID. Further, the history acquisition unit 102 acquires the processing time for acquiring the FD number. Then, the history acquisition unit 102 registers the processing time for acquiring the acquired FD number in association with the task ID.

Further, the history acquisition unit 102 acquires information about the function that called the network task every time a network task having the registered task ID is called. Then, the history acquisition unit 102 sequentially registers the acquired function information in the call information column of the FD number acquisition table 120.

Thereafter, when the task execution unit 101 releases the FD number, the history acquisition unit 102 acquires the processing time for releasing the FD number. Then, the history acquisition unit 102 registers the acquired FD number release processing time in association with the task ID. This history acquisition unit 102 is an example of an “acquisition unit”.

When the history acquisition unit 102 updates the FD number acquisition processing time or the FD number release processing time, the basic value calculation unit 103 acquires the updated information from the FD number acquisition table 120. The basic value calculation unit 103 holds the FD number acquisition processing time or FD number release processing time of each network task acquired within a prespecified measurement time. The measurement time is, for example, 10 minutes after the storage device 1 is started. However, this measurement time is preferably determined depending on the configuration of the storage system 10, the performance of the storage device 1, and the like.

Then, the basic value calculation unit 103 calculates and holds the maximum value, average value, and mode of the FD number acquisition processing time and FD number release processing time as basic values for each network task. For example, in the case of the maximum value of the FD number acquisition processing time, the basic value calculation unit 103 holds the first acquired FD number acquisition processing time as the maximum value. After that, if the newly acquired FD number acquisition processing time is larger than the maximum value held, the basic value calculation unit 103 repeats the process of setting the newly acquired FD number acquisition processing time to the maximum value, thereby increasing the maximum value. Determine. In addition, in the case of the average value of the FD number acquisition processing time, the basic value calculation unit 103 calculates the total of the FD number processing times held, and calculates the average value by dividing the calculated total value by the number of calls. do. Furthermore, in the case of the most frequent value of the FD number acquisition processing times, the basic value calculation unit 103 counts the number of matching FD number acquisition processing times in nano sec units. Then, the basic value calculation unit 103 sets the most counted value as the mode value of the FD number acquisition processing time.

When the measurement time has elapsed and the measurement is completed, the basic value calculation unit 103 uses the maximum value, average value, and minimum value of the FD number acquisition processing time or FD number release processing time of each network task as the basic value and generates a delay time generation unit. Output to 104.

The delay time generation unit 104 has a basic value priority in advance. In this embodiment, the priority of the basic values is in the order of maximum value, average value, and most frequent value. The priority is the order in which the double release failure condition is likely to be reproduced. This priority is preferably determined according to the configuration of the storage system 10, the performance of the storage device 1, and the like.

Furthermore, the delay time generation unit 104 generates three types of delay time: descending order generation that increases the delay time step by step, ascending order generation that reduces the delay time step by step, and random generation that randomly determines the delay time within a certain range. It has a generation method. Here, a method for generating each delay time will be explained.

In the case of descending order generation, the delay time generation unit 104 sets one-tenth of the base value as the base time, and sets the first delay time as the base time. Thereafter, the delay time generation unit 104 repeatedly adds the basic times to sequentially generate delay times. Then, when the delay time is obtained by adding nine basic times, that is, when the delay time becomes the same value as the basic value in the next addition, the delay time generation unit 104 determines that all the delay times in the descending order generation have been generated. Then, the generation of delay time by descending order generation is completed.

Further, in the case of ascending order generation, the delay time generation unit 104 uses the first delay time as a base value. Thereafter, the delay time generation unit 104 repeatedly subtracts the base time, which is one-tenth of the base value, from the delay time to generate delay times in order. Then, when the delay time obtained by subtracting nine basic times, that is, the delay time becomes the same value as the basic value in the next subtraction, the delay time generation unit 104 determines that all delay times in the ascending order generation have been generated. Then, the generation of delay time by ascending order generation is completed.

In addition, in the case of random generation, the delay time generation unit 104 randomly selects a number from 1 to 9, and adds the selected number with a base time that is one-tenth of the base value as the delay time. do. Thereafter, the delay time generation unit 104 repeatedly selects the number of items, excluding duplicates, and generates delay times in order. If all of 1 to 9 are selected, the delay time generation unit 104 determines that all delay times in random generation have been generated, and completes the generation of delay times by random generation.

Here, in this embodiment, the delay time is generated using one-tenth of the basic value as the reference value, but this reference value may be any other value. For example, a value obtained by dividing the basic value into another number may be used as the reference value, or a value calculated based on the basic value by another method may be used. A plurality of delay times that are sequentially generated by the delay time generation unit 104 using these basic values are an example of "individual delay times."

Furthermore, the delay time generation unit 104 has a priority of the delay time generation method in advance. In this embodiment, the priority of the delay time generation methods is descending order generation, ascending order generation, and random generation. The priority is the order in which the double release failure condition is likely to be reproduced. Preferably, this priority is also determined according to the configuration of the storage system 10, the performance of the storage device 1, and the like.

The delay time generation unit 104 receives input of basic values of the FD number acquisition processing time and the FD number release processing time for each network task from the basic value calculation unit 103. When receiving a request from the reproduction control unit 105 to provide a delay time for the FD number acquisition processing time, the delay time generation unit 104 generates a test matrix 140 shown in FIG. 4 . FIG. 4 is a diagram showing an example of a test matrix.

Next, the delay time generation unit 104 determines the basic value of the FD number acquisition processing time to be used and the delay time generation method to be used according to the priority order, and uses the basic value to be used to determine the delay time generation method to be used. The delay time is sequentially generated by The delay time generation unit 104 then outputs the generated delay time to the reproduction control unit 105. Thereafter, the delay time generation section 104 sequentially generates delay times every time a certain period of time elapses and outputs them to the reproduction control section 105. The delay time generation unit 104 sequentially generates delay times for each network task. When the delay time generation unit 104 completes all delay time generation using one generation method using one base value for a certain network task, it adds a mark indicating completion to the corresponding column of the test matrix 140.

By referring to the test matrix 140, the delay time generation unit 104 can confirm that delay time generation using all basic values and all generation methods has been completed for all network tasks. When delay time generation using all basic values and all generation methods is completed for all network tasks, the delay time generation unit 104 controls the reproduction control unit 105 to indicate completion of generation of the delay time of the FD number acquisition processing time. to notify.

After that, when receiving a request from the reproduction control unit 105 to provide the delay time of the FD number release processing time, the delay time generation unit 104 generates the delay time of the FD number release processing time using all basic values and all generation methods. generation for each network task. The method of generating the delay time of the FD number release processing time is the same as that of generating the delay time of the FD number acquisition processing time, although the basic value is different. Thereafter, upon receiving notification of the end of the reproduction test from the reproduction control unit 105, the delay time generation unit 104 terminates generation of the delay time.

Furthermore, when the delay time generation unit 104 receives a notification of the end of delay time generation from the reproduction control unit 105 while performing the delay time generation process, the delay time generation unit 104 ends the delay time generation at that point.

The reproduction control unit 105 has in advance conditions for starting a reproduction test of double release failure occurrence. For example, the start conditions for a double release failure reproduction test are when a start instruction is input by the operator using the console of storage device 1, when storage device 1 has been operating for a certain period of time, or when a network task is completed. This is the case when a predetermined constant number is reached.

Furthermore, the reproduction control unit 105 has the time of acquiring the FD number or the time of releasing the FD number as a delay timing. The delay timing is information representing the timing at which a delay time is added to the processing of the network task in order to change the processing time of the network task.

The reproduction control unit 105 determines whether the start conditions for the reproduction test of double release failure occurrence are satisfied. When the start condition is satisfied, the reproduction control unit 105 selects the delay timing for the reproduction test to be performed first. In this embodiment, a reproduction test is first performed in which the delay timing is when the FD number is obtained.

Next, the reproduction control unit 105 requests the delay time generation unit 104 to provide the delay time of the FD number acquisition processing time. Then, the reproduction control unit 105 obtains the delay time from the delay time generation unit 104. Thereafter, the reproduction control unit 105 instructs the task execution unit 101 to add the acquired delay time to the FD number after acquisition.

When the reproduction control unit 105 receives a notification of the occurrence of an abnormality from the abnormality detection unit 106 while causing the task execution unit 101 to execute the process of the network task to which a delay time has been added, the reproduction control unit 105 decides to stop the reproduction test. . Then, the reproduction control unit 105 notifies the delay time generation unit 104 of the end of delay time generation. This reproduction control unit 105 is an example of a “control unit”.

When a certain period of time has passed without receiving notification of the occurrence of an abnormality from the abnormality detection unit 106, the reproduction control unit 105 acquires the next delay time of the FD number acquisition processing time from the delay time generation unit 104. Then, the reproduction control unit 105 instructs the task execution unit 101 to add the newly acquired delay time after acquiring the FD number. If the reproduction control unit 105 does not receive a notification of the occurrence of an abnormality from the abnormality detection unit 106, the reproduction control unit 105 causes the task execution unit 101 to sequentially add the delay time acquired at fixed time intervals to the FD number after acquisition. Thereafter, the reproduction control unit 105 receives a notification from the delay time generation unit 104 that the generation of the delay time for the FD number acquisition processing time has been completed.

Next, the reproduction control unit 105 selects the FD number release time as the delay timing. Then, the reproduction control unit 105 requests the delay time generation unit 104 to provide the delay time when releasing the FD number. In this case, the reproduction control unit 105 causes the task execution unit 101 to add the acquired delay time before releasing the FD number. If the reproduction control unit 105 does not receive a notification of the occurrence of an abnormality from the abnormality detection unit 106, the reproduction control unit 105 causes the task execution unit 101 to sequentially add the delay time acquired at regular intervals to the FD number acquisition and release. Thereafter, the reproduction control unit 105 receives a notification from the delay time generation unit 104 that the generation of the delay time of the FD number release processing time has been completed. Then, the reproduction control unit 105 determines that the reproduction test is completed. Thereafter, the reproduction control unit 105 notifies the delay time generation unit 104 of the completion of the reproduction test.

The abnormality detection unit 106 acquires the task ID and FD number of the network task when the task execution unit 101 releases the FD number. Then, the abnormality detection unit 106 searches the FD number acquisition table 120 included in the history acquisition unit 102 and determines whether the task ID and FD number pair match. Here, if no FD number is assigned to any network task, the abnormality detection unit 106 determines that there is a match.

If the task ID and FD number do not match, the abnormality detection unit 106 determines that a double release failure has occurred. Then, the abnormality detection unit 106 notifies the notification unit 107 and the abnormality processing execution unit 108 of the occurrence of the abnormality.

On the other hand, if the pair of task ID and FD number match, the abnormality detection unit 106 detects the FD number and call information of the network task whose FD number has been released in the FD number acquisition table 120 of the history acquisition unit 102. , initializes the FD number acquisition processing time and FD number release processing time.

The notification unit 107 receives a notification of the occurrence of an abnormality from the abnormality detection unit 106. Next, the notification unit 107 acquires the task ID, call information, and time information of the network task whose FD number has been released from the FD number acquisition table 120 that the history acquisition unit 102 has. Then, the notification unit 107 notifies the administrator of the storage device 1 of the task ID, call information, and time information of the network task whose FD number has been released, along with notification of the occurrence of the abnormality. For example, the notification unit 107 causes a monitor connected to the storage device 1 to display a message notifying the occurrence of an abnormality, and also causes the task ID, call information, and time information of the network task whose FD number has been released to be displayed. This will notify the administrator.

The abnormality processing execution unit 108 receives notification of the occurrence of an abnormality from the abnormality detection unit 106. Then, the abnormality processing execution unit 108 executes panic processing on the storage device 1, saves the dump, and shuts down the storage device 1.

FIG. 5 is a diagram for explaining an overview of the reproduction test according to Example 1. Here, a case will be explained in which network tasks A and B are processed. In the period 111, the task execution unit 101 first issues an FD number acquisition request to the number management table 110, and acquires the unused number #1 as the FD number. Thereafter, the task execution unit 101 makes a request to release the FD number #1 to the number management table 110, and upon receiving a release completion response, #1 is released. Next, the task execution unit 101 similarly secures #1 as the FD number for network task B, and then releases it. Further thereafter, the task execution unit 101 releases the FD number #1 by executing a double release process P1. In this case, network task B has not already been assigned #1 as the FD number, so no double release failure occurs.

Therefore, the task execution unit 101 receives an instruction to add a delay time after acquiring the FD number from the reproduction control unit 105, and responds with #1, which is an unused number, when securing the FD number for the network task A in the period 112. A delay time T is added before receiving the number from the number management table 110. After the delay time T has elapsed, the task execution unit 101 acquires the unused number #1 as the FD number of the network task A. Thereafter, the task execution unit 101 makes a request to release the FD number #1 to the number management table 110, and upon receiving a release completion response, #1 is released. Next, the task execution unit 101 similarly reserves #1 as the FD number for network task B. Here, since the timing of executing the double release process P2 of network task A is shifted due to the delay time, the task execution unit 101, before releasing the FD number #1 acquired by network task B, Double release processing P2 is executed to release the #1 FD number. In this case, the #1 FD number assigned to network task B is released, resulting in a double release failure.

In this manner, by providing a delay time, the execution timing of the process of the network task to which the delay has been provided is shifted, increasing the possibility that a double release failure will occur.

Next, the flow of basic value acquisition processing will be explained with reference to FIG. FIG. 6 is a flowchart of basic value acquisition processing.

The storage device 1 is powered on (step S101). The storage device 1 then completes normal startup (step S102).

Next, the task execution unit 101 starts a network task (step S103). In this case, multiple network tasks are sequentially activated in response to input requests.

Next, the task execution unit 101 obtains the FD number of the network task. The history acquisition unit 102 acquires the FD number acquisition time of the network task for which the FD number was acquired (step S104).

Next, the task execution unit 101 executes network task processing (step S105).

After that, the task execution unit 101 completes the processing of the network task (step S106).

Next, the task execution unit 101 releases the FD number of the network task. The history acquisition unit 102 acquires the FD number release processing time (step S107).

Next, the history acquisition unit 102 updates the average value, maximum value, and mode of the FD number acquisition processing time and FD number release processing time of the network task (step S108).

After that, the history acquisition unit 102 determines whether a predetermined measurement time has elapsed (step S109). If the measurement time has not elapsed (step S109: negative), the process returns to step S103.

On the other hand, if the measurement time has elapsed (step S109: affirmative), the history acquisition unit 102 ends the basic value acquisition process.

Here, the processing in steps S103 to S109 in FIG. 6 is executed for each network task.

Next, the overall flow of the reproduction test will be explained with reference to FIG. FIG. 7 is a flowchart of a reproduction test according to Example 1.

The reproduction control unit 105 determines whether the start conditions for a reproduction test of failure occurrence due to double release are satisfied (step S201). If the start condition is not met (step S201: negative), the reproduction control unit 105 waits until the start condition is met.

On the other hand, if the start condition is satisfied (step S201: affirmative), the reproduction control unit 105 creates the test matrix 140 (step S202).

The task execution unit 101 starts each network task (step S203).

The reproduction control unit 105 selects a network task to which a delay is to be applied (step S204).

Next, the reproduction control unit 105 selects a delay timing to be used from among the delay timings that have not been selected (step S205).

Then, the task execution unit 101, delay time generation unit 104, and reproduction control unit 105 execute FD number acquisition processing (step S206). This FD number acquisition process will be explained in detail later.

After that, the task execution unit 101 executes each network task (step S207).

The task execution unit 101 completes the processing of each network task (step S208).

Thereafter, the task execution unit 101, delay time generation unit 104, and reproduction control unit 105 execute FD number release processing (step S209). This FD number release process will be explained in detail later.

The abnormality detection unit 106 refers to the FD number acquisition table 120 based on the information of the released FD number and task ID, and determines whether a double release failure has occurred (step S210).

If a double release failure occurs (step S210: affirmative), the abnormality detection unit 106 notifies the notification unit 107 and the abnormality processing execution unit 108 of the failure occurrence. Upon receiving the notification of the failure occurrence, the notification unit 107 notifies the administrator of the task ID, FD number, and read information registered in the FD number acquisition table 120 (step S211).

Further, upon receiving the notification of the failure occurrence, the abnormality processing execution unit 108 executes panic processing, saves the dump information, and shuts down the storage device 1 (step S212).

On the other hand, if a double release failure has not occurred (step S210: negative), the delay time generation unit 104 generates the selected delay time using the selected basic value for the network task targeted for delay assignment. It is determined whether all delay time generation using the generation method has been completed (step S213). When the delay time generation unit 104 completes all delay time generation using the selected delay time generation method using the selected basic value for the network task targeted for delay assignment, the delay time generation unit 104 generates the corresponding column of the test matrix 120. Register information indicating completion. Then, the delay time generation unit 104 refers to the test matrix 140 and determines whether or not all delays have been applied to the network task to be delayed at the selected delay timing. If all delays regarding the network task to be delayed at the selected delay timing have not been completed (step S213: negative), the process returns to step S206.

On the other hand, if all the delays for the network tasks to be delayed at the selected delay timing are completed (step S213: affirmative), the delay time generation unit 104 completes the generation of the delay time at the selected delay timing. is notified to the reproduction control unit 105. The reproduction control unit 105 determines whether the reproduction test by adding a delay has been executed at all delay timings (step S214). If there remains a delay timing for which the reproduction test by adding a delay has not been performed (step S214: negative), the process returns to step S205.

On the other hand, if the reproduction test by adding a delay has been completed at all delay timings (step S214: affirmative), the reproduction control unit 105 determines whether the reproduction test by adding a delay has been executed for all activated network tasks. (Step S215).

If there remain network tasks that have not been subjected to the delay-based reproduction test (step S215: negative), the process returns to step S204. On the other hand, if the delay-applied reproduction test is completed for all activated network tasks (step S215: affirmative), the reproduction control unit 105 ends the delay-applied reproduction test.

Next, the flow of the FD number acquisition process will be explained with reference to FIG. FIG. 8 is a flowchart of the FD number acquisition process. The process shown in FIG. 8 is an example of the process executed in step S206 in FIG.

The task execution unit 101 acquires the FD number of the activated network task (step S301).

Next, the reproduction control unit 105 determines whether the task ID of the network task that has acquired the FD number matches the task ID of the network task that provides the selected delay (step S302). If the task IDs do not match (step S302: negative), the reproduction control unit 105 ends the process without applying a delay.

On the other hand, if the task IDs match (step S302: affirmative), the reproduction control unit 105 determines whether the selected delay timing is when the FD number is acquired (step S303). If the delay timing is when the FD number is released (step S303: negative), the reproduction control unit 105 ends the process without applying a delay.

On the other hand, if the delay timing is when the FD number is acquired (step S303: affirmative), the delay time generation unit 104 and reproduction control unit 105 execute delay control processing (step S304).

Next, the flow of the FD number release process will be explained with reference to FIG. FIG. 9 is a flowchart of the FD number acquisition process. The process shown in FIG. 9 is an example of the process executed in step S209 in FIG.

The reproduction control unit 105 determines whether the task ID of the network task that has acquired the FD number matches the task ID of the network task that provides the selected delay (step S401). If the task IDs do not match (step S401: negative), the reproduction control unit 105 does not apply a delay, and the process proceeds to step S404.

On the other hand, if the task IDs match (step S401: affirmative), the reproduction control unit 105 determines whether the selected delay timing is when the FD number is released (step S402). If the delay timing is when the FD number is obtained (step S402: negative), the reproduction control unit 105 does not apply a delay, and the process proceeds to step S404.

On the other hand, if the delay timing is at the time of the FD number (step S402: affirmative), the delay time generation unit 104 and reproduction control unit 105 execute delay control processing (step S403).

The task execution unit 101 releases the FD number of the network task for which processing has been completed (step S404).

Next, the flow of the delay control process will be explained with reference to FIG. FIG. 10 is a flowchart of delay control processing in FD number acquisition processing and FD number release processing. The process shown in FIG. 10 is an example of the process executed in step S304 in FIG. 8 and step S403 in FIG. Here, a network task selected as a target for delay addition is referred to as a target network task. This selection is performed, for example, in step S204 in FIG. 7.

The reproduction control unit 105 requests the delay time generation unit 104 to provide a delay time. The delay time generation unit 104 determines whether a certain period of time has passed since the previous delay time generation (step S501). Here, if the delay time has not yet been generated for the target network task, the delay time generation unit 104 determines that a certain period of time has passed. If a certain period of time has not passed since the previous delay time generation (step S501: negative), the delay time generation unit 104 proceeds to step S508.

On the other hand, if a certain period of time has passed since the previous generation of the delay time (step S501: affirmative), the delay time generation unit 104 refers to the test matrix 140. Then, the delay time generation unit 104 determines whether all delay time generation based on the selected reference value for the target network task has been completed (step S502). If there remains a delay time that has not been generated for the selected reference value (step S502: negative), the delay time generation unit 104 proceeds to step S504.

On the other hand, if all delay time generation based on the selected reference value is completed (step S502: affirmative), the delay time generation unit 104 selects the next reference value according to the priority of the reference value. (Step S503).

Next, the delay time generation unit 104 obtains the value of the selected basic value in the target network task. Then, the delay time generation unit 104 divides the basic value into 1/10 (step S504).

Next, the delay time generation unit 104 refers to the test matrix 140 and determines whether all delay time generation using the selected generation method has been completed (step S505). If there is a delay time that has not been generated among the delay times generated using the selected generation method (step S505: negative), the delay time generation unit 104 proceeds to step S507.

On the other hand, if all delay time generation using the selected generation method is completed (step S505: affirmative), the delay time generation unit 104 selects a delay time generation method according to the priority (step S505: Yes). S506).

The delay time generation unit 104 generates the next delay time according to the selected generation method (step S507). The delay time generation unit 104 then provides the generated delay time to the reproduction control unit 105.

The reproduction control unit 105 transmits the delay time information obtained from the delay time generation unit 104 to the task execution unit 101 and instructs the task execution unit 101 to add a delay. The task execution unit 101 receives the instruction from the reproduction control unit 105 and executes delay addition (step S508).

When the delay time generation unit 104 completes generation of the delay time for the target network using the selected generation method based on the selected reference value, the delay time generation unit 104 adds a mark indicating completion to the corresponding column of the test matrix 140 and then updates the test matrix 140. Update (step S509).

As described above, the storage device according to the present embodiment repeatedly executes a double release failure reproduction test in which a delay is added to the processing of each network task while changing the delay time. As a result, it is possible to automatically perform a reproduction test of the occurrence of a double release failure by varying the processing time of each network task. In this case, since the processing timing of each network task is changed, the state in which a specific FD number is acquired for a certain network task and the timing when another network task performs double release of that specific FD number are different. The probability that overlapping conditions will occur is increased. Therefore, the reproducibility of double release is improved and fault investigation can be facilitated. Furthermore, by making fault investigation easier, it is possible to shorten the time it takes to resolve a problem and reduce costs.

Further, in this embodiment, the information collected as history information for the reproduction test of the double release failure occurrence is the FD number, call information, FD number acquisition processing time, and FD number release processing time. Compared to normal cases in which as much information as possible is collected as trace information, the storage device according to this embodiment can reduce the amount of information acquired as history information, thereby reducing the size of the memory area for storing history information. can be reduced.

Furthermore, in the storage system according to the present embodiment, since the operation on the server side when performing a reproduction test is a normal operation, it is possible to reduce the impact of the execution of a reproduction test on the actual operation of the storage system. In addition, since the processing time of network tasks is changed to improve the reproducibility of failures caused by double release, it is possible to perform reproduction tests by running at least two network tasks, and It is not necessary to greatly increase the multiplicity. In this respect as well, failure investigation can be facilitated.

Next, Example 2 will be explained. The storage device according to this embodiment can use the number of network tasks whose FD numbers have been acquired in parallel during network task processing to determine the priority when selecting a network task to be delayed. This is different from Example 1. A block diagram of the storage device according to this embodiment is also shown in FIG. In the following description, explanations of the operations of the respective parts similar to those in the first embodiment will be omitted.

FIG. 11 is a diagram illustrating an example of an FD number acquisition table according to the second embodiment. The FD number acquisition table 120 according to this embodiment has columns for registering an existence flag and the number of existences.

The history acquisition unit 102 acquires the FD number acquired by the task execution unit 101 and the task ID of the network task from which the FD number is acquired. Then, the history acquisition unit 102 searches the FD number acquisition table 120 using the acquired task ID. If the acquired task ID does not exist in the FD number acquisition table 120, the history acquisition unit 102 adds the task ID to the FD number acquisition table 120.

Then, the history acquisition unit 102 enables the existence flag corresponding to the acquired task ID in the FD number acquisition table 120. This makes it possible to check from the FD number acquisition table 120 whether or not a specific network task is being processed. Furthermore, the history acquisition unit 102 increments the number of existence by one. At this time, the number of existence increases by one, indicating that the self is in operation.

Furthermore, when the history acquisition unit 102 acquires an FD number for a certain network task, it extracts other network tasks whose existence flags are valid in the FD number acquisition table 120. Then, the history acquisition unit 102 increments the number of extracted network tasks by one. That is, the number of network tasks that have acquired FD numbers during their operation can be grasped from the number of existing tasks that is increased at this time.

When the FD number is released, the abnormality detection unit 106 acquires the task ID of the network task whose FD number has been released. Then, the abnormality detection unit 106 detects the FD number corresponding to the acquired task ID, the call information, the FD number acquisition processing time, the FD number release processing time, and the existence of the FD number, if no double release failure occurs due to the release of the FD number. Delete the number. Furthermore, the abnormality detection unit 106 sets the existence flag to invalid.

When the delay time generation unit 104 receives a request to provide a delay time from the reproduction control unit 105, the delay time generation unit 104 acquires the number of existing network tasks registered in the FD number acquisition table 120 of the history acquisition unit 102. Then, the delay time generation unit 104 determines the priority for selecting network tasks in descending order of the number of existing tasks. Here, network tasks that exist in the same number are freely selected in order. For example, if the number of existing network tasks is the same, the delay time generation unit 104 may select the network tasks in order from the top of the FD number acquisition table 120.

Next, with reference to FIG. 12, the flow of registration processing of the FD number acquisition table 120 in this embodiment will be explained. FIG. 12 is a flowchart of the FD number acquisition table registration process in the second embodiment.

The task execution unit 101 acquires an FD number for the activated network task (step S601).

Next, the task execution unit 101 notifies the history acquisition unit 102 of the acquired FD number and the task ID of the network task from which the FD number is acquired. The history acquisition unit 102 acquires from the task execution unit 101 the acquired FD number and the task ID of the network task from which the FD number is acquired. Then, the history acquisition unit 102 searches the FD number acquisition table 120 using the acquired task ID (step S602).

Then, the history acquisition unit 102 determines whether the network task that acquired the FD number has been registered in the FD number acquisition table 120 (step S603). If the network task that has acquired the FD number has been registered (step S603: affirmative), the history acquisition unit 102 proceeds to step S605.

On the other hand, if the network task that acquired the FD number is unregistered (step S603: negative), the history acquisition unit 102 registers the task ID in the FD number acquisition table 120 and registers the network task that acquired the FD number. Information is added (step S604).

Next, the history acquisition unit 102 sets the existence flag corresponding to the acquired task ID to valid (step S605).

Furthermore, the history acquisition unit 102 increments the number of existences corresponding to the acquired task ID by one (step S606).

Next, the history acquisition unit 102 selects one other network task from the FD number acquisition table 120 (step S607).

Then, the history acquisition unit 102 determines whether the existence flag of the selected network task is valid (step S608). If the existence flag is invalid (step S608: negative), the history acquisition unit 608 proceeds to step S610.

On the other hand, if the existence flag is valid (step S608: affirmative), the history acquisition unit 102 increments the number of existence of the selected network task by one (step S609).

After that, the history acquisition unit 102 determines whether or not the existence flags of all network tasks have been checked (step S610).

If there is a network task whose existence flag has not been checked (step S610: negative), the history acquisition unit 102 returns to step S607. On the other hand, when the existence flags of all network tasks have been checked (step S610: affirmative), the history acquisition unit 102 ends the acquisition process.

The history acquisition unit 102 executes the above process every time a new FD number is acquired.

Next, referring to FIG. 13, the operation when releasing the FD number in this embodiment will be described. FIG. 13 is a flowchart of the operation when releasing the FD number in the second embodiment.

The task execution unit 101 releases the FD number of the network task for which processing has been completed (step S701).

The abnormality detection unit 106 acquires the released FD number and the task ID of the network task whose FD number has been released from the task execution unit 101. Then, the abnormality detection unit 106 searches the FD number acquisition table 120 held by the history acquisition unit 102 using the acquired FD number and task ID (step S702).

Then, the abnormality detection unit 106 determines whether a combination matching the combination of the acquired task ID and the acquired FD number exists among the combinations registered in the FD number acquisition table 120 (step S703 ).

If a matching combination exists in the FD number acquisition table 120 (step S703: affirmative), the abnormality detection unit 106 detects the FD number, call information, FD number acquisition processing time, and FD number release processing time corresponding to the obtained task ID. and the number of existences is initialized (step S704).

Further, the abnormality detection unit 106 sets the existence flag corresponding to the acquired task ID to invalid (step S705). Then, the abnormality detection unit 106 ends the process.

On the other hand, if the matching combination does not exist in the FD number acquisition table 120 (step S703: negative), the abnormality detection unit 106 detects an abnormality (step S706).

Then, the abnormality detection unit 106 notifies the notification unit 107 of the occurrence of a failure due to double release. The notification unit 107 receives the notification from the abnormality detection unit 106 and notifies the administrator of failure information regarding the occurrence of a failure due to double release (step S707).

Furthermore, the abnormality detection unit 106 notifies the abnormality processing execution unit 108 of the occurrence of a failure due to double release. The abnormality processing execution unit 108 receives the notification from the abnormality detection unit 106, executes panic processing, collects and stores a dump, and shuts down the storage device 1.

As described above, the storage device according to this embodiment acquires the number of existing network tasks for which FD numbers have been acquired in parallel with execution of processing of a specific network task. Then, the storage device according to the present embodiment determines the priority order of network task selection according to the size of the number of existing tasks. In this case, the priority of network task selection is determined by the number of other network tasks whose FD numbers were acquired while a certain network task is being processed. If many network tasks are started in parallel during operation, there is a greater possibility that failures due to double frees will occur. Therefore, by prioritizing the selection of network tasks according to the number of tasks present, reproducibility can be further improved and fault investigation can be facilitated.

1 Storage device 2 Switch 10 Storage system 11 CPU
12 Memory 13 Disk 14, 15 Connection port 31, 32 Server 101 Task execution unit 102 History acquisition unit 103 Basic value calculation unit 104 Delay time generation unit 105 Reproduction control unit 106 Abnormality detection unit 107 Notification unit 108 Abnormality processing execution unit

Claims (5)

a task execution unit that executes processing of multiple network tasks;
an acquisition unit that acquires the processing time of a predetermined network task by the task execution unit;
a basic value calculation unit that calculates one or more of the maximum value, average value, and mode of the processing time acquired by the acquisition unit as a basic value of the delay time for each of the network tasks;
The basic value is selected from among the basic values calculated by the basic value calculation unit according to a predetermined priority order of the basic values, and based on the selected basic value, the basic value is calculated for the predetermined network task. a delay time generation unit that generates a plurality of different individual delay times as delay times for each of the basic values ;
The individual delay time causes the task execution unit to execute processing of a plurality of network tasks so that the processing time of the predetermined network task is delayed according to the individual delay time generated by the delay time generation unit. An information processing device comprising: a control unit that repeats each time ; The information processing apparatus according to claim 1 , wherein the control unit has the priority order in the order of the maximum value, the average value, and the mode. The delay time generation unit generates the delay time for each generation method using a plurality of different generation methods,
The control unit causes the task execution unit to execute processing of the plurality of network tasks for each of the generation methods so that the processing time of the predetermined network task is delayed according to the delay time. 2. The information processing device according to 1 or 2 . The task execution unit secures an identification number used for executing processing for each of the network tasks, executes the processing of each network task using the secured identification number, and when the processing is completed, transfers the identification number to the network task. release,
2. The acquisition unit acquires, as the processing time, one or both of the time required for the task execution unit to secure or release the identification number for the predetermined network task. The information processing device according to any one of -3 . Performs processing of multiple network tasks,
Get the processing time of a given network task,
Calculating one or more of the maximum value, average value, and mode of the acquired processing time as a basic value of delay time for each network task,
From among the calculated basic values, the basic value is selected according to a predetermined priority order of the basic values, and based on the selected basic value, a plurality of different individual delay times for the predetermined network task are determined . Generate a delay time for each of the basic values ,
Repeating execution of the processing of the plurality of network tasks for each individual delay time so that the processing time of the predetermined network task is delayed according to the generated delay time.
An information processing program that causes a computer to perform processing.

Images