JP6368842B2 - Process monitoring program and process monitoring system - Google Patents

Description

  The present invention relates to process monitoring in an information processing system.

  In recent years, various services are provided from a server to a client in a client-server system. In order to appropriately provide these services, various processes executed on the server are monitored by a process monitoring system or the like. For example, in order to determine whether each process on the server is properly activated, the process monitoring system periodically performs life / death determination on each process. For example, this life / death determination is performed based on whether the process monitoring system periodically transmits a message to the monitoring target server and a normal response (OK response) to the message is received within the timeout period. As a result of the life / death determination, when it is detected that the process is not operating normally (for example, when an NG response is received or when a response is not received within the timeout period), the process monitoring system Switch the process to another server (takeover).

JP 2006-146319 A JP 10-154085 A

  However, as the client / server system becomes complicated, an erroneous life / death determination may be performed. For example, when a response to a process that operates normally on the server is returned as an NG response, or an OK response does not reach the process monitoring system within the time-out period, a normally operating process operates normally. If not, it may be misjudged.

  Such an erroneous determination is typically caused by a bug in each program, and the bug is corrected by the program provider. However, as the middleware executed on the server increases, such a misjudgment occurs accidentally even though the probability of occurrence is low due to a bug that relies on a system environment in which many middlewares are mixed. Events are beginning to appear. This type of bug is difficult to reproduce artificially and may not be expected to be fixed by the program provider. When takeover is executed based on such an erroneous determination, various problems such as time taken by the takeover, waste of manpower, and interruption of service during takeover occur.

  In view of the above problems, an object of the present invention is to provide a process monitoring technique for reducing erroneous determination in process life / death determination caused by a bug that appears accidentally.

  In order to solve the above-mentioned problem, according to one aspect of the present invention, a step of transmitting each process monitoring message of a set of process monitoring messages to a monitoring target process at different transmission timings within a first time; The present invention relates to a process monitoring program that causes a computer to execute a step of determining whether or not the process is in a failure state based on a reception state of a response to a set of process monitoring messages.

  According to another aspect of the present invention, there is provided a transmitting unit that transmits each process monitoring message of a set of process monitoring messages at different transmission timings within a first time with respect to a process to be monitored, and the one set of process monitoring messages. The present invention relates to a process monitoring system including a receiving unit that receives a response to, and a determination unit that determines whether the process is in a failure state based on a reception state of the response.

  According to the present invention, it is possible to reduce erroneous determination in process life / death determination due to a bug that appears accidentally, and to detect a failure without waiting for timeout and to quickly start takeover.

FIG. 1 is a schematic diagram showing an information processing system according to an embodiment of the present invention. FIG. 2 is a block diagram showing a hardware configuration of a process monitoring system according to an embodiment of the present invention. FIG. 3 is a block diagram showing a functional configuration of a process monitoring system according to an embodiment of the present invention. FIG. 4 is a flowchart showing processing in the process monitoring system according to the embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  In an embodiment described later, a process monitoring system for reducing erroneous determination in process life / death determination due to a bug that appears accidentally is disclosed. The process monitoring system transmits each process monitoring message of a set of process monitoring messages to the monitoring target process executed on the server at different transmission timings within the first time. In one embodiment, two process monitoring messages are sent for each process on the server with a minute time difference (eg, a time difference of several hundred milliseconds). When the process monitoring message is received, each process returns a normal response (OK response) to the process monitoring system if it is operating normally, and an NG response to the process monitoring system if it is not operating normally. Or return no response within the timeout period. As described above, the process monitoring system determines whether the process is in a failure state based on the reception status of the response to the transmitted set of process monitoring messages, and takes over to another server according to the determination result.

  In this way, by sending multiple process monitoring messages at different transmission timings and making life / death determinations based on these response results, misjudgment due to bugs that occur accidentally with low probability is avoided. Is possible. This is because such bugs often occur due to accidental coincidence at various timings, and it is extremely unlikely that an erroneous determination will occur at other timings. This makes it possible to avoid takeover based on incorrect life / death judgments, network interruptions and service interruptions associated with this, and to detect a failure without waiting for a timeout when the process is in a failure state. And takeover can be started quickly.

  First, an information processing system according to an embodiment of the present invention will be described with reference to FIG. The information processing system is, for example, a client / server system, and the server provides various services in response to requests from the client. In this embodiment, the information processing system is suitable for a system that requires high availability, and switching to a spare server (takeover or failover) in response to the occurrence of a failure in a running server or service It is designed to minimize interruptions and to quickly initiate takeover in the event of a failure.

  FIG. 1 is a schematic diagram showing an information processing system according to an embodiment of the present invention. As illustrated in FIG. 1, the information processing system 10 includes a process monitoring system 100, servers 201 and 202, a database (DB) 250, and a terminal device 300. In the illustrated embodiment, the server 201 is an active server, and the server 202 is a spare server for the server 201.

  The process monitoring system 100 monitors each process executed in the operating server 201. As will be described in detail below, the process monitoring system 100 transmits a plurality of life / death monitoring messages to each process executed in the server 201 at different transmission timings, and a response from each process to the life / death monitoring message. Based on the reception result, it is determined whether the process is operating normally or a failure has occurred. If it is determined that a failure has occurred, the process monitoring system 100 switches the server 201 operating for the process to the spare server 202 (takeover or failover).

  The server 201 is an active server and provides various services to the terminal device 300 that is a client. In one embodiment, when receiving a processing request from the terminal device 300, the server 201 acquires data related to the processing from the database 250, executes the requested processing on the acquired data, and sends the processing result to the terminal. Return to device 300. In order to execute these processes, various programs such as an operating system (OS), middleware, and applications are activated in the server 201, and various processes are executed with respect to these programs. Each process returns a normal response (OK response) to the life / death monitoring message transmitted from the process monitoring system 100 when the process is operating normally, and the process is not operating normally. An NG response is returned or no response is returned due to failure.

  The server 202 is a spare server for the active server 201. When a failure occurs in the server 201, the server 202 starts a corresponding process instead of the server 201 in response to a takeover instruction from the process monitoring system 100. Typically, this takeover may take 10 to 15 minutes or the like, and service provision to the terminal device 300 may be temporarily interrupted during the takeover execution. For this reason, in the information processing system 10 that requires particularly high availability, the execution of takeover should be minimized.

  The database 250 stores various data necessary for executing various processing requests from the terminal device 300. For example, these data are acquired and manipulated via database middleware activated in the servers 201 and 202.

  The terminal device 300 is a client device in the information processing system 10 and is typically realized by an information terminal such as a desktop computer or a notebook computer. The terminal device 300 transmits various processing requests to the server 201 in operation, and receives processing results from the server 201. When a failure is detected in the process of the server 201 and a takeover to the server 202 is executed, the terminal device 300 starts an exchange with the server 202.

  Next, a process monitoring system according to an embodiment of the present invention will be described with reference to FIGS. As described above, the process monitoring system 100 transmits each process monitoring message of one set of process monitoring messages to the monitoring target process executed on the server 201 at different transmission timings within a predetermined transmission time. Based on the reception status of the response to the transmitted set of process monitoring messages (for example, whether the response is received within a predetermined reception time, whether the received response is an OK response, etc.) Determine if it is a fault condition. The process monitoring system 100 executes a takeover to the server 202 for the process according to the determination result.

  FIG. 2 is a block diagram showing a hardware configuration of a process monitoring system according to an embodiment of the present invention. As shown in FIG. 2, the process monitoring system 100 includes a drive device 101, an auxiliary storage device 102, a memory device 103, a CPU (Central Processing Unit) 104, an interface device 105, and a timer 106 that are interconnected via a bus B. Have.

  Various programs including a process monitoring program that realizes various functions and processes described later in the process monitoring system 100 may be provided by a recording medium 107 such as a CD-ROM (Compact Disk-Read Only Memory). When the recording medium 107 storing the program is set in the drive device 101, the program is installed from the recording medium 107 to the auxiliary storage device 102 via the drive device 101. However, it is not always necessary to install the program using the recording medium 107, and the program may be downloaded from any external device via a network (not shown). The auxiliary storage device 102 stores the installed program and also stores necessary files and data.

  The memory device 103 reads the program and data from the auxiliary storage device 102 and stores them when there is an instruction to start the program. The CPU 104 executes various functions and processing of the process monitoring system 100 as described later according to various data such as a program stored in the memory device 103 and parameters necessary for executing the program. The interface device 105 is used as a communication interface for connecting to a network or an external device. The timer 106 is provided as a time measuring means.

  However, the process monitoring system 100 is not limited to the hardware configuration described above, and may be realized by any appropriate information processing device such as a server, a personal computer, or a mobile device.

  FIG. 3 is a block diagram showing a functional configuration of a process monitoring system according to an embodiment of the present invention. As illustrated in FIG. 3, the process monitoring system 100 includes a transmission unit 110, a reception unit 120, a determination unit 130, and a migration instruction unit 140.

  The transmission unit 110 transmits each process monitoring message of a set of process monitoring messages to the monitoring target process of the server 201 at different transmission timings within a predetermined transmission time. In one embodiment, the transmitting unit 110 transmits two process monitoring messages for each process by duplicating or duplicating them with a minute time difference (for example, several hundred milliseconds). The predetermined transmission time is not limited to this, but is set to a time shorter than at least the timeout time. Note that transmission of each group of process monitoring messages is typically performed periodically at intervals longer than the transmission time.

  The receiving unit 120 receives a response to the set of process monitoring messages transmitted by the transmitting unit 110. When the process that is the target of the process monitoring message is operating normally, a normal response (OK response) is returned to the receiving unit 120. On the other hand, if the process targeted by the process monitoring message is not operating normally, an NG response is returned to the receiving unit 120. Alternatively, when the process targeted by the process monitoring message is not operating normally, the response itself may not be returned to the receiving unit 120.

  The determination unit 130 determines whether the process to be monitored is in a failure state based on a reception state of a response to each process monitoring message of a set of process monitoring messages. When determining that the process to be monitored is in a failure state, the determination unit 130 notifies the migration instruction unit 140 that the process is in a failure state.

  In one embodiment, the determination unit 130 determines whether a response to the transmitted set of process monitoring messages has been received within a predetermined reception time (for example, a timeout time), and more than a predetermined number of these responses. Is not received within a predetermined reception time, it is determined that the process is in a failure state. As an example, the reception time may be compared with the time from the transmission time of each process monitoring message to the reception time. That is, when the time from the transmission time to the reception time of a predetermined number of process monitoring messages out of a plurality of transmitted process monitoring messages exceeds a predetermined reception time such as a timeout value, the determination unit 130 Is determined to be in a failure state. Typically, the predetermined reception time is set to a timeout value, and is set longer than the predetermined transmission time (minute time) described above.

  As an example, when the transmission unit 110 transmits two process monitoring messages as a set of process monitoring messages at different transmission timings, the determination unit 130 determines that both of the two responses to the two process monitoring messages are predetermined. If it is not received within the reception time (timeout time), it may be determined that the process is in a failure state. In other words, a response to the first process monitoring message is not received within a predetermined reception time from the transmission time of the first process monitoring message, and a response to the second process monitoring message is predetermined from the transmission time of the second process monitoring message. If not received within the reception time, it may be determined that the process is in a failure state.

  As another example, when the transmission unit 110 transmits two process monitoring messages as a set of process monitoring messages at different transmission timings, the determination unit 130 determines which of the two responses to these two process monitoring messages. If either one is not received within a predetermined reception time, it may be determined that the process is in a failure state. That is, a response to the first process monitoring message has not been received within a predetermined reception time from the transmission time of the first process monitoring message, or a response to the second process monitoring message is predetermined from the transmission time of the second process monitoring message. If it is not received within the reception time, it may be determined that the process is in a failure state.

  Which criterion is used may be determined according to the embodiment of the information processing system 10. In one embodiment, the determination criterion may be determined based on the characteristics of the process, the cost of takeover, the occurrence frequency of bugs that cause erroneous determination, and the like.

  For example, when another process alternative to the process is operating normally, the determination unit 130 responds to the fact that both of the two responses are not received within a predetermined reception time. May be determined to be in a failure state. On the other hand, when there is no other process alternative to the process, the determination unit 130 determines that one of the two responses is not received within the predetermined reception time, and the process is in a failure state. It may be determined that This is because even if the process is in a failure state with a considerable degree of accuracy, it is not necessary to take over immediately because the alternative process executes the process in place of the process.

  When the cost for takeover (for example, the time required for takeover, the availability of the server 201, etc.) is high, the determination unit 130 responds that both of the two responses have not been received within the predetermined reception time. Thus, it may be determined that the process is in a failure state. On the other hand, when the cost for takeover is not so high, the determination unit 130 determines that the process is in a failure state in response to the fact that one of the two responses is not received within the predetermined reception time. You may judge. It is possible to determine whether takeover is necessary in consideration of the cost associated with takeover.

  When it is estimated that the occurrence frequency of bugs that cause erroneous determination is high, the determination unit 130 responds that both of the two responses are not received within a predetermined reception time, You may determine with it being in a failure state. On the other hand, when it is estimated that the occurrence frequency of bugs causing erroneous determination is low, the determination unit 130 responds to the fact that either one of the two responses has not been received within a predetermined reception time. It may be determined that the process is in a failure state. Thereby, it is possible to reduce the execution of takeover due to erroneous determination.

  When the transmission unit 110 transmits two process monitoring messages as a set of process monitoring messages at different transmission timings, the determination unit 130 determines that either one of the two responses to the two process monitoring messages is In response to not having been received within a predetermined reception time, the transmitting unit 110 may be retried before determining that the process is in a failure state. For example, the determination unit 130 may instruct the transmission unit 110 to retransmit two process monitoring messages for the process at different transmission timings.

  In another embodiment, the determination unit 130 determines whether a response to the transmitted set of process monitoring messages is an NG response. If a predetermined number or more of the responses are NG responses, the process is You may determine with it being in a failure state.

  As an example, when the transmission unit 110 transmits two process monitoring messages as a set of process monitoring messages at different transmission timings, the determination unit 130 determines that both of the two responses to the two process monitoring messages are NG. If it is a response, it may be determined that the process is in a failure state. As another example, when the transmission unit 110 transmits two process monitoring messages as a set of process monitoring messages at different transmission timings, the determination unit 130 determines which of the two responses to these two process monitoring messages. If either one is an NG response, it may be determined that the process is in a failure state.

  Which criterion is used may be determined according to the embodiment of the information processing system 10 or the accuracy of fault estimation. In one embodiment, the determination criterion may be determined based on the characteristics of the process, the cost of takeover, the occurrence frequency of bugs that cause erroneous determination, and the like.

  For example, when another process alternative to the process is operating normally, the determination unit 130 responds that both of the two responses are NG responses, and the process is in a failure state. May be determined. On the other hand, if there is no alternative process in the process, the determination unit 130 determines that the process is in a failure state in response to the fact that one of the two responses is an NG response. May be. Further, when the cost for takeover (for example, the time required for takeover, the availability of the server 201, etc.) is large, the determination unit 130 responds that both of the two responses are NG responses, You may determine with it being in a failure state. On the other hand, when the cost for takeover is not so large, the determination unit 130 may determine that the process is in a failure state in response to the fact that one of the two responses is an NG response. . Further, when it is estimated that the occurrence frequency of bugs causing erroneous determination is high, the determination unit 130 determines that the process is in a failure state in response to both of the two responses being NG responses. You may judge. On the other hand, when it is estimated that the occurrence frequency of bugs that cause erroneous determination is low, the determination unit 130 responds that one of the two responses is an NG response, and the process is in a failure state. You may determine that there is.

  When the transmission unit 110 transmits two process monitoring messages as a set of process monitoring messages at different transmission timings, the determination unit 130 determines that either one of the two responses to the two process monitoring messages is In response to being an NG response, the transmitting unit 110 may be made to retry before determining that the process is in a failure state. For example, the determination unit 130 may instruct the transmission unit 110 to retransmit two process monitoring messages for the process at different transmission timings.

  In still another embodiment, the above-described embodiments are combined, and the determination unit 130 determines whether a response to the transmitted set of process monitoring messages has been received within a predetermined reception time (for example, a timeout time), and It is determined whether a response to the transmitted set of process monitoring messages is an NG response, and when a predetermined number or more of these responses are not received within a predetermined reception time and / or a predetermined number of the responses If the above response is an NG response, it may be determined that the process is in a failure state.

  In still another embodiment, the determination unit 130 compares the time required to transmit the transmitted set of process monitoring messages with the time required to receive a response to the process monitoring messages. If the time required for reception is greater than or equal to a predetermined threshold than the time required for transmission, the process may be determined to be in a failure state. In general, the time required to transmit a set of process monitoring messages, that is, the time from the transmission of the first process monitoring message to the transmission of the last process monitoring message is the first time for the transmitted process monitoring message. It is considered that it is approximately equal to the time from when the response is received until the last response is received. Therefore, the time from receiving the first response to the process monitoring message until receiving the last response greatly exceeded the time from sending the first process monitoring message to sending the last process monitoring message. In other words, that is, when the time from the transmission of the first process monitoring message to the transmission of the last process monitoring message exceeds a predetermined threshold, the determination unit 130 determines that the process is in a failure state. Also good. Here, if there is a process monitoring message that has not received a response within the timeout period, the time required to receive a response to the process monitoring message is calculated considering only the received response, and the process fails. You may determine whether it is in a state. In addition, if the number of process monitoring messages that could not be received within the timeout period is counted and the number is equal to or greater than the predetermined number, the process fails regardless of the time required to receive a response to the process monitoring message. You may determine with a state.

  When it is determined that the process to be monitored is in a failure state, the migration instruction unit 140 stops the process by the process and continues the process by a process that replaces the process. In one embodiment, the migration instructing unit 140 switches (takes over) the process of the server 201 determined to be in a failure state to the alternative process of the spare server 202, and causes the alternative process to continue the process.

  Next, processing in the process monitoring system according to an embodiment of the present invention will be described with reference to FIG. FIG. 4 is a flowchart showing processing in the process monitoring system according to the embodiment of the present invention.

  As shown in FIG. 4, in step S101, the process monitoring system 100 transmits each process monitoring message of a set of process monitoring messages at different transmission timings within a predetermined transmission time. For example, each process monitoring message may be transmitted with a predetermined time difference (for example, several hundred milliseconds). Each set of process monitoring messages is periodically transmitted at a predetermined interval.

  In step S102, the process monitoring system 100 executes a takeover necessity determination. That is, the process monitoring system 100 determines whether or not a takeover is necessary based on a predetermined determination criterion based on the reception status of each response of the plurality of process monitoring messages transmitted in step S101. Execute. In one embodiment, the takeover necessity determination may be to determine whether each response to the plurality of process monitoring messages transmitted in step S101 is received within a predetermined reception time (for example, a timeout time). . In another embodiment, the takeover necessity determination may be to determine whether each response to the plurality of process monitoring messages transmitted in step S101 is an NG response. In still another embodiment, the determination as to whether takeover is necessary is performed by determining whether or not each response to the plurality of process monitoring messages transmitted in step S101 has been received within a predetermined reception time (for example, a timeout time). It may be determined whether each response to the process monitoring message is an NG response.

  In step S103, the process monitoring system 100 determines whether takeover is necessary. Corresponding to the takeover necessity determination performed in step S102, it is determined whether takeover is necessary. For example, when the determination as to whether takeover is necessary is to determine whether each response to a plurality of process monitoring messages has been received within a predetermined reception time (for example, a timeout time), the process monitoring system 100 determines that the process monitoring message It may be determined that it is necessary to execute takeover in response to a response that a predetermined number or more of responses to is not received within a predetermined time. Further, when the determination as to whether takeover is necessary is to determine whether each response to a plurality of process monitoring messages is an NG response, the process monitoring system 100 determines that a predetermined number or more of the responses to the process monitoring message are NG. In response to the response, it may be determined that a takeover needs to be performed. Furthermore, the determination as to whether takeover is necessary is to determine whether or not each response to a plurality of process monitoring messages has been received within a predetermined reception time, and to determine whether or not each response to the received process monitoring message is an NG response. In this case, the process monitoring system 100 does not receive a predetermined number or more of the responses to the process monitoring message within a predetermined reception time, and receives a predetermined number or more of the responses to the received process monitoring message. In response to being an NG response, it may be determined that a takeover needs to be executed.

  If it is determined that a takeover is necessary (step S103: Y), in step S104, the process monitoring system 100 performs a takeover on the process in the server 201 and starts an alternative process in the server 202. Then, the processing related to the process is continued. On the other hand, if it is determined that no takeover is necessary (step S103: N), the process monitoring system 100 resumes step S101 after a predetermined interval has elapsed.

  As mentioned above, although the Example of this invention was explained in full detail, this invention is not limited to the specific embodiment mentioned above, In the range of the summary of this invention described in the claim, various deformation | transformation・ Change is possible.

DESCRIPTION OF SYMBOLS 10 Information processing system 100 Process monitoring system 110 Transmission part 120 Reception part 130 Determination part 140 Migration instruction | indication part 201,202 Server 250 Database 300 Terminal device

Claims (1)

Sending each process monitoring message of a set of process monitoring messages with a minute time difference to the process to be monitored;
Determining whether the process is in a failure state based on a reception state of a response to the set of process monitoring messages;
A process monitoring program for causing a computer to execute
The determining step determines whether or not a response to the set of process monitoring messages has been received within a timeout time longer than the minute time difference, and a predetermined number or more of the responses to the set of process monitoring messages is the timeout. A process monitoring program for determining that the process is in a failure state when it is not received in time.

Images