JP6295176B2 - Message processing apparatus and message processing method - Google Patents

Description

  The present invention relates to a message processing apparatus and a message processing method for classifying and extracting a message that matches a predetermined pattern with respect to an input message, and more particularly to a message processing method having real-time characteristics.

  Log messages output by an operating system (OS) or application software often include information indicating a failure or error. It is useful to detect these messages by pattern matching and automatically perform error handling such as predetermined restoration processing and reporting processing. The log message may include a dynamically changing parameter in addition to a fixed range in which the content does not change, and error handling depending on the content of the parameter may be performed.

  The log message may be output by being divided into a plurality of messages for one occurrence event. In particular, when parameters necessary for error handling are included in the plurality of messages described above, it is necessary to detect these plurality of messages collectively as a series of messages. Furthermore, in a plurality of contexts, when the same type of event that outputs a series of messages occurs at the same time, the messages that make up the series of messages may be mixed together. In this case, it is not possible to unconditionally determine which message and which message made up the original series of messages. In order to perform proper error handling, it is necessary to separate a series of mixed messages and reconstruct the original series of messages.

  In addition, depending on the type of error and SLO (Service Level Objective), immediate automatic recovery processing may be required, and in this case, the above-mentioned message detection and series of message reconstruction processing satisfy real-time requirements. Is required. That is, it must be possible to process within a predictable time.

  In particular, regarding the separation and reconfiguration processing of a series of mixed messages, although the technical field is different, application of network monitoring technology can be considered (for example, Patent Document 1).

Special table 2013-534105

  In Patent Document 1, it is assumed that streams are distinguished using transmission / reception ports and IP address information attached to packets. However, in the log message analysis processing, since information for distinguishing the output context is not usually added to the log message, the output context cannot be uniquely identified. This also means that the end of a series of messages cannot be uniquely determined. At this time, when the tree is reconstructed using the tree state indicator by the method of Patent Document 1, that is, when the original series of messages is reconstructed, the necessary memory size is unlimited. That is, it is necessary to hold a tree for all combinations of messages that satisfy a series of message composition rules. Further, since the end of a series of messages is unclear, the holding period is unlimited.

  In a computer system, the process of dynamically securing memory may take a long time, such as saving the contents of the memory to a secondary storage device or waiting for memory release. Therefore, in order to ensure real-time performance, it is necessary to reserve memory of a finite size in advance without dynamically securing memory. However, as described above, since the required memory size cannot be finitely determined by the method of Patent Document 1, real-time performance cannot be ensured.

  One of the typical message processing devices that solve the above-described problems according to the present invention includes a storage unit that sequentially stores the messages received within a predetermined time from an input interface that sequentially receives messages output from a plurality of contexts. A message extraction unit that decomposes the message stored in the storage unit into element messages that are predetermined message output units on the output side of the plurality of contexts; and a predetermined occurrence pattern of the element message And a message restructuring unit that reconstructs the message.

  The present invention is also grasped as a message processing method performed by the message processing apparatus.

  According to the present invention, the size of the main memory required for message processing can be limited to a storage area sufficient to store messages output within a predetermined time. Therefore, if a memory of a necessary size is secured in advance, processing delay due to memory exhaustion can be avoided, and processing within a predictable time becomes possible.

It is a hardware block diagram at the time of implement | achieving this invention on a computer. It is a functional block diagram in an exemplary embodiment of the present invention. It is an example of a specific input message in one of the application destinations of this invention. FIG. 4 is an example of a result in which input messages in FIG. 3 are output at the same time and mixed with each other. FIG. 5 is an example of a result obtained by disassembling mixed messages in FIG. 4 into element messages that are inseparable message units. It is a flowchart in representative embodiment of this invention. It is a figure which shows the example of the content of the message storage area | region described in FIG. It is an image figure of the process which the message extraction part 40 shown in FIG. 6 and the message reconstruction part 50 perform.

  FIG. 1 is an example of a hardware configuration of a message processing apparatus according to the present invention, and a computer 10 having an arithmetic unit 11 that executes at least a software program and a storage unit 12 that stores the program and data necessary for the execution. Implemented above. Further, the communication device 13 may be provided and connected to the communication device 21 provided in the external monitoring terminal 20 to transmit the message processing result. The monitoring terminal 20 can be replaced with, for example, another computer that configures a dual system. For example, the monitoring terminal 20 may transmit a message processing result to perform master-slave switching. The display 14 is used for outputting message processing results, and the keyboard 15 is used for changing settings.

  Here, an embodiment will be described by taking as an example a case where the present invention is applied to a message output by a kernel in a Linux (registered trademark) OS.

  FIG. 3 and FIG. 4 are examples of outputting messages regarding errors occurring on the SCSI disk at the same time in the three contexts of processes P1, P2 and P3. In particular, FIG. 3 shows a series of messages output by each process, and FIG. 4 shows an example of the result of mixing the message groups of FIG. For example, the message output by the process P2 indicates that the Read (10) command issued to the SCSI disk identified by the name sdb has timed out. However, as shown in FIG. 4, when mixed with a series of similar messages, the original message becomes unclear. For example, it is the SCSI disk sdb that originally timed out, but in the example of FIG. 4, it appears as if it occurred in sdc. Note that the examples shown in FIGS. 3 and 4 are slightly simplified from the actually output messages for the sake of explanation.

  FIG. 2 is a functional block diagram according to one embodiment of the present invention. The input interface 30 is an interface for the message extraction unit 40 to read a message. When reading is performed on this interface, messages output so far can be obtained in chronological order. In the Linux (registered trademark) OS, a / proc / kmsg pseudo file or the like corresponds to the input interface 30.

  The message extraction unit 40 reads a message from the input interface 30 according to the procedure described in FIG. Further, the message extraction unit 40 extracts element messages that are inseparable message units from the messages in the message storage area 70. The inseparable message unit is, for example, a message output unit predetermined by another system or device that outputs a message received by the input interface 30.

  FIG. 5 shows the result of extracting element messages for the examples of FIGS. 3 and 4, meaning that each row is one element message. The type column in FIG. 5 is a name given for convenience for each type of element message, and will be referred to in later description. For example, the message X "sd 4: 0: 0: 0: [sdc] CDB: Read (10)" is actually "sd 4: 0: 0: 0: [sdc]", "CDB:" Although “Read (10)” is output in three steps, the messages output at a time are inseparable, and the three messages are element messages. The extraction of the element message can be easily realized by a well-known pattern matching technique such as regular expression matching.

  The extracted element message is managed in the element message table 80. Specifically, for example, management is performed by arranging pointers that refer to element message character strings in the input order of the original message. For example, in the case of the message X described above, it corresponds to the element messages “sd 4: 0: 0: 0: [sdc]”, “CDB:”, “Read (10)” extracted from the message X. The types are recognized as Sdev, CDB, and Cmd, respectively, and by storing the order, the extraction order of element messages from messages and the reconstruction order of messages from element messages are managed.

  The message reconstruction unit 50 selects an element message string that matches a predetermined element message appearance pattern, and sends the series of element message strings to the output interface 60 as a series of originally output messages. Here, the information to be sent to the output interface 60 does not necessarily need to be a complete reproduction of the original series of messages, and any information can be used as long as it can uniquely identify the contents of the original series of messages. For example, taking the message X above as an example, the element messages “sd 4: 0: 0: 0: [sdc]”, “CDB:”, and “Read (10)” included in the message are extracted in this order. The types Sdev, CDB, and Cmd corresponding to each element message are sequentially extracted, and a series of extracted element message strings are output to the output interface 60.

  The output interface 60 performs post-processing of a series of messages reconstructed by the message restructuring unit 50, and various forms are conceivable. For example, there are a display for displaying a series of reconstructed messages, a communication device 13 for notifying an external monitoring terminal, and an error handler call for performing automatic recovery processing.

  FIG. 6 shows an example of an operation flow of the message extraction unit 40 and the message reconstruction unit 50, and FIG. 7 shows details of how the message extraction unit 40 stores the input message in the message recording area 70.

The operation of the message extraction unit 40 based on FIG. 6 is as follows.
Step 400: The message extraction unit 40 waits until a new message arrives at the input interface 30.

  Step 401: The message extraction unit 40 reads all messages that can be read at this time from the input interface 30 and stores them in the message storage area 70. The area used at this time is the first half 71, and the time is t1 (FIG. 7).

  Step 402: The message extraction unit 40 waits for a predetermined time T to elapse. Here, T is set to a sufficiently large value with respect to the time required for outputting a series of messages. For example, in the example of the SCSI error message described above, if the time required for one context to output a series of error messages is several tens of microseconds, it is sufficient to set T to about 1 millisecond. Note that the value of the predetermined time T may be changed by inputting a setting file describing the time T to the setting input unit 90.

  Step 403: The message extraction unit 40 reads all messages that can be read at this time from the input interface 30, and stores them immediately after the first half portion 71 in the message storage area 70. The area used at this time is the second half 72, and the time is t2 (FIG. 7). If no new message is input, the latter half 72 is empty.

  Step 404: Using a well-known pattern matching technique, the message extraction unit 40 extracts element messages from the messages stored in the entire storage area 70 including the first half portion 71 and the second half portion 72, and extracts the extraction results. Store in the element message table 80. For example, as means for extracting the element messages “Sense Key: Recovered Error” and “Sense Key: Medium Error” in FIG. 5, pattern matching using the regular expression “Sense Key:. +” Can be used.

  Step 405: The message extraction unit 40 sends a notification to the message reconstruction unit 50, and starts or resumes the processing of the message reconstruction unit 50.

  Step 406: The message extraction unit 40 waits until it receives a processing completion notification 505 from the message reconstruction unit 50.

  Step 407: When the message extraction unit 40 cannot read any message in the message reading at Step 403 (no new message has been input for T seconds), the process proceeds to Step 408. If even one message has been read, the process returns to step 402.

The operation of the message reconstruction unit 50 based on FIG. 6 is as follows.
Step 500: The message reconstruction unit 50 receives from the message extraction unit 40 a notification 405 that the storage of the input message in the message storage area 70 and the storage of the extracted element message in the element message table 80 are completed. Wait for.

  Step 501: The message reconstruction unit 50 selects an element message string that matches a preset element message appearance pattern. 3-5, the element message string when a media error occurs on the SCSI disk is Sdev, Res, Sdev, SenseME, Sdev, CDB, Cmd order. Such an appearance pattern of the element message can be determined by checking a result of a prior error injection test and an error message output source code. In addition, selection of an element message string that matches the preset element message appearance pattern can be realized by pattern matching using a well-known finite automaton.

  Here, in FIG. 5, one of the element message strings that match the appearance pattern of the element message is “sd 3: 0: 0: 0: [sdb]”, “Result: driverbyte = DRIVER_TIMEOUT”, “sd 2 : 0: 0: 0: [sda] "," Sense Key: Medium Error "," CDB: "," Read (10) ". Another one is "sd 4: 0: 0: 0: [sdc]", "Result: driverbyte = DRIVER_SENSE", "sd 4: 0: 0: 0: [sdc]", "Sense Key: Medium “Error”, “CDB:”, “Read (10)”.

  As shown in the column of process P3 in FIG. 3, a series of actually output messages corresponds to the latter element message sequence, but an element message sequence different from the actual message output may be selected as in the former case. is there. That is, the selection process has a false positive. It is possible to reduce false positive selection results by imposing certain rules and excluding selection results that do not match the rules. However, since the method depends on the specific contents of the target message, no further details will be mentioned here.

  Step 502: The message reconstruction unit 50 reconstructs a message corresponding to the content of the element message string selected in Step 501. This may be a simple concatenation of the element message strings, or may carefully convert information contained in the element message string and convert it into a simpler message. For example, the message reconstruction unit 50 concatenates the latter element message sequences in order, and reconstructs them as input messages.

  Step 503: The message reconstruction unit 50 performs error handling according to the reconstructed message. For example, it is conceivable to reset a component in which an error has occurred, notify the reconstructed message to the log file, the display 14, or the external monitoring terminal 20, or perform master-slave switching.

  If a plurality of element message strings can be selected in step 501, steps 501 to 503 are repeated by that number.

  Step 504: The message reconstruction unit 50 deletes the message for which processing has been completed from the storage area 70 and the element message table 80. Specifically, the first half portion 71 storing messages read from time t0 to t1 and the element message extracted from the first half portion 71 are deleted. On the other hand, the message of the latter half 72 read from time t1 to t2 and the element message extracted from the latter half 72 are not deleted at this time. The reason is as follows.

  The message included in the second half portion 72 may include the first half portion of a series of messages whose output is started immediately before time t1. In this case, since the element message sequence corresponding to the series of messages is not included in the element message table 80 in a complete form at this time, the series of messages is not selected in step 501. If the second half portion 72 is also deleted in step 502, only the second half portion of the series of messages remains in the message storage area 70 and the element message table 80 when the next message is read. Also at this time, in step 501, the series of messages is not selected. This means that the message to be error-handled at step 503 is lost. In order to prevent this, the contents of the latter half 72 are not deleted but are left as the first half 73 in the next processing.

  Step 505: The message reconstruction unit 50 sends a notification to the message extraction unit 40 to accept the next message. The message extraction unit 40 that has received this notification returns to step 400 and repeats the same processing.

  FIG. 8 is an image diagram of processing performed by the message extraction unit 40 and the message reconstruction unit 50 shown in FIG. As shown in the upper part of FIG. 8, the message extraction unit 40 sets the time from t0 to t1 and from t1 to t2 to be sufficiently longer than the output time of a series of messages. At this time, a series of messages input from t0 to t1 includes a case (A) including the entire message and a case (B) including only the first half of the message. In this case, if extraction of element messages and a series of message reconstruction processes are performed with the contents of the buffer read from t0 to t1, (B) cannot be correctly reconstructed. However, when these processes are executed with the contents of the buffer read from t0 to t2, the above reconstruction (B) is possible. In this case, since (D) contains only the first half of the message, it cannot be reconstructed correctly.However, (D) can be reconstructed correctly by performing the same process at the next timing. Can do.

  In other words, as shown in the lower part of FIG. 8, if processing is performed on the message stored in the buffer read between t0 and t2, (A), (B), and (C) can be correctly reconstructed. (D) cannot be reconstructed correctly. On the other hand, if a message stored in the buffer read between t1 and t3 is processed, (B) cannot be reconstructed correctly, but (C) and (D) can be reconstructed correctly. That is, for the message stored in the buffer read between tn and tn + 2, the reconstruction process is performed on a series of messages including the element message included in the message at the head of the range of tn to tn + 1. By doing so, the series of messages can be correctly reconstructed at any timing.

  In this way, in this process, the message extraction unit 40 and the message reconstruction unit 50 are the first part (for example, the first half) that stores the message read within a certain time in the message storage area 70. A portion 71), a message stored in a second portion (for example, the second half portion 72) that stores a message read in the next fixed time, and a message read in the next fixed time Of the messages stored in the third part (for example, the next first half part 71) which is the part to be stored, the message including the first element message stored in the first half part 71 and the second half part 72 first. (For example, (A) and (B) at the bottom of FIG. 8) are extracted and reconstructed. Subsequently, (C) and (D) which are stored in the second half 72 and the next first half 71 and do not include the first element message at the first timing are extracted and reconstructed.

  In other words, in this process, the range to be processed for reconstructing the message stored in the message storage area 70 is regarded as a window, and the window is shifted at the timing of reading the message, and is in the window to be processed. By repeating the process of sequentially extracting and reconstructing messages, it is possible to reconstruct messages correctly while suppressing the memory size.

  In this method, the memory size particularly required is the message storage area 70 and the element message table 80. Hereafter, the approximate method of the size is shown.

  Since the minimum required size of the message storage area 70 is the sum of the first half 71 and the second half 72 in FIG. 7, it can be estimated as 2TM from the predetermined waiting time T and the maximum message flow rate M. The maximum message flow rate M can be estimated as follows in the case of an example of a message output from the above-described Linux (registered trademark) kernel, for example. Among messages output by the Linux (registered trademark) kernel, a flow rate restriction is applied to messages that may be output in large quantities. The limit is 10 messages per 5 seconds by default. Since there are cases where this restriction is applied to each message type, it can be approximated by multiplying that amount as a coefficient and <factor> × <average message length> × 10/5. When a message is input through a network, the network bandwidth can be used as a flow rate.

  The minimum required memory size of the element message table 80 is a value proportional to the maximum number of element messages that can be included in the message storage area 70. Therefore, it can be easily obtained from the minimum size of the element message to be extracted and the size of the message storage area 70.

  As described above, in a situation where the flow rate of the input message is limited, the necessary memory size can be fixed to a finite size, and it becomes possible to prevent processing delay due to dynamic memory reservation and memory exhaustion.

  In this embodiment, in the first embodiment, the message storage area 70 and the element message table 80 are enlarged and handled in the form of a ring buffer, so that the processing of the message extraction section 40 and the message reconstruction section 50 operates asynchronously. It is possible. This reduces the risk of message overflow (message loss) due to message reading by the message extraction unit 40 not being performed during processing by the message reconstruction unit 50.

  Specifically, first, in step 406 of FIG. 6, the message extraction unit 40 is changed so as not to wait for notification until the free space in the message storage area 70 is insufficient. Further, in steps 401 and 403, the message extraction unit 40 adds the message read in the step to the message storage area 70. At this time, as in the first half portion 71 and the second half portion 72 in FIG. 7, there is no change in the point of storing while conscious of the range of the message input in the step. The handling of the element message table is the same as that of the message storage area 70.

  On the other hand, in step 500, when two or more parts (for example, the first half part 71 and the second half part 72) remain in the message storage area 70, the message reconstruction unit 50 changes so as not to wait. In step 501, the above-described pattern matching process is performed on the element message string extracted from the oldest part (first half part 71) and the next part (second half part 72) in the message storage area 70. To do. For example, if the part deleted at the most recent step 504 is the part stored at time tn (the part immediately before the first half part 71), the message reconstructing unit 50 sets the time tn + 1 (first half part 71). ) And tn + 2 (second half part 72).

  In this embodiment, instead of the message extracting unit 40 waiting for a predetermined time and reading the message, the message extracting unit 40 reads the message at any time and assigns the time as a time stamp to each message. If a time stamp has been added to the message in advance, there is no need to add a time stamp again.

  Specifically, in steps 401 to 403 in FIG. 6, the message extraction unit 40 continues to read the message for 2T time (T is the above-mentioned predetermined waiting time), and adds a time stamp of the reading time to each message. Then, the message storage area 70 is changed. Further, in step 501, the message reconstructing unit 50, for the messages stored in the message storage area 70 within the above-mentioned 2T time, the message belonging to the first T time is the first half portion 71, the message belonging to the next T time. As the second half portion 72, the subsequent processing is changed in the same manner as in the first embodiment.

  The present embodiment is a modified example in which, instead of waiting for a predetermined time in the first embodiment, the completion of outputting a series of messages being output is waited until a certain break position or timing arrives. For example, in the case of a series of messages output by the above-mentioned Linux (registered trademark) kernel, basically a series of messages are output continuously, and there is no point during which process switching is performed (however, in a preemptive kernel) Except in some cases). Therefore, the message extraction unit 40 confirms that a process switch is performed on another system or device that outputs a message received by the input interface 30 at least once for all units that can be executed in parallel on the computing device 11. By waiting, it waits for the completion of the output of a series of messages being output indirectly, and for the messages output between one process switch and the next process switch, element messages are extracted and the series of messages are reconstructed be able to. In other words, the message extraction unit 40 waits for a message to be output until a certain delimiter position or timing such as a process switch arrives at the output side of multiple contexts, and targets messages output during that delimiter position or timing Can be broken down into element messages.

  In the present embodiment, in step 501 of FIG. 6 of the first embodiment, a series of element messages including the first element message in the latter half portion 72 of the message storage area 70 is changed so as not to be selected. . As a result, the series of element messages is selected only when the second half 72 becomes the first half 73 (FIG. 7) in the next lap. That is, the message reconstructing unit 50 selects the first element message obtained from the message that has not been deleted from the message storage area 70 that includes the first element message in the second half portion 72 from the selected element messages. In other words, if the series of element messages from the beginning to the end are completely included in the latter half 72 at a certain timing, the reconstruction process (step 502) and the error handling process (step 503) must be performed twice. Can be prevented.

30 ... input interface, 40 ... message extraction unit, 50 ... message reconstruction unit, 60 ... output interface, 70 ... message storage area, 80 ... element message table.

Claims (7)

A storage unit that sequentially stores the messages received within a predetermined time from an input interface that sequentially receives messages output from a plurality of contexts;
A message extraction unit that decomposes the message stored in the storage unit into element messages that are predetermined message output units on the output side of the plurality of contexts , and the input is performed within a predetermined time period A message received from the interface is added to and stored in a message that has not been deleted from the storage unit, and a newly stored message including a message that has not been deleted from the storage unit and a message that has been added and stored The message extractor for decomposing into element messages ;
A message reconstruction unit that selects an element message that matches a predetermined occurrence pattern of the element message, reconstructs the message , and stores the message in the storage unit after reconstructing the message from the element message Delete the reconstructed message from the stored messages, select the element message that matches the appearance pattern from the element messages obtained from the newly stored message, and reconstruct the message Said message restructuring unit to
A message processing apparatus comprising: The message processing device according to claim 1,
The message extraction unit is a first part that stores a message received at the predetermined time in the storage unit and a second part that stores a message received within the next predetermined time. A first extraction process in which the message stored in the part is a processing target range, a second part and a third part which is a part for storing a message read in the next predetermined time, And the second extraction processing as the message processing target range stored in
The message reconstructing unit reconstructs the message by selecting an element message that matches the appearance pattern from among the element messages obtained in each extraction process.
A message processing device. The message processing device according to claim 1,
In addition, a setting input / output unit that accepts a change in the predetermined time,
The message extraction unit decomposes the message stored in the storage unit into the element message within a changed time;
A message processing device. The message processing device according to claim 1 ,
The message restructuring unit selects a head element message obtained from the message not deleted from the selected element messages.
A message processing device. The message processing device according to claim 1,
The message extraction unit waits for the output of a message until a certain delimiter position or timing arrives on the output side of the plurality of contexts, and disassembles the message output during the delimiter position or timing into the element message ,
A message processing device. A reception step for sequentially receiving messages output from a plurality of contexts;
The message received within a predetermined time is received from the input interface within the next predetermined time in the extraction step of decomposing the message into element messages that are predetermined message output units on the output side of the plurality of contexts A newly stored message including a storing step for storing a message in addition to a message that has not been deleted from the storage unit, a message that has not been deleted from the storage unit, and a message that has been added and stored, and the element message And a decomposition step that decomposes into
The element message that matches a predetermined occurrence pattern of the element message is selected, and in the reconstruction step for reconstructing the message, the message is reconstructed from the element message and then stored in the storage unit. A deletion step of deleting a reconstructed message of the messages, selecting an element message that matches the occurrence pattern from among the element messages obtained from the newly stored message, and re-storing the message To construct,
Message processing wherein a call. The message processing method according to claim 6 ,
In the extraction step, a first part which is a part for storing a message received at the predetermined time and a second part which is a part for storing a message received within the predetermined time next to the storage part Are stored in the first extraction process in which the message stored in the process range is processed, and in the second part and the third part which is a part for storing the message read within the next predetermined time. The second extraction process as the message processing target range that has been performed,
In the reconstruction step, the element message that matches the appearance pattern is selected from the element messages obtained in each extraction process, and the message is reconstructed.
A message processing method.

Images