JP5042323B2 - Distributed resource monitoring system, method and apparatus - Google Patents

Description

  The present invention relates to technology for monitoring performance and faults of information systems that are applied to social infrastructures, especially systems that use wide-area distributed resources.

  Services using information and communication technology (ICT) such as electronic money, railway control, and power control have become widespread and have become part of the social infrastructure. An ICT that supports a social infrastructure needs a highly reliable and low-latency response to ever-increasing data and processing requirements. Furthermore, due to recent social and global demands, ICT is also required to improve energy efficiency. In addition, as long as it is a system that supports social infrastructure, it is indispensable to monitor the operation status and performance including faults.

  There are the following three conventional techniques for monitoring the operating status (including faults) and performance of a conventional ICT system. These are monitoring technologies mainly for data centers and information systems in enterprises.

  First, as a first prior art, a failure monitoring of whether or not an information system is operating normally by performing virtual use on an information system to be monitored by an access means equivalent to a general user, There is also a method that enables performance monitoring by response time.

  As a second conventional technique, there is a method of monitoring the resource usage status for individual servers, routers, and storages constituting an information system to be monitored. This method enables performance monitoring and fault monitoring of information systems by statistically collecting performance-related data of each device.

  Furthermore, as a third prior art, there is a method of using a stress test program that simulates a large number of users when performing virtual use. As a result, the load resistance performance of the information system can be monitored.

  Examples of the prior art documents as described above include Patent Documents 1, 2, 3, and the like.

JP 2005-506605 A JP 2003-283565 A JP 2000-315198 A

  In the first prior art, since monitoring based on virtual use from the outside is performed, detailed performance information cannot be obtained regarding the internal operation of the information system. Furthermore, even when a failure occurs, it cannot be determined in which part of the information system the failure has occurred. Also, in a wide-area distributed and hierarchical information system, a request depending on the position of the information processing request source must be generated, which is difficult to realize. In addition, since the independence from the actual service is not maintained, in the case of monitoring with faults, the actual service must be stopped in view of the impact on society.

  In the second prior art, detailed statistical performance information for each device can be acquired, and the location of the failure can be specified and the operation can be estimated to some extent. The association is unknown and the "estimated" area cannot be left.

  The third conventional technique only determines the load resistance of the information system and cannot be used for fault monitoring. Moreover, in the information processing system that supports the social infrastructure, there is a possibility that the social infrastructure may be paralyzed if the load resistance confirmation is performed, so that it is highly possible that the information infrastructure is not feasible.

  An object of the present invention is to provide a distributed resource monitoring system that measures the operating status and performance of distributed information communication devices without affecting the actual business operations.

  Another object of the present invention is the ability to specify an accurate part when a failure occurs while obtaining detailed performance information of an information system distributed over a wide area without changing the application or data for realizing the service. It is an object to provide a fault monitoring method and apparatus.

  In order to achieve the above object, in the present invention, a distributed resource monitoring system in which a distributed resource comprising a management server and a plurality of processing nodes performs hierarchical information processing via a network and provides a desired service. The plurality of processing nodes includes a production application that performs information processing and a pseudo application that simulates the operation of the production application. When a pseudo packet corresponding to a service is received, the pseudo application orders the pseudo packet in order. The information is added and transmitted to another processing node, and the management server acquires operation information when each of the pseudo applications of the plurality of processing nodes transmits / receives the pseudo packet, and the order information and the operation information of the acquired pseudo packet Distributed resource monitoring system that estimates the fault location on the network by evaluating To provide.

  In order to achieve the above object, according to the present invention, a distributed resource composed of a plurality of processing nodes performs distributed information processing via a network to estimate a fault occurrence site in a system that provides a service. A resource monitoring method, in which a production application that performs information processing is set in a plurality of processing nodes, and a pseudo application that simulates the operation of the production application is set, and the processing node sends a pseudo packet corresponding to a service provided via the network. Received, after the pseudo-application is simulated, attaches order information to the pseudo-packet and sequentially sends it to other processing nodes. The management manager operates when each pseudo-application of the plurality of processing nodes transmits / receives the pseudo-packet. Information, and based on the obtained pseudo packet order information By evaluating the information, it provides a distributed resource monitoring method for estimating the failure site on the network.

  Furthermore, in order to achieve the above object, according to the present invention, there is provided a distributed resource monitoring apparatus for a system in which a plurality of processing nodes perform hierarchical information processing via a network and provide a desired service. A plurality of processing nodes having a production application that performs information processing and a pseudo application that simulates the operation of the production application receive a pseudo packet corresponding to the service, and the pseudo application adds order information to the pseudo packet, Provided is a distributed resource monitoring apparatus that collects order information and operation information when transmitting to other processing nodes, and estimates a faulty part based on the obtained order information and operation information of a pseudo packet.

  That is, in order to achieve the above object, in the present invention, a pseudo application is arranged on the same resource as the production application in a processing node arranged in a wide area to perform hierarchical information processing. By simulating the flow of processing, it becomes possible to identify a problematic processing path and a faulty part from detailed performance information.

  According to the present invention, it is possible to provide a distributed resource monitoring system that measures the operating status and performance of distributed information communication devices without affecting the actual business operations. In addition, it is possible to provide a performance / fault monitoring method and apparatus for obtaining detailed performance information of a distributed information system and specifying a fault occurrence site without changing the application or data for realizing the service. .

1 is a diagram illustrating a configuration example of a performance / fault monitoring system according to a first embodiment. FIG. It is a figure which shows an example of the information processing system to which the monitoring system is applied based on a 1st Example. It is the figure which showed typically the functional operation | movement of the performance and fault monitoring system based on 1st Example. It is a flowchart figure which shows the processing flow of the performance and fault monitoring system based on a 1st Example. It is a figure which shows the example of 1 structure of the pseudo AP of the processing server based on 1st Example. It is a figure which shows the example of 1 structure of the management manager of the management server based on a 1st Example. It is a figure which shows one structural example of the packet used for the performance and fault monitoring system based on 1st Example. It is a figure which shows the detailed flow of a process at the time of the pseudo packet reception of a pseudo application (Application; AP) based on a 1st Example. It is a figure for demonstrating the flow of the pseudo packet based on 1st Example. It is a figure which shows the processing flow of the management manager 141 based on a 1st Example. It is a figure which shows the other structural example of the actual information processing system to which the performance and fault monitoring system which concerns on a 1st Example is applied. It is a figure which shows the table of the variation | change of operation | movement of pseudo | simulation AP based on 1st Example. It is a figure which shows the table of the variation of the residence time of pseudo AP based on a 1st Example. It is a sequence diagram explaining the whole processing flow of the information processing system concerning a 1st Example. It is a sequence diagram explaining the whole processing flow of the information processing system concerning a 1st Example. It is a figure which shows an example of the specific structure of the information processing system concerning a 1st Example. It is a figure which shows the example of a table of the various operation information accumulate | stored in the operation information storage part concerning a 1st Example.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. In the present specification, hierarchical information processing refers to processing in which a plurality of applications (APs) realize a desired service in the real world. Preferably, the plurality of APs are executed by a plurality of processing nodes connected to the network, and are specifically referred to as production APs in this specification. In each service, production APs in a plurality of processing nodes cooperate in information processing based on information acquired in the real world, and process results are fed back to the real world.

  In addition, a pseudo AP on the same resource (a case, a blade, a central processing unit, a memory, etc.) simulates the operation of the production AP and specifies a route used by the service. Furthermore, the pseudo AP adds the time at the time of data transmission / reception to the transmission packet to the next processing node, and accumulates operation information around that time in the processing node. The management information in the management server collects the accumulated operation information of each processing node, and it is possible to follow the flow of a series of information processing of the service by associating it, and it is possible to identify the problem part at the time of failure occurrence Can be.

  In the first embodiment, a pseudo AP is arranged on the same resource as the real AP in a plurality of processing nodes arranged in a wide area to perform hierarchical information processing, and the process flow is simulated. This is a performance / fault monitoring system that identifies a processing path to be identified and identifies a faulty part from detailed performance information.

  FIG. 1 is a diagram showing an outline of the performance / fault monitoring system of the first embodiment. In the figure, reference numerals 11, 12, and 13 denote processing nodes distributed in a wide area, reference numeral 14 denotes a management server, and reference numerals 15, 16, and 17 denote networks such as public networks and intranets. Various services are provided by the production AP, which is a plurality of information processing APs, straddling the plurality of processing nodes 11, 12, 13 distributed in a wide area. In the processing nodes 11, 12, and 13, processing result transmission destinations are generated according to the respective information processing results.

  The processing node 11 includes a production AP 111, a pseudo AP 112, an operating system (OS) or an AP execution base 113, and hardware 114. The hardware 114 includes a memory (MM) 115 as a storage unit, an input / output (I / O) 116, a central processing unit (CPU) 117 as a processing unit, and a storage unit. A hard disk drive (HDD) 118 and an interface (I / F) 119 with a network are included. The hardware configurations of the other processing nodes 12 and 13 and the management server 14 are the same, and an example thereof will be shown later. A management manager 141 operates on the management server 14.

  In the performance / fault monitoring system configuration shown in FIG. 1, in the real service, the production APs in the processing nodes 11, 12, and 13 perform information processing based on information acquired in the real world, and the processing results are obtained in the real world. To give feedback. Further, a pseudo AP that simulates the operation of the production AP operates in the processing nodes 11, 12, and 13. To explain the processing node 11 as an example, the pseudo AP 112 on the same resource consisting of the hardware 114 simulates the operation of the production AP 111 and specifies the route used by the service. That is, the pseudo AP 112 in the present embodiment provides a function of collecting operation information while simulating the operation of the production AP 111 using the same resources as the production AP 111 that handles information processing.

  Further, the pseudo AP 112 adds the time at the time of data transmission / reception to the transmission packet to the next processing node, and accumulates operation information around that time in the processing node 11. The management manager 141 in the management server 14 collects all the accumulated operation information of the processing nodes 11, 12, and 13 and associates them so that it is possible to follow the flow of a series of information processing of the service, and a failure occurs. Enables identification of problem areas at times.

  Here, a specific service by the production AP in the present embodiment will be described with an example. Specific services of the production AP include, for example, public video monitoring and security services by companies. In such a monitoring service, the central monitoring room monitors the video of each base, but at most, it is at best to monitor the video in several places. Therefore, by performing hierarchical monitoring processing, the subject that is the monitoring target that caused the problematic behavior is automatically detected, and an alert is displayed on the display of the central monitoring room. The role of the management node 14 at this time is to detect a failure of the processing nodes 11, 12, 13 and the networks 15, 16, and 17 in advance and take measures to prevent the service from stopping.

  In this system configuration, it is not always necessary to transmit the video from each base to the central monitoring room by the hierarchization processing of the monitoring video. However, when a subject that has caused a problem behavior is detected, it is necessary to preferentially transmit the video to the central monitoring room and display it on the display of the monitoring room. At this time, in this system, for example, securing the network bandwidth when transmitting the video data acquired by the processing node 11 to the central monitoring room via the several processing nodes 12, 13 and the like, switching the priority of the transmission data, Network path arbitration with video data transmitted from other sites is required.

  Taking the above specific monitoring service as an example, the hierarchical processing function of this embodiment will be described in more detail. In the conventional system, all the camera images of the processing node 11 are transmitted to the central monitoring room, but in the performance / fault monitoring system configuration of the present embodiment, the processing is performed by processing nodes distributed in the middle. This makes it possible to transmit only the necessary video data to the central monitoring room. That is, the network bandwidth is not wasted, and more bases can be monitored with the same network bandwidth as before.

  An example of the processing flow of this system for performing the monitoring service is as follows.

  (1) The processing node 11 includes, for example, a Web camera and a human sensor, and an AP that transmits human sensor information and camera video data as a packet.

  (2) The processing node 12 is equipped with a detection AP for detecting the problem behavior of the subject from the camera video data.

  (3) The processing node 11 detects that a person has come close to the sensor, and starts transmitting camera video data to the processing node 12.

  (4) The video acquired by the problem behavior detection AP of the processing node 12 is processed, and the video determined to be the problem behavior is transmitted to the central monitoring room. If it is not determined that the behavior is problematic, the video is terminated at the processing node 12 and is not transmitted to the central monitoring room. By such a hierarchization process, the network bandwidth is not wasted, and more bases can be monitored with the same network bandwidth as before. An actual information processing system to which this embodiment is applied can take various configurations.

  FIG. 2 shows a configuration example of an actual information processing system to which the performance / fault monitoring system of this embodiment is applied. In the figure, reference numerals 21 to 28 denote processing nodes corresponding to the processing nodes 11, 12, 13 and the like in FIG. Reference numeral 29 denotes a network such as a public network or an intranet. Among the processing nodes 21 to 28, some of the processing nodes 21, 22, 23, 24, etc. are sensors 31, 32 and actuators 33, 34, which function as various communication terminals existing in the real world 30 as described above. Alternatively, sensing information and video data are received from a web camera or the like, and control information is transmitted.

  FIG. 11 shows another configuration example of an actual information processing system to which the performance / fault monitoring system of this embodiment is applied. In the figure, among the processing nodes 1101, 1102, 1103 and 1104, the processing nodes 1101 and 1102 are intelligent nodes capable of information processing, and the processing nodes 1103 and 1104 are capable of information processing connected to the wide area network 1105. A server such as a data center. Various communication terminals 1109 such as sensors and cameras are connected to local networks 1107 and 1108 to which the processing nodes 1101 and 1102 are connected. Further, an edge node 1210 having an information filtering function is connected to the local network 1207, and sensing information is sucked up between the plurality of sensors 1111 and 1112 and the actuator 1113 which are communication terminals via the edge node 1110. And transmission of control information. In such an information processing system, the edge node 1110 and the processing nodes 1101 to 1104 correspond to the processing nodes 11, 12, and 13 in FIG. 1, and the management server 1106 corresponds to the management server 14.

  Subsequently, based on FIG. 3 which is a diagram schematically showing the functional operation of the performance / failure monitoring system of the present embodiment, information processing for providing services by cooperation of information processing by the processing nodes 11, 12, and 13 of the present embodiment. The system performance / fault monitoring method will be described in detail.

  FIG. 3 shows a system configuration for simulating the flow of processing by arranging a pseudo AP on the same resource as a production AP in the plurality of processing nodes 11, 12, and 13 that are distributed as described in FIG. Show. The hardware configurations of the processing nodes 11, 12, and 13 have the same configuration as the hardware configuration of the processing node 11 shown in FIG. 1, but only the main elements are shown in a simplified manner in FIG. The management server 14 is also not shown.

  In FIG. 3, a packet at the time of production processing is input from a terminal (Address; Addr = D) of address D in the real world, and the result of information processing by the production APs 111, 121, 131 at the processing nodes 11, 12, 13 The real packet is sent to a real-world terminal (address = D). Similarly, a pseudo packet is input from the terminal (Addr = D), and the pseudo packet is sent to the terminal (Addr = D) based on the result of the simulation processing by the pseudo APs 112, 122, 132 in the processing nodes 11, 12, and 13. The

  As shown in FIG. 3, the transmission destination address B of the production AP 111 is obtained from the user or administrator who is the owner of the production AP 111 together with the residence time (0.4) to the pseudo AP 112 in the processing node 11. Implemented. Similarly, the destination addresses C and D of the production APs 121 and 131 which are information processing of the processing nodes 12 and 13 are stored in the pseudo APs 122 and 132 together with the respective residence times (0.1) and (0.2). Is set. In addition, instead of obtaining data such as destination address and dwell time from the user, each processing server monitors the production AP for a certain period of time as described later, and estimates the dwell time and its destination. Also good.

  FIG. 4 shows a schematic flowchart for explaining the entire processing sequence of this embodiment. In FIG. 4, when the processing flow for performance / fault monitoring starts, first, the management manager 141 sets a pseudo AP for each of the processing nodes 11, 12, and 13 (step 41; hereinafter, steps are omitted in parentheses). Subsequently, the corresponding client terminal in the real world transmits a pseudo packet described later (42). Each processing node gives information processing order information that can identify that information processing has been hierarchically performed by the pseudo AP when the corresponding pseudo packet is transmitted / received to the pseudo packet (43).

  The pseudo AP acquires operation information of the processing node at the time of transmitting and receiving the pseudo packet (44). The processing node that has received the pseudo packet accumulates the pseudo packet (45). The management manager 141 acquires operation information when each pseudo AP transmits and receives a pseudo packet (46). The management manager 141 acquires the operation information of the processing node based on the information processing order information of the pseudo packet received by the client terminal (47). The management manager 141 evaluates the acquired operation information and estimates a faulty part on the network (48).

  In the performance / failure monitoring processing sequence of this embodiment, the client terminal transmits a pseudo packet in step 42. This pseudo packet evaluates the quality of service provided to the client terminal and identifies the location where the quality is a problem. It provides information to do. This pseudo packet makes it possible to clarify the breakdown of communication time and information processing time, and to investigate processing nodes and networks in which a delay occurs. Therefore, it functions as a tool for investigating whether there is a problem in the platform on which the service operates rather than investigating the problem of the service itself.

  Further, this pseudo packet goes through information processing on a plurality of processing nodes, so that the node identifier, arrival time, and transmission time are added to the payload, and the pseudo packet that has flowed end-to-end is managed by the management node 14. Are collected and checked against the network topology of the system possessed by the management manager 141 and the staying location causing the delay.

  In the performance / failure monitoring system of this embodiment, End-to-End exists in the real world, and the client terminal and communication terminal that use the service are the End part. In other words, End-to-End assumes a section from when a client terminal transmits a request to a service until a response returns. For this reason, the sending / receiving source of the pseudo packet arranges a terminal which is an information processing apparatus that sends / receives the pseudo packet in the real world. Alternatively, there is a method of inputting a pseudo packet to the application on the processing node from the management middleware of the next processing node.

  The former can be verified in the same situation as when using the actual service, and more accurate verification is possible. However, at the time of verification, it is necessary to go to the place where the processing node 11 exists and input a pseudo packet. . On the other hand, in the latter case, a mechanism for sending / receiving a pseudo packet is arranged in the verification processing node, but verification is easy because all operations end with the management node. In this case, the pseudo packet to be transmitted is deployed at the same time as the deployment of the pseudo AP (41). In addition to executing various settings and installing a pseudo application, a pseudo packet is registered in the management middleware of the processing node 11. The management middleware is software having an interface with the management node 14 in each processing node. The pseudo AP is waiting for input in response to a service start instruction from the management node 14 and is registered in the management middleware of the processing node 11. A process of flowing a packet into the pseudo AP as input data is executed.

  FIG. 5 shows an embodiment of the pseudo AP in the performance / fault monitoring system of the first embodiment. In the figure, reference numerals 51, 52, and 53 denote an operation information storage unit, a pseudo AP, and a pseudo packet storage unit, respectively. The operation information storage unit 51 and the pseudo packet storage unit 53 are formed in the hardware storage unit described above. The pseudo AP 112 has functions of a packet arrival time collection unit 54, an operation information collection unit 55, a production AP operation simulation unit 56, a packet transmission time collection unit 57, a collection data holding unit 58, a management manager IF unit 59, and a profile information addition unit 60. Consists of blocks. These functional blocks are composed of programs executed by a hardware processing unit.

  FIG. 6 shows an embodiment of the management manager in the performance / fault monitoring system of the first embodiment. In the figure, reference numerals 61 and 62 denote a collection data storage unit and a management manager, respectively. The management manager 62 includes a collection data collection unit 63, a pseudo API / F unit 64, a pseudo AP control unit 65, and a collection data association processing unit 66. Consists of functional blocks. Details of this functional block will be described later.

  FIG. 7 shows an example of a packet configuration used in the information processing system of this embodiment. In FIG. 7, 71 indicates a communication header portion, and 72 indicates a payload portion. In the payload portion 72, information processing order information # 1 --- # n is sequentially recorded. The information processing order information 73 includes node identification information 74, service identification information 75, application identification information 76, reception time information 77, and transmission time information 78.

  Now, referring back to FIG. 5, the production AP operation simulation unit 56 of the pseudo AP 52 does not affect the production AP 111 by sleeping for a typical processing time (such as no load time) of information processing by the production AP 111. Consume and simulate in form. The time is specified as a simulated load via the management server IF unit 59. That is, the production AP operation simulation unit 56 is instructed by the management manager 14 to specify the transmission destination of the pseudo AP 52 and the simulated load via the management manager IF unit 59. This destination specification is fixed destination, destination distribution with a certain probability, condition specification type multiple destination allocation (memory usage exceeded the threshold, etc.), multiple transmission (fixed destination), multiple transmission (destination Change at a certain probability). In addition, the simulated load includes a sleep processing time, a sleep time that changes with a certain probability, and an operational status of the production AP (real time, non-real time, etc.).

  The packet arrival time collection unit 54 functions so that the collection data holding unit 58 holds the time when a packet arrives at a server that is a processing node that performs information processing. Further, the packet transmission time collection unit 57 uses the CPU, memory, IO usage rate or operation information (including fault information) at the time of arrival, time consumption in the real AP operation simulation unit 56, and then passes through the management manager IF unit 59. The collection data holding unit 58 functions to hold the time when the processing request (packet) is transmitted to the designated transmission destination. When there are a plurality of designated transmission destinations, the transmission destination is determined based on a probability based on the actual operation.

  The collected data holding unit 58 of the simulated AP 52 continues to record the collected data 54 and 57 in a wrap around manner, and functions to stop the wrap around when an event such as a failure occurs. The event at which collection is to be stopped is specified via the management manager IF unit 59. As will be described later, the profile information adding unit 60 has a function of adding the identifier of the processing node, the reception time information temporarily stored, and the transmission time information to the payload portion 72 of the received pseudo packet.

  The pseudo packet storage unit 53 holds the received pseudo packet together with its reception time information and transmission time information. The transmission time information can be configured to be stored directly from the packet transmission time collection unit 57. At the same time, pseudo packet information is transmitted to the management manager 14 via the management manager IF unit 59. The pseudo packet information is used by the management manager 14 as information for investigating which path, that is, via which processing node the service is provided. In addition, the time difference between the management manager 14 and each processing node is corrected, and this is used as information for investigating a delay portion of the network.

  The operation information accumulation unit 51 accumulates hardware operation information and network statistical information from the timing when the packet arrival time collection unit 54 receives the pseudo packet to an arbitrary time. The collection is performed by the operation information collection unit 55. The hardware operation information assumes, for example, CPU, memory, IO usage rate, failure information, and the like. Further, statistical information such as MIB (Management Information Base) defined in RFC1213 can be used as the network statistical information.

  Reference numerals 1601 to 1604 in FIG. 16 indicate examples of tables of server operation information, process operation information, network operation information, and storage operation information that are stored in the operation information storage unit 51 in the HDD 118 of the processing node 11, respectively. As shown in the figure, the server operation information 1601 shows various information about the server operation, the process operation information 1602 shows various information about the process operation, the network operation information 1603 shows various information about the network operation, and the storage operation. The information 1604 accumulates various types of information indicating storage operation as a storage unit.

  FIG. 8 shows a detailed flow of processing when the pseudo AP 52 receives a pseudo packet. This detailed flow corresponds to steps 43 to 45 of the overall flow of this embodiment shown in FIG.

  When the pseudo AP 522 of the processing node receives the pseudo packet transmitted from the client terminal (801), the packet arrival time collection unit 54 obtains the current time from the time information acquisition device 127, and stores it as primary packet reception time information (802). ) The operation information collection unit 55 starts collecting the operation information of the processing node such as the CPU load factor, network throughput, memory usage rate, HDD usage amount (803).

  The real AP operation simulation unit 56 executes the simulation operation instruction transmitted from the management manager 141 via the management manager IF unit 59 before the pseudo packet is transmitted (804). Thereby, the simulation operation is executed (805) in the processing node 11. When the operation information stays in the pseudo AP 52 for the stay time set by the management manager 141 when the application is deployed, the operation information is transmitted as a pseudo packet to the destination IP address designated in advance by the management manager 141.

  The packet transmission time collection unit 57 acquires the current time from the time information acquisition device 127 and temporarily stores (806) the transmission time information in the pseudo AP 52. At the time of pseudo packet transmission, the packet transmission time information is output to a log, and the profile information adding unit 60 adds the identifier of the processing node, the temporarily stored reception time information, and transmission time information to the payload portion of the received pseudo packet ( 807) and transmit the pseudo packet to which the information is added (808).

  The operation content of how the real AP operation unit simulation unit 56 of the pseudo AP 52 described above monitors the real AP will be described.

  (1) The service administrator creates and specifies pseudo AP operation definition data.

  (2) Monitor the production AP and determine the probability distribution of the destination with respect to the source. The monitoring is performed by the management middleware that monitors the production AP, and the management manager 141 collects the monitoring results at regular intervals. The management manager 141 creates operation definition data (transmission source IP address, dwell time, transmission destination address 1, probability 1, transmission destination address 2, probability 2...) For each production AP from the collection result, and creates a pseudo AP 52. To the actual AP operation simulation unit 56.

  The operation definition data is received by each pseudo AP through the management manager IF unit 59 of the pseudo AP at the start of processing, and is registered in the actual AP operation simulation unit 56. When specified by the service manager, behavior behavior data is created and behavior behavior data is transmitted to each pseudo AP.

An example of the operation definition data format is as follows.
<Time = dwell time & source IP address number = 2 & source IP address 1 = xxx.xxx.xxx.xxx & source IP address 2 = yyy.yyy.yyy.yyy & destination IP address number = 2 & destination address 1 = zzz.zzz.zzz.zzz & probability 1 = 10 & destination address 2 = qqq.qqq.qqq.qqq & probability 2 = 90>
Here, the operation variation of the operation definition data is shown in the table 1201 of FIG. The variations of the operations are shown in the numbers 1 to 4, and the contents of the operations are as illustrated. Similarly, the variation of the residence time is shown in the table 1301 of FIG. Variations of residence time are shown in numbers 1 to 4, and the contents of the time are as shown in the figure.

  FIG. 9 shows how a pseudo packet that functions as a probe packet is transferred as a result of processing in each processing node 11, 12, 13 in this embodiment. In the figure, reference numerals 94, 95, 96, and 97 denote packets that are sequentially transferred. A packet 94 indicates a pseudo packet that is first transferred from the real world. The destination is described in the communication header portion 71 shown in FIG. 7, and information processing order information is added to the payload portion 72 every time it passes through the processing nodes 11, 12, and 13. For example, the pseudo packet 97 sent to the real-world terminal (address = D) describes all the information processing order information recorded by the processing nodes 11, 12, and 13.

  Next, the operation of the management manager 141 of this embodiment will be described based on the processing flow of the management manager 141 shown in FIG.

  In the figure, a collection data collection unit 63 of the management manager 62 collects data collected by the pseudo AP from each processing node via the pseudo API / F unit 64. The pseudo AP control unit 65 transmits the next destination address of the pseudo AP and the operation simulation information of the production AP to the pseudo AP. For example, a plurality of pseudo APs are distributed according to residence time and probability.

  First, the collection data collection unit 63 collects pseudo packet information and the like stored in the pseudo packet storage unit of the pseudo AP 52 of each processing node via the pseudo API / F unit 64 (1001). The collected pseudo packet information of each processing node is stored in the collection data storage unit 61 (1002), and the time information of the management node 14 and each processing node 11, 12, 13 is corrected (1003). The collection data association processing unit 66 calculates the service roundtrip time and the communication time between the pseudo APs based on the time information given to the pseudo packet and the time correction of each processing node. In addition, network operation information and hardware operation information at the same timing are associated. Then, the pseudo packet information acquired by the collection data association processing unit 66 is sorted in order of time information (1004), and the sorting result is presented on a display unit (not shown) (1005).

  The processing contents of the collection data association processing unit 66 in the management manager configuration described above are organized as follows.

  (1) First, the collection data collection unit uses the pseudo API / F unit 64 to acquire a log of pseudo packets from each pseudo AP. The pseudo packet log is acquired from the final arrival processing node of the pseudo packet.

  (2) The collection data collection unit 63 uses the pseudo API / F unit 64 to acquire various operation information and registers it in the collection data storage unit 61. The collection data association processing unit 66 is notified of the completion of data acquisition.

  (3) The collection data association processing unit 61 corrects the time of each processing node. This correction method can be corrected by using a GPS (Global Positioning System) described later.

  (4) Subsequently, the round trip time of the service is calculated. The round trip time of this service corresponds to the time from when a pseudo packet is input to the first pseudo AP, through several pseudo APs, to the final processing node, and until a log is output.

  (5) The processing time breakdown is calculated, and the communication time between the pseudo APs is calculated.

  (6) The network operation information around the time when the pseudo packet passes, and the hardware operation information at the peripheral time when the pseudo packet arrives and is transmitted are acquired from the collected data storage unit 61. At the same time, statistical information of the same data at the same operation information acquisition location is acquired from the collected data storage unit 61.

  (7) Breakdown of processing time and network operation information and hardware operation information acquired in (5) are allocated and displayed on the display unit with a graphical user interface (GUI).

  (8) The operation information (1) around the time when the pseudo packet passes is compared with the statistical information. When the distribution of residence time is obtained from statistical information, if (1) information does not fall within a certain range, it is displayed on the GUI as an error location.

  In addition, it is possible to realize time correction by periodically updating the time information of the node by installing the GPS in the management server and the processing node described above or having an I / F for acquiring time information from the GPS. it can. As a result, even when there are a plurality of processing nodes distributed in a wide area, the time difference of the time difference can be ignored by the same method.

  14A and 14B are sequence diagrams for explaining the entire processing flow in the information processing system in the first embodiment described in detail above. Following the process of FIG. 14A, the process of FIG. 14B is performed. In the figure, management manager 1401 and processing nodes 1402, 1403 and 1404 correspond to management manager 141 and processing nodes 11, 12 and 13, respectively.

  In FIG. 14A, as described above with reference to FIG. 4, the management manager 1401 first registers a pseudo packet in the processing node 1402 (1405 to 1407). Subsequently, the management manager 1401 issues a service start instruction (1408, 1411, 1414) to each of the processing nodes 1402, 1403, 1404, and activates the pseudo AP (1409, 1412, 1415). Thereafter, the pseudo AP of the processing node 1402 acquires a pseudo packet (1417), assigns a transmission time (1418), holds it in the pseudo packet storage unit (1419), and starts acquiring operation information (1420). The pseudo AP of the processing node 1402 transmits a pseudo packet to the processing node 1403 (1421).

  Subsequently, the pseudo AP of the processing node 1403 receives the pseudo packet, holds the operation information acquisition start, the reception time and the transmission time, holds the pseudo packet in the pseudo packet storage unit, and transmits the pseudo packet to the processing node 1404 (1422-1427). ). Similar processing (1428 to 1432) is performed in the processing node 1404.

  14B, when the processing node 1404 transmits a pseudo packet to the processing node 1403 (1433), processing 1434 to 1442 are executed.

  In response to the above processing, the management manager 1401 issues a service stop instruction to each processing node, and operation information acquisition is stopped (1443 to 1451). Thereafter, the management manager 1401 acquires operation information and pseudo packet information from the processing nodes 1402, 1403, and 1404 (1452-1454) and stores them in the collected data storage unit (1455). Then, the time information between the processing nodes is corrected (1456), and the route through which the pseudo packet passes and the time taken for processing / passing are displayed on the display unit such as the display of the management server 14 (not shown). (1457). The service manager who operates the management manager 141 of the management server 14 compares the display data with the statistical information, identifies a greatly different portion, and performs evaluation (1458). As a result, it is possible to estimate the faulty part on the network described above.

  FIG. 15 shows an example of a specific configuration of the information processing system of the present embodiment described above. As shown in the figure, a processing node 151 and a management server 152 are connected to a network 150 such as a LAN. The network 150, processing node 151, and management server 152 correspond to the public network / intranet 17, processing nodes 11, 12, and 13, and the management server 14 described above. The management server 152 is a computer including a memory 142 connected to the internal bus 146, a data storage unit 143 such as an HDD, a LAN adapter 144 that is a network I / F, and a CPU 145. The inside of the management manager 62 has a functional configuration as shown in FIG.

  On the other hand, as shown in FIG. 1, the processing node 151 has a memory 115, a CPU 117, an HDD 118 as a data storage unit, an I / F 119 such as a LAN adapter, and the bus 125 also receives time information via the I / F 126. An acquisition device 127 is connected. In the HDD 118, the operation information storage unit 51 and the pseudo packet storage unit 53 described with reference to FIG. The memory (MM) 115 stores a real AP 111 and a pseudo AP 112 whose details are shown in FIG.

  Needless to say, the information processing system described above with reference to FIG. 11 is also constructed with the same specific system configuration.

  Although the embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and can be widely applied to other systems that perform hierarchical information processing.

  INDUSTRIAL APPLICABILITY The present invention is useful as a technique for acquiring performance information and monitoring a fault occurrence site in an information system applied to a social infrastructure, particularly a system that uses a wide area distributed resource.

11, 12, 13, 21-28, 1101-1104 ... processing nodes 14, 1106 ... management servers 15, 16, 17, 1107, 1108, 150 ... network 30 ... real world 31, 32, 1111, 1112 ... sensor 33, 34, 1113 ... Actuator 62, 141 ... Management manager 71 ... Communication header part 72 ... Payload part 111, 121, 131 ... Production AP
52, 112, 122, 132 ... pseudo AP
113, 123, 133 ... OS or AP execution platform 114, 124, 134 ... Hardware 115, 142 ... Memory (MM)
116 ... I / O
117, 145 ... CPU
118, 143 ... HDD
119, 144 ... I / F
1109 ... Terminal 1110 ... Edge node.

Claims (15)

A distributed resource monitoring system in which a distributed resource including a management server and a plurality of processing nodes performs hierarchical information processing via a network to provide a service,
The plurality of processing nodes are
A production application that performs the information processing, and a pseudo application that simulates the operation of the production application,
When receiving a pseudo packet corresponding to the service, the pseudo application includes sequence information including node identification information for identifying the processing node and transmission / reception time information indicating a time at which the pseudo packet is transmitted / received. And send it to the other processing nodes,
The management server
Acquire operation information when each of the pseudo applications of the plurality of processing nodes transmits / receives the pseudo packet, and corresponds to a time when the processing node transmits / receives the pseudo application based on the order information of the acquired pseudo packets. Determining that a failure has occurred in the processing node when the operation information is not included in a predetermined statistical value range ;
Distributed resource monitoring system characterized by that. The distributed resource monitoring system according to claim 1,
The management server sets the pseudo application to a plurality of the processing nodes via the network.
Distributed resource monitoring system characterized by that. The distributed resource monitoring system according to claim 1,
The management server issues a service stop instruction to the plurality of processing nodes via the network, and acquires the operation information and the pseudo packet from each of the processing nodes after the service stop instruction.
Distributed resource monitoring system characterized by that. The distributed resource monitoring system according to claim 1,
The plurality of processing nodes include a storage unit that stores the pseudo packet and the operation information, and transmits the pseudo packet and the operation information stored in the storage unit to the management server via the network.
Distributed resource monitoring system characterized by that. The distributed resource monitoring system according to claim 1,
The operation information includes server operation information, network operation information, and storage operation information.
Distributed resource monitoring system characterized by that. A distributed resource monitoring method for estimating a failure occurrence part of a system that provides a service by performing hierarchical information processing through a network in which a distributed resource including a management server and a plurality of processing nodes includes:
A production application that performs the information processing and a pseudo application that simulates the operation of the production application are set in a plurality of the processing nodes,
The processing node receives a pseudo packet corresponding to the service;
The pseudo application adds order information including node identification information for identifying the processing node and transmission / reception time information indicating the time at which the pseudo packet was transmitted / received to the pseudo packet, and transmits the pseudo packet to the other processing node. ,
The management manager of the management server
Acquire operation information when each of the pseudo applications of the plurality of processing nodes transmits / receives the pseudo packet, and corresponds to a time when the processing node transmits / receives the pseudo application based on the order information of the acquired pseudo packets. Determining that a failure has occurred in the processing node when the operation information is not included in a predetermined statistical value range ;
A distributed resource monitoring method characterized by the above. The distributed resource monitoring method according to claim 6,
The management manager sets the pseudo application to a plurality of the processing nodes via the network.
A distributed resource monitoring method characterized by the above. The distributed resource monitoring method according to claim 7,
The management manager can instruct a transmission destination designation of the pseudo application set in the processing node and a simulated load.
A distributed resource monitoring method characterized by the above. The distributed resource monitoring method according to claim 6,
The pseudo application running on the processing node collects the operation information of the processing node during reception and transmission of the pseudo packet;
A distributed resource monitoring method characterized by the above. The distributed resource monitoring method according to claim 6,
The management manager issues a service stop instruction to the plurality of processing nodes via the network, and acquires the operation information and the pseudo packet from each of the processing nodes after the service stop instruction.
A distributed resource monitoring method characterized by the above. A distributed resource monitoring apparatus for a system in which a plurality of processing nodes perform hierarchical information processing via a network to provide a desired service,
And production application implementing the information processing respectively, a plurality of said processing nodes with pseudo application that simulates the operation of the production application receives the pseudo packet corresponding to the service, to the pseudo application the pseudo packet, the node identification information for identifying the processing node, and the virtual packet by adding the order information including the transmission and reception time information indicating the time at which the transmitted and received, transmission and reception with the sequence information when transmitted to another of said processing nodes of said virtual packet A collection unit that collects operational information when
Based on the collected the sequence information of the pseudo packet, the operation information the processing nodes corresponding to the time that reception of the pseudo application, if not included in the scope of predetermined statistical value, a failure in the processing node generates Provided with a processing unit for determining that
A distributed resource monitoring apparatus. The distributed resource monitoring apparatus according to claim 11,
An interface unit that receives the order information and the operation information from the plurality of processing nodes via the network;
A distributed resource monitoring apparatus. The distributed resource monitoring apparatus according to claim 12, wherein
A pseudo application control unit configured to set the pseudo application to the plurality of processing nodes via the interface unit and the network;
A distributed resource monitoring apparatus. The distributed resource monitoring apparatus according to claim 11,
The management manager that functions as the collection unit and the processing unit, and a storage unit that stores the collected order information and operation information.
A distributed resource monitoring apparatus. The distributed resource monitoring apparatus according to claim 11,
The operation information includes server operation information, network operation information, and storage operation information.
A distributed resource monitoring apparatus.

Images