JP5130968B2 - Fault location identification method - Google Patents

Description

  The present invention is based on log information stored in each card in an information processing apparatus composed of cards in which a large number of functional units are integrated on a circuit board or the like, such as a radio network control apparatus of a mobile communication system. The present invention relates to a failure location identification method for identifying failure locations and factors.

  For example, as shown in FIG. 9, the radio network control device of the mobile communication system includes an external interface unit 401, a cell switching device 402, a trunk device 403, a No. In addition to the 7 signal termination unit 404 and the signal termination unit common unit 405, functional blocks 1 and 2 (indicated by reference numerals 406 and 407 in FIG. 9) that perform various functions, and a control processing unit 408 that controls them are provided. I have. The wireless network control device shown in FIG. 9 is connected to the maintenance terminal 410 via the maintenance terminal control unit 409. When a failure occurs, the maintenance operator 410 and the maintenance terminal Maintenance work for the network control apparatus is performed via the control unit 409. Note that the control signal that the maintenance terminal 410 shown in FIG. 9 exchanges with an external device via the wireless network control device is terminated by the terminal control signal termination unit 411.

  The external interface unit 401, the cell switching device 402, the trunk device 403, No. 4 shown in FIG. Each of the cards in which the 7 signal termination unit 404 and the functional blocks 1 and 2 are integrated is duplicated, and each of these cards is provided with a diagnostic processing unit. When a failure is detected inside the card in the diagnostic processing unit provided in each card, an alarm is generated, and in response to this, switching of duplicated cards is automatically performed, and the entire device The improvement of availability has been attempted.

  However, among the failures that occur in various types of cards as described above, there are failures that occur very frequently and failures that are extremely difficult to detect. Such failure detection processing is not incorporated into the diagnostic processing unit provided in each card or cannot be incorporated. Such a failure is called a silent failure because an alarm is not notified even if it occurs. However, due to the influence of such a silent failure, the processing of call control related events may not be completed normally, and the service quality deteriorates when such events occur frequently.

  In the past, the effect of silent failure, such as when the rate of completion of call control-related events decreases or the control loop is repeatedly disconnected from the opposite node, even though no alarm has occurred. In case of doubt, the following measures were taken.

  In other words, the maintenance worker switches each suspicious card via the signal communication route related to the problematic event to the standby system one by one, and determines whether or not the above-mentioned event has been recovered. In addition to identifying the card in which the error occurred, log information was collected from the suspicious card, and the collected log information was analyzed to identify the failure location and the cause of the failure in the unit.

Such a silent failure is also recognized in a network system composed of a wide variety of network devices, and various techniques for detecting the silent failure have been proposed (see Patent Documents 1 and 2).
Japanese Patent Laid-Open No. 9-6571 (4th and 6th pages, FIG. 1) JP 2006-245849 A

  By the way, as described above, in the method of collecting logs in parallel with the operation of switching the duplicated card to the standby system by the maintenance worker, the selection of the card for executing the switching to the standby system is also the log collection target. Since the selection of the card to be performed is left to the maintenance worker, the work load on the maintenance worker is heavy, and the time required for the recovery varies greatly depending on the knowledge and experience of the maintenance worker.

Thus, the time to recover the system by switching the card fails protection system at present that would be influenced by the skill of the maintenance worker, the card finally silent failure is identified However, a long time may elapse until necessary log information is collected. In such a case, the log information stored in the card where the silent failure has occurred is overwritten with the passage of time, making it impossible to collect log information when the silent failure occurs. May end up. The log information at the time when the silent failure occurs is exactly necessary information for identifying the location of the failure and the cause of the failure, and simply switching to the standby system without collecting and analyzing this log information. As a result, if the system is restored to the operating state, the problem of the card is left unattended, resulting in the recurrence of the failure.

  An object of the present invention is to provide a failure location specifying method that can collect and analyze log information when a silent failure occurs without omission.

  The above-described object can be achieved by the failure location method disclosed below.

The feature of this fault location identification method is that in the fault location identification method that collects and analyzes log information at the time of fault event occurrence according to fault event occurrence in a system consisting of multiple types of function cards, each fault case Correspondingly, information indicating the function card related to each failure case is registered in the collection management table, and when the failure event occurs, the collection management table is referenced corresponding to the failure event example corresponding to the failure event. And select the target card for log collection based on the registered information , collect the log information accumulated in the target card in the process of failure occurrence from the selected target card, and collect the collected log information to each log information item contained therein, it performs a process of detecting a failure factors, the analysis process and a process of aggregation for each target card detected failure factor, target card Abnormality corresponding to the log information item calculates an evaluation index indicating a possibility of the occurrence Doo lies in identifying the target of switching cards based on the evaluation index.

  In the fault location identification method configured in this way, for example, in the past maintenance process, a function card (for example, via a signal communication route) whose relevance to each fault event is empirically known. Is registered in the collection management table corresponding to each failure case, and is used for the process of selecting the function card that should be the log information collection target during the log information collection process . In this way, by collecting log information from the function card selected according to the occurrence of the failure case based on the information registered in the collection management table, the log information of the function card related to the failure that has occurred, It can be collected selectively and quickly. The log information collected from these function cards includes information when there is a silent failure related to the occurrence of the failure event. Therefore, by analyzing these log information for each log information item (for example, connection information, NG processing history information, CPU usage rate, etc.), each function card can be silently operated from the viewpoint of corresponding to each log information item. An evaluation index indicating a possibility that an abnormality such as a failure has occurred is calculated. For example, by comparing the evaluation indexes obtained for each function card in this way, it is possible to determine the magnitude of the possibility that an abnormality has occurred from the viewpoint of continuity of connection information, etc. It is possible to support the process of identifying

  According to the failure location identification method described above, the location of the silent failure and the cause of the failure are identified by automatically collecting log information related to the appropriate log information item of the appropriate card in response to the failure case that occurred. It is possible to collect necessary information for the analysis process quickly and without omission. In addition, by analyzing the collected log information for each log information item, an evaluation index indicating the magnitude of the possibility that an abnormality has occurred in each function card is calculated from the perspective corresponding to these items, and the switching target It can be used for card decision processing.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows an embodiment of a fault location specifying method disclosed in the section for solving the problem.

  1 that are the same as those shown in FIG. 9 are given the same reference numerals, and descriptions thereof are omitted.

  The radio network control apparatus shown in FIG. 1 includes a failure location specifying processing unit including a log information collection unit 211, a collection management table 212, and a log analysis processing unit 213 in addition to the units shown in FIG. Has been.

  FIG. 2 shows a detailed configuration of the failure location identification processing unit.

  In the collection management table 212 shown in FIGS. 1 and 2, as shown in FIG. 3 (a), failure cases such as control loop disconnection and completion rate decrease (in FIG. 3, as failure cases a, b, and c) The target card information indicating the type of card for collecting log information is stored. The target card information stored in correspondence with each failure case is, for example, whether or not the communication path related to the failure is going through, whether or not the failure point has been detected by dealing with the same failure case so far, etc. Can be determined in advance.

  In the log information collection unit 211 illustrated in FIG. 2, the collection processing unit 214 performs log information from each card indicated in the collection management table 212 according to the case designation information received from the control processing unit 408 illustrated in FIG. 1. Are collected and stored in the log storage unit 215.

  Here, in each card, log information including various types of log information items is accumulated corresponding to the time stamp indicating the date and time when the log information was acquired in each card, as shown in FIG. . For example, in the example shown in FIG. 4, the log information of the external interface unit includes at least connection information, signal discard information, failure history, NG processing history, and CPU usage rate as log information items, while resource information is It is shown not to contain. Therefore, when the collection processing unit 214 collects log information from each card indicated by the collection management table 212, log information including a plurality of types of log information items corresponding to each card is stored in the log storage unit 215. Is done.

  The log information held in the log storage unit 215 in this manner is passed to the link analysis unit 217 and the aggregation processing unit 218 of the log analysis processing unit 213 via the item information extraction unit 216. Then, based on the processing results of the link analysis unit 217 and the totalization processing unit 218, the identification processing unit 219 identifies the card in which the silent failure has occurred.

  Hereinafter, the operation of the failure location specifying processing unit shown in FIG. 2 will be described in detail.

  FIG. 5 is a flowchart showing the failure location specifying operation.

  For example, when a decrease in the completion rate of a call control event is detected or when a control loop break is frequently detected, case designation information indicating a detected failure case is obtained from the control processing unit 408 shown in FIG. The data is transferred to the collection processing unit 214 of the failure location identification processing unit shown in FIG. 2 (step 301). For example, according to the case designation information indicating the failure case a relating to the disconnection of the control loop, the collection processing unit 214 registers the target card (external interface unit, Log information is collected from the cell switching device, functional block 1, trunk device), and the collected log information is held in the log storage unit 215 (step 302).

  In this way, by registering in advance the target card for which log information is collected for each failure case in the collection management table 212, the card related to the failure case is immediately detected according to the detection of the failure case. The log information accumulated in can be collected. Thereby, before necessary log information is overwritten, log information can be reliably collected and used for analysis processing. Further, by limiting the log collection target cards for each failure case, the log information storage unit 215 can be realized by a memory having a realistic capacity. Note that the target card to be registered in the collection management table 212 corresponding to each failure case can be determined in advance based on, for example, results or experiments dealing with the previous failure cases.

  By the way, the applicant of the present invention specifies a failure factor that is likely to cause each failure case and log information items related to these failure factors based on the results of dealing with the failure case. Therefore, for example, as shown in FIG. 3B, log information items related to the failure factor described above are registered in the collection management table 212 in correspondence with the failure case, and the item information extraction unit 216 performs log By extracting the log item information shown in the collection management table 212 from the log information stored in the storage unit 215 (step 303 in FIG. 5), the log analysis processing unit 213 needs to identify the fault location. Information can be passed selectively.

  In this way, by narrowing down the log information to be used for the processing of the log analysis processing unit 213 in advance, the efficiency of the log analysis processing described later can be improved, and the failure location can be quickly identified.

  At this time, the item information extraction unit 216 transfers the connection information of each card extracted from the log storage unit 215 based on the collection management table 212 to the link analysis unit 217, and the log information of other items to the aggregation processing unit 218. Can be transferred.

  For example, as shown in FIG. 3B corresponding to the failure case a, when the collection target item includes connection information (affirmative determination in step 304 in FIG. 5), other log information items Prior to the analysis, a link analysis process based on the connection information is performed (step 505 in FIG. 5).

  FIG. 6 is a flowchart showing the link analysis operation.

  The link analysis unit 217 shown in FIG. 2 first arranges the connection information of each card received from the item information extraction unit 216 in time series for each target card as shown in FIG. 7 (step 321). FIG. 7 shows an example in which connection information collected from the external interface unit, the cell switching device, the function block 1, and the trunk device registered as the target card corresponding to the failure case a is arranged.

  Next, the link analysis unit 217 pays attention to the fact that the output virtual channel index (VCI) set in each card matches the input VCI of the link destination card in the connection information arranged as described above. The link of each connection is traced to search for a discontinuous portion of the link (step 322).

  For example, as shown by a thick line in FIG. 7, when the link is sequentially traced from the output VCI set in the connection C of the external interface unit, the function block 1 returns via the cell switching apparatus, and the cell switching apparatus Although the output VCI (55) is assigned, it can be seen that no link is set in the trunk device (in FIG. 7, this is indicated by a symbol “x”).

  By following the link in this way, a link check table as shown in FIG. 8A is created, and the location where the link is discontinuous, that is, the abnormal location of the connection information is detected as a failure factor. can do. In the example shown in FIG. 8A, the discontinuous portion is indicated by “−”.

  After the search for all the connections is completed, the link analysis unit 217 counts the detection frequencies of the discontinuous portions of the link for each card (step 323), and reports the count result to the specific processing unit 219 (step 324). ), And is used for the failure point identification process described later.

  Thereafter, the aggregation processing unit 218 illustrated in FIG. 2 performs aggregation processing for other items to be collected. When connection information is not included in the collection target item (No determination in step 304), the above-described link analysis processing is skipped, and aggregation processing for other collection target items is started.

  The aggregation processing unit 218 first classifies the log information of each target card received from the item extraction unit 216 for each log information item, and further, for example, data of each log information item (for example, data indicating CPU usage rate). From this, data indicating a value exceeding a corresponding threshold value (for example, CPU usage rate threshold value Thc) is detected as a failure factor (step 306). When the log information item data indicates the occurrence of an event as in the case of an NG processing history that is one of the log information items, the aggregation processing unit 218 uses the data indicating the occurrence of the event as a failure factor. Can be detected as

Next, the count processing unit 218 counts the frequency at which the failure factor is detected for each log collection item for each target card (step 307), and the totaled result is subjected to the specifying process by the specifying processing unit 219.

  In this way, for example, as shown in FIG. 8 (b), four target cards (external interface unit, cell switching apparatus, The total result of the failure factor detection frequency is reported to the specific processing unit 219 for each functional block 1 and trunk device).

  The identification processing unit 219 shown in FIG. 2 first compares the detection frequencies obtained for each target card for each log information item (step 308), and based on the comparison result, for example, the detection frequencies are concentrated. A card that is present is identified as a faulty card (step 309). For example, by comparing the detection frequency of each failure factor in each card with the average value for all target cards, and determining whether the detection frequency of any card is significantly higher than the average value, It is possible to determine a card in which the detection frequency of failure factors such as abnormal connection information is concentrated.

  For example, in the example shown in FIG. 8B, since both the abnormality of the connection information detected for the trunk device and the number of NG processes exceed the respective average values, the trunk device detects the failure factor. It can be determined that the frequency is concentrated. At this time, a failure factor (for example, an abnormality in connection information) for which the detection frequency is concentrated can be specified as a failure factor in this card (trunk device). In addition, a threshold value can be determined in advance for the detection frequency of each log information item, and a card having a detection frequency exceeding the threshold value can be specified as a failure card.

  As described above, according to the failure location identification processing unit shown in FIGS. 1 and 2, according to the occurrence of the failure case, the log information is immediately collected from the card related to the failure case, and the analysis is performed. The location of failure and the cause of failure can be identified quickly and reliably.

  In particular, by performing the above-described link analysis processing based on the connection information collected from each card through which the signal communication route passes, it is possible to quickly and reliably detect the occurrence of the connection abnormality. Such connection abnormality detection work is extremely complicated when performed manually, and by automating this work, the time required for recovery work including identification of the fault location and switching of the faulty card is greatly increased. It is possible to shorten the time, and it is possible to suppress the influence on the service of the occurrence of the failure case due to the silent failure.

  Further, the card that has detected the alarm as described above is notified to the control processing unit 408 that the alarm has been detected for the card (for example, the trunk device), and the alarm processing for this card is started, whereby the card in which the silent failure has occurred. Rapid system recovery can be achieved.

  On the other hand, only the card identified as described above is notified to the control processing unit 408 together with the analysis result as a suspected card suspected of generating a silent failure, and the final judgment can be left to the maintenance worker. In this case, the maintenance worker can determine whether or not to execute the alarm process for the identified suspect card while referring to the analysis result performed on the log information collected in a timely manner. Since the analysis result obtained as described above is a clear index indicating the possibility that a silent failure has occurred in the suspect card, by providing this to the maintenance worker, the identified suspect card Can help to make final decisions and encourage quick and accurate decisions.

  Furthermore, since all the log information collected from each target card is held in the log storage unit 215 shown in FIG. 2, after the failure card and the failure factor are identified as described above, It is also possible to investigate the behavior of a failure card when a silent failure occurs by performing detailed analysis including data of log information items that have not been processed by the link analysis unit 217 and the aggregation processing unit 218. is there.

  By the way, as in the example described above (see FIG. 8B), when abnormal connection information is intensively detected in one card (for example, trunk device), silent connection related to connection setting in this card is performed. It is very likely that a failure has occurred. Therefore, when the failure card can be identified based on the link analysis processing by the link analysis unit 217 without waiting for the analysis result regarding the other log information items, the analysis processing of other log information items is omitted, It is also possible to speed up the identification of the fault location.

  On the other hand, link analysis processing and aggregation processing for other log information items can be executed in parallel, and the results of link analysis processing and analysis results for other log information items are combined. It is also possible to evaluate the failure and identify the fault location based on the evaluation result.

  For example, as shown in FIG. 8C, a weight is set for each failure factor (for example, connection information abnormality, number of NG processes, CPU usage rate abnormality), and corresponds to the failure factor detection frequency of each target card. By accumulating the values multiplied by the weights to be evaluated, an evaluation value representing the severity of the failure in each target card can be obtained. Since the evaluation value obtained in this way reflects the detection frequency of the above-mentioned multiple failure factors and the severity of each failure factor, a failure occurs by comparing the magnitudes of the evaluation values. It is possible to reliably determine the high possibility of being. For example, based on this evaluation value, the suspicion rank is determined in descending order of the possibility of being a failure card (see FIG. 8C), and for example, the target card given the highest suspicion rank is specified as the failure card. be able to.

It is a figure which shows embodiment of the failure location identification method. It is a figure which shows the detailed structure of a fault location specific process part. It is a figure which shows the example of a collection management table. It is a figure which shows the example of log information. It is a flowchart showing a failure location specific operation. It is a flowchart showing link analysis operation. It is a figure explaining link analysis processing. It is a figure explaining log information analysis processing. It is a figure which shows the structural example of a wireless network control apparatus.

Explanation of symbols

211 Log Information Collection Unit 212 Collection Management Table 213 Log Analysis Processing Unit 214 Collection Processing Unit 215 Log Storage Unit 216 Item Information Extraction Unit 217 Link Analysis Unit 218 Total Processing Unit 219 Specific Processing Unit 401 External Interface (I / F) Unit 402 Cell Exchange device 403 Trunk device 404 7 signal termination unit 405 signal termination unit common unit 406 functional block 1
407 Function block 2
408 Control processing unit 409 Maintenance terminal control unit 410 Maintenance terminal 411 Terminal control signal termination unit

Claims (5)

In a failure location identification method that collects and analyzes log information when a failure event occurs according to the failure event occurrence in a system composed of multiple types of function cards,
Corresponding to each of multiple failure cases, register information indicating the function card related to each failure case in the collection management table,
According to the occurrence of the failure event, refer to the collection management table corresponding to the failure event example corresponding to the failure event, select the target card to be the target of log collection based on the registered information,
Collect log information accumulated in the target card in the process of occurrence of the failure case from the selected target card,
For each log information item included in the collected log information, an analysis process including a process of detecting a failure factor and a process of aggregating the detected failure factors for each target card is performed . Calculate an evaluation index that indicates the possibility of an abnormality corresponding to the information item,
A failure location specifying method, wherein a switching target card is specified based on the evaluation index. In the fault location identification method according to claim 1,
In correspondence with each of the plurality of failure cases, log information items to be collected are registered in the collection management table,
When collecting log information from the log collection target card according to the occurrence of a failure event, the information on the log information item registered in the collection management table is extracted and used for analysis processing. Identification method. In the fault location identification method according to claim 1,
A fault location identifying method characterized in that alarm processing is executed for the identified switching target card. In the fault location identification method according to claim 1,
When analyzing log information including connection information,
Arrange the connection information included in the collected log information according to the time series,
By detecting the link is discontinuous by following the link indicated by the arranged connection information,
A failure location specifying method, comprising: obtaining an evaluation index related to an abnormality in connection information of each log collection target card based on a result of counting the number of discontinuous portions of the link for each log collection target card. In the fault location identification method according to claim 1,
When identifying the card to be switched,
Corresponding to each log information item, giving a weight according to the importance of the corresponding abnormality,
For each function card that is the target of log collection, add the weight corresponding to each log information item to the calculated evaluation index,
A fault location identification method, wherein the function card having the largest addition result is selected as a switching target.

Images