CN115756929A - Abnormal root cause positioning method and system based on dynamic service dependency graph - Google Patents

Abstract

The invention provides an abnormal root cause positioning method and system based on a dynamic service dependency graph, and belongs to the field of intelligent operation and maintenance. The method comprises the following steps of positioning an abnormal root cause, extracting a service dependency relationship during deployment based on service configuration information, discovering the service dependency relationship during operation based on the correlation of key performance indexes among services and log frequency information, and dynamically constructing a service dependency graph; automatically constructing an abnormal propagation diagram based on the input abnormal service and dynamic service dependency diagram; and constructing an abnormal propagation path based on depth-first search, calculating the abnormal root cause score, positioning abnormal root cause service, and reporting the abnormal propagation path. The method can better capture the change of the service dependency relationship during operation, is beneficial to depicting more accurate abnormal propagation relationship, and improves the capability of positioning abnormal root causes; the method is suitable for positioning various abnormal root causes, and has good universality; the method directly locates abnormal services and provides abnormal propagation paths, and has good practicability and interpretability. And the manual participation is not needed, and the labor cost is saved.

Description

A Method and System for Abnormal Root Cause Location Based on Dynamic Service Dependency Graph

technical field

The invention belongs to the field of intelligent operation and maintenance, and in particular relates to a method for locating abnormal root causes based on a dynamic service dependency graph.

Background technique

The system of microservice architecture (referred to as microservice system) is composed of a large number of fine-grained microservices, which work together to realize system functions, and are widely used in communication, transportation, logistics, finance and other fields. Modern Internet applications often use microservice systems as the underlying implementation. The core design idea of the microservice architecture is to split the application into several highly cohesive, low-coupling, single-function microservices that can be independently developed, deployed, and updated, so as to achieve agile development and continuous delivery of applications. The microservice architecture makes the development and iteration of applications more convenient, but it also brings new challenges to the operation and maintenance of the system, that is, the complex microservice system makes the application prone to abnormalities and it is difficult to diagnose the root cause of the abnormalities.

The microservice system is complex, which is specifically reflected in the large number of microservices, dynamic changes in dependencies between microservices, and heterogeneous implementation of microservices. Due to the single function of microservices, in order to complete the functions of the system, the microservice system may have a large number of microservices. For example, Netflix's microservice system runs more than 6,000 microservice instances and needs to process more than 2 billion service requests every day; Uber's microservice system also runs more than 4,000 microservice instances. At the same time, in order to respond to changes in user needs and application development, the microservice system can dynamically deploy and register microservice instances, and new microservices are constantly added, making the dependencies between microservices dynamic and intricate. Moreover, the service interface encapsulation of the microservice system allows different microservices in the system to adopt different programming languages, service frameworks, and communication mechanisms to cope with different application scenarios. The heterogeneous design and implementation in this system further deepens the complexity of the microservice system.

The complexity of the microservice system makes it prone to exceptions and the root cause of the exceptions is difficult to diagnose. An anomaly means that the operating state of the system deviates from the design expectation of the system. The loosely coupled design and heterogeneous implementation of microservices make the cooperation between microservices often complete through network remote calls, which is prone to abnormalities caused by network instability, version incompatibility, configuration errors, and code defects. The huge number of microservices further increases the possibility of the above abnormalities. Moreover, due to the complex and dynamic changes in the dependencies between services, any minor system abnormality may cause a chain reaction, causing multiple abnormalities to occur at the same time, and only one or a few of them are the root causes of other abnormalities. This is because the exception will continue to spread with the dependencies between services, forming an exception propagation chain. There is a causal relationship between these microservice exceptions caused by dependencies, and the source of the exception propagation chain is often the root cause of the exception. In order to ensure the reliability of the system, when an abnormality is detected in the system operation, the operation and maintenance personnel need to locate the root cause of the abnormality as soon as possible and take corresponding measures, such as rollback of changes. Otherwise, the exception may cause service interruption or service quality degradation, and cause serious losses. The International Data Corporation has reported that an hour of service outage costs an average of about $100,000. However, although the microservice system can provide a wealth of operation and maintenance data, such as logs, key performance indicators, and call chains, the differentiated data format, sparse valuable information, and dynamically changing service dependencies make it difficult for operation and maintenance personnel to Locating the root cause of anomalies in a complex microservice system can lead to serious anomalies that cannot be repaired in time, and bring more serious economic damage and reputation damage to the enterprise.

However, the existing abnormal root cause location methods for microservice systems still have deficiencies, often ignoring the dynamic changes of service dependencies and the interpretability of root cause location. Existing methods mostly rely on service dependency graphs. The service dependency graph is used to describe the dependency relationship between services, which can also be used to describe the exception propagation between services, which helps the operation and maintenance personnel to locate the root cause of the exception. Abnormal root cause location based on the service dependency graph first constructs the service dependency graph through indicators, system logs or tracking data, and then when an exception occurs, starting from the abnormal node in the service dependency graph, the cause is obtained through graph search, random walk and other algorithms A collection of candidate anomaly root causes of anomalies, and then sort the candidate anomaly root causes by anomaly score, correlation with anomalous nodes, or number of visits. However, existing methods often rely on static service dependency graphs, ignoring the dynamic changes between service dependencies, resulting in differences between the constructed service dependencies and the real situation, which limits the accuracy of existing abnormal root cause location methods. At the same time, existing methods often only provide abnormal root cause services or root cause indicators, which lack interpretability, making it difficult for operation and maintenance personnel to quickly understand the report results and determine the accuracy of the root cause.

Contents of the invention

In order to reduce the additional analysis and rollback caused by inaccurate location of abnormal root causes, reduce operation and maintenance costs, improve the interpretability of abnormal location, and ensure the reliability of the microservice system, the present invention provides a dynamic service dependency graph based exception Root cause location method. Extract service dependencies during deployment based on service configuration information, discover runtime service dependencies based on the correlation of key performance indicators between services and log frequency information, and dynamically build service dependency graphs; based on input abnormal services and service dependency graphs, automatically Build an anomaly propagation map; construct an anomaly propagation path based on depth-first search, calculate the root cause score of the anomaly, locate the abnormal root cause service, and report the anomaly propagation path.

In the present invention, the construction of the dynamic service dependency graph, the construction of the abnormal propagation graph, and the location of the root cause of the abnormality are all carried out automatically without manual participation, which saves labor costs. The abnormal root cause location of the present invention is based on the dynamic service dependency graph, which can better capture the change of the service dependency relationship at runtime, help to describe a more accurate abnormal propagation relationship, and improve the ability to locate the abnormal root cause; it is applicable to various The root cause location of various types of abnormalities has good versatility; it directly locates the abnormal service and provides the propagation path of the abnormality, which has good practicability and explainability.

The technical scheme provided by the invention is:

An abnormal root cause location method based on a dynamic service dependency graph, characterized in that it includes dynamic service dependency graph construction, exception propagation graph construction, and abnormal root cause location; the specific steps include:

1) Dynamic service dependency graph construction, specifically perform the following steps:

11) Extract deployment-time service dependencies: The principle here is that two microservices that are deployed on the same location (for example, a host) or depend on the same resource (for example, a database) have deployment-time dependencies. The configuration information of the service is extracted from a configuration management database (Configuration Management Database, CMDB), including deployment location information and associated resource information. Define the attribute set K=(k ₁ ,k ₂ ,...,k _n ) of the configuration information to be collected, so that the collected configuration information c _i =( _ki ,v _i ) must have k _i ∈ K, where, k _i represents the attribute (metadata) of the configuration information, and v _i represents the value of the configuration information. Denote the configuration set of a service s _i as C _i = (c ₁ , c ₂ , ..., c _n ), if there is an intersection between the configuration sets of service s _i and service s _j , that is, C _i ∩C _j ≠φ, Then there is deployment dependency between service s _i and service s _j .

12) Extract runtime service dependencies: The principle here is that if there is a causal relationship between the number of logs or key performance indicators (Key Performance Indicators, KPIs) generated by the two services as the system load changes, then there is a runtime dependency between the two services , and the strength of the causal relationship can represent the strength of the runtime dependency. The causal inference algorithm used here is the PC algorithm. The reason is that given a reliable conditional independence test method, the PC algorithm can handle various types of data distribution and causality, which is complex compared to other causal inference methods. Low, good effect advantages. The PC algorithm judges the causal relationship between variables through the conditional independence test, and then uses the d-separation condition to determine the direction of the causal relationship. The conditional independence test method used here is the G ² conditional cross-entropy measure, as shown in formula (1), which obeys the χ ² distribution with D degrees of freedom, as shown in formula (2).

D=(N _X -1)(N _Y -1)Π _Z'∈Z N _Z'Formula (2)

Z是Z′的集合，m是采样的个数。in,

Z is a collection of Z', and m is the number of samples.

13) Constructing a dynamic service dependency graph: Steps 11) and 12) will be performed each time an abnormal root cause location task is executed, and service dependencies at deployment time and runtime service dependencies are dynamically obtained. And based on the deployment-time service dependency and runtime service dependency, build a service dependency graph G=<V, E>, where G is a directed graph (Directed Graph, DG), V represents a service, which is a node of G; E represents a dependency relationship , is the edge of G. The construction rule is, if there is a runtime dependency between service s _i and service s _j , then add a directed edge between s _i and s _j , and the weight of the edge is the ^G2 value between s _i and s _j . If there is a deployment-time dependency between service s _i and service s _j , then add directed edges from s _i to s _j and from s _j to s _i , and the weights of the edges are the mean value of the edge weights of the runtime dependencies.

2) To construct the exception propagation graph, the specific steps are as follows:

21) Build an exception propagation graph: mark the input exception set A={a ₁ , a ₂ ,..., a _n } on the service dependency graph G, take out the service nodes marked as exceptions, and form a subgraph of G G', while retaining the edges between nodes, the value of the node is the reciprocal of the time when the abnormality occurs. Wherein, a _i =(s _i , t _i ) represents the i-th abnormality, s _i represents the service where the abnormality a _i occurs, and t _i represents the time when the abnormality a _i occurs. In order to facilitate the calculation, the time when the abnormality occurs is standardized. Set the time when the first abnormality occurs as 1; the other abnormalities follow the order of occurrence, and the occurrence time is incremented on the basis of 1.

3) Locating the root cause of the abnormality, specifically perform the following steps:

31) Constructing an anomaly propagation path: On the anomaly propagation graph G′, randomly select an abnormal service s′ ₀ to start, and use depth-first search to find candidate root cause nodes and their anomaly propagation paths.

32) Calculating the abnormal root cause score: For s′ ₀ , use the abnormal propagation graph G′ and the abnormal propagation path Path(s′ ₀ , s′ _t ) to calculate the abnormal root cause score for each s′ _t ∈ R. The calculation formula is shown in formula (3).

Among them, score(s′ _t ) represents the abnormal root cause score of abnormal service s′ _t , M represents the number of abnormal propagation paths from s′ ₀ to s′ _t , N represents the number of hops in the propagation path o _k , w _{k, i} represents the dependency weight between node i and node i-1 in the kth propagation path, and s _i represents the value of node i, which is the reciprocal of the moment when the abnormality occurs. The design principle of the root cause score calculation formula here is that the earlier the anomaly occurs, the more likely it is the root cause, and the more likely the anomaly with more propagation paths is the root cause.

33) Report abnormal root cause and propagation path: arrange the abnormal root cause in descending order according to the score, and report the corresponding abnormal propagation path.

The present invention further provides an anomaly root cause location system based on a dynamic service dependency graph, which is characterized in that it includes a dynamic service dependency graph construction module, an anomaly propagation graph construction module, and an anomaly root cause location module.

Dynamic service dependency graph building modules, including deployment-time service dependency discoverer, runtime service dependency discoverer, dynamic service dependency graph builder, service dependency graph storage; deployment-time service dependency Deployment-time dependencies between services; runtime service dependency discoverer uses service operation logs and key performance indicators to discover runtime dependencies between microservices; dynamic service dependency graph builder uses deployment-time service dependencies and runtime Time service dependency, dynamically construct service dependency graph; service dependency graph storage is used to store service dependency graph;

Exception propagation graph building block, including exception propagation graph builder, exception propagation graph storage. The exception propagation graph builder uses the service dependency graph and the input exception set to construct an exception propagation graph; the exception propagation graph storage is used to store the exception propagation graph;

Abnormal root cause location module, including abnormal propagation path builder, abnormal root cause calculator, and abnormal root cause reporter. The abnormal propagation path builder uses the abnormal propagation graph to find all possible abnormal root cause nodes, and constructs the abnormal propagation path; the abnormal root cause calculator uses the abnormal propagation path, according to the number of paths, the weight on the path and the occurrence At time, the root cause score of the anomaly is calculated; the anomaly root cause reporter sorts the anomalies in descending order of the scores, and reports the corresponding anomaly propagation path.

Compared with prior art, the beneficial effect of the present invention is:

The invention provides a method for locating abnormal root causes based on a dynamic service dependency graph and the system dynamically builds a service dependency graph by reading service configuration information, service operation logs and key performance indicators. The service dependency graph is stored for subsequent generation of an exception propagation graph. When an exception occurs in the system, an exception propagation graph is constructed based on the service dependency graph and exception collection. Then use depth-first search to traverse all possible root cause anomalies and their propagation paths, use the number of anomaly propagation paths, the weight on the path, and the time when the anomaly occurs to calculate the root cause score of the anomaly, sort in descending order of the score, and report The corresponding exception propagation path. The present invention can automatically build a dynamic service dependency graph, generate an abnormality propagation graph, construct an abnormality propagation path, calculate root cause scores, thereby locate the root cause of the abnormality, and report the corresponding propagation path. The present invention mainly has the following characteristics:

(1) The system and method provided by the present invention automatically build a dynamic service dependency graph based on service configuration information and runtime information (key performance indicators and logs).

(2) The system and method provided by the present invention can use the input exception and service dependency graph to automatically build an exception propagation graph.

(3) The system and method provided by the present invention can not only locate the abnormal root service, but also provide an abnormal propagation path, which has good practicability and explainability.

The technical solution of the present invention can automatically construct a dynamic service dependency graph, construct an anomaly propagation graph, locate the root cause of anomalies, and provide an anomaly propagation path, which is suitable for microservice systems.

Description of drawings

Fig. 1 is an abnormal root cause location method based on a dynamic service dependency graph provided by the present invention;

Fig. 2 is an abnormal root cause location system based on a dynamic service dependency graph provided by the present invention.

Detailed ways

Below in conjunction with accompanying drawing, further describe the present invention through embodiment, but do not limit the scope of the present invention in any way.

Fig. 1 is a flow chart of an abnormal root cause location method based on a dynamic service dependency graph provided by the present invention. The invention includes dynamic service dependency graph construction, abnormal propagation graph construction, abnormal root cause location;

The dynamic service dependency graph construction uses service configuration information, service operation logs and key performance indicators to mine the deployment-time dependencies and runtime dependencies between services, and dynamically constructs the service dependency graph. Configuration information is used to describe the location of the service and the resources and services it depends on. Each configuration item exists in the form of two tuples, including configuration attributes and configuration values. The running log is used to record the running status of the system, including the output of key variables and the marking of key running positions. The operation log exists in the form of time series text, which is converted into the form of time series of log frequency in the present invention. Key performance indicators are used to monitor the operating status of the system and to monitor whether the system is abnormal. Key performance indicators exist in the form of time series. The service dependency graph is a directed graph, which depicts the dependency relationship between various services in the microservice system. The nodes of the graph are services, and the edges are the dependencies between services. The service dependency graph is dynamically constructed with the accumulation of system operation data, and is adaptive to system iteration.

Exception propagation graph construction uses the input exception collection and service dependency graph to construct an exception propagation graph for subsequent propagation path analysis. Each exception item exists in the form of a two-tuple, including the service where the exception occurred and the time when the exception occurred. The exception propagation graph is also in the form of a directed graph, which is a subgraph of the service dependency graph.

Abnormal root cause location uses the abnormal propagation graph to find all possible abnormal root cause nodes, and constructs the corresponding abnormal propagation path, which is used to calculate the root cause score of the nodes, and sorts in descending order to generate an abnormal root cause report. An anomaly root cause report contains root cause service scores and their corresponding anomaly propagation paths.

For the above abnormal root cause location method based on the dynamic service dependency graph, the construction of the dynamic service dependency graph specifically performs the following steps:

11) Extract deployment-time service dependencies: The principle here is that two microservices that are deployed on the same location (for example, a host) or depend on the same resource (for example, a database) have deployment-time dependencies. The configuration information of the service is extracted from a configuration management database (Configuration Management Database, CMDB), including deployment location information and associated resource information. Define the attribute set K=(k ₁ ,k ₂ ,...,k _n ) of the configuration information to be collected, so that the collected configuration information c _i =( _ki ,v _i ) must have k _i ∈ K, where, k _i represents the attribute (metadata) of the configuration information, and v _i represents the value of the configuration information. Denote the configuration set of a service s _i as C _i = (c ₁ , c ₂ , ..., c _n ), if there is an intersection between the configuration sets of service s _i and service s _j , that is, C _i ∩C _j ≠φ, Then there is deployment dependency between service s _i and service s _j .

12) Extract runtime service dependencies: The principle here is that if there is a causal relationship between the number of logs generated by the two services or the Key Performance Indicator (KPI) as the system load changes, the two services have runtime dependencies. And the strength of the causal relationship can represent the strength of the runtime dependency. The causal inference algorithm used here is the PC algorithm. The reason is that given a reliable conditional independence test method, the PC algorithm can handle various types of data distribution and causality, which is complex compared to other causal inference methods. Low, good effect advantages. The PC algorithm judges the causal relationship between variables through the conditional independence test, and then uses the d-separation condition to determine the direction of the causal relationship. The conditional independence test method used here is the G ² conditional cross-entropy measure, as shown in formula (1), which obeys the χ ² distribution with D degrees of freedom, as shown in formula (2).

D=(N _X -1)(N _Y -1)Π _Z'∈Z N _Z'Formula (2)

Z是Z′的集合，m是采样的个数。in,

Z is a collection of Z', and m is the number of samples.

时间窗口个的总数为n。初始时，假设在m个服务中，任意两个服务之间存在因果关系。利用时间序列数据T，计算任意s_i与s_j之间的G²值，并查询其在χ²分布的p值，若p值＞ξ,则s_i与s_j之间的条件独立性假设被接受，否则拒绝。若s_i与s_j之间的条件独立性假设被接受，则记录此时条件Z作为s_i与s_j的分割条件S(s_i，s_j)。同理，利用日志频数序列数据L，对s_i与s_j进行条件独立性检验。如果针对s_i与s_j的关于T和L的两次独立性假设均被接受，则判定sa_i与s_j之间没有因果关系，否则判定s_i与s_j之间存在因果关系。遍历所有的s_i与s_j对，直至确定所有的s_i与s_j对之间的因果关系。121) Determine runtime service dependency weights: Specifically, the KPI used here is a Service Level Objective (SLO) indicator, such as service request delay, used to evaluate whether a service is running normally. Record the KPI time series of a service s _i as T _i ={t ₁ , t ₂ ,...,t _n }, and the log frequency sequence as L _i ={l ₁ , l ₂ ,...,l _n }, The time window size is

The total number of time windows is n. Initially, it is assumed that among m services, there is a causal relationship between any two services. Using the time series data T, calculate the G ² value between any s _i and s _j , and query its p value in the χ ² distribution, if the p value > ξ, then the conditional independence assumption between s _i and s _j Accepted, otherwise rejected. If the conditional independence assumption between s _i and s _j is accepted, record the condition Z at this time as the split condition S(s _i , s _j ) between s _i and s _j . Similarly, use the log frequency sequence data L to test the conditional independence of s _i and s _j . If the two independence assumptions about T and L for s _i and s _j are accepted, then it is judged that there is no causal relationship between sa _i and s _j , otherwise it is judged that there is a causal relationship between s _i and s _j . Traverse all s _i and s _j pairs until the causal relationship between all s _i and s _j pairs is determined.

122) Determining the direction of service dependency at runtime: then use the d-separation condition to determine the direction between causal relationships. There are four rules for the d-separation condition,

则将X-Z-Y赋予方向X→Z←Y。(1) For any two variables X and Y that are not adjacent (no causal relationship), and have a common neighbor variable Z, if

Then assign XZY to the direction X→Z←Y.

(2) If X→Y exists, assign all Y-Z to the direction Y→Z.

(3) If X→Z→Y exists, assign all X-Y to the direction X→Y.

(4) If XZ ₁ →Y and XZ ₂ →Y exist at the same time, assign all XY to the direction X→Y.

Among them, rule (1) takes precedence over rule (2)(3)(4), that is, to ensure that all rules (1) are executed, and then execute rule (2)(3)(4). Rules (2)(3)(4) are executed in no order.

If the direction of dependence between s _i and s _j cannot be determined after executing the above rules, add a two-way dependency, that is, add the dependency relationship from s _i to s _j , and add the dependency relationship from s _j to s _i at the same time.

13) Constructing a dynamic service dependency graph: Steps 11) and 12) will be performed each time an abnormal root cause location task is executed, and service dependencies at deployment time and runtime service dependencies are dynamically obtained. And based on the deployment-time service dependency and runtime service dependency, build a service dependency graph G=<V, E>, where G is a directed graph (Directed Graph, DG), V represents a service, which is a node of G; E represents a dependency relationship , is the edge of G. The construction rule is, if there is a runtime dependency between service s _i and service s _j , then add a directed edge between s _i and s _j , and the weight of the edge is the ^G2 value between s _i and s _j . If there is a deployment-time dependency between service s _i and service s _j , then add directed edges from s _i to s _j and from s _j to s _i , and the weights of the edges are the mean value of the edge weights of the runtime dependencies.

For the above abnormal root cause location method based on the dynamic service dependency graph, the construction of the abnormal propagation graph specifically performs the following steps:

21) Build an exception propagation graph: mark the input exception set A={a ₁ , a ₂ ,..., a _n } on the service dependency graph G, take out the service nodes marked as exceptions, and form a subgraph of G G', while retaining the edges between nodes, the value of the node is the reciprocal of the time when the abnormality occurs. Wherein, a _i =(s _i , t _i ) represents the i-th abnormality, s _i represents the service where the abnormality a _i occurs, and t _i represents the time when the abnormality a _i occurs. In order to facilitate the calculation, the time when the abnormality occurs is standardized. Set the time when the first abnormality occurs as 1; the other abnormalities follow the order of occurrence, and the occurrence time is incremented on the basis of 1.

For the above abnormal root cause location method based on the dynamic service dependency graph, the abnormal root cause location specifically performs the following steps:

31) Constructing an anomaly propagation path: On the anomaly propagation graph G′, randomly select an abnormal service s′ ₀ to start, and use depth-first search to find candidate root cause nodes and their anomaly propagation paths.

311) Finding candidate root cause node sets: In the stage of finding candidate root cause nodes, initialize, s' _i =s' ₀ , o _k =φ, R=φ, and then perform the following recursive steps:

(1) If s′ _i ∈ o _k , then return to the superior call.

(2) Add s′ _i to _ok .

(3) If s' _i does not have adjacent abnormal nodes, then add s' _i to the set R of candidate root cause nodes.

令s′_i＝s′_j，并执行步骤(1)。(4) If s′ _i has an adjacent abnormal node s′ _j , for each s′ _j , if

Set s' _i =s' _j , and execute step (1).

(5) Return to the superior call.

Recursively execute steps (1) to (5) until no new exception service is added to R.

312) Determine the abnormal propagation path: in the stage of finding the propagation path, for each s′ _t ∈ R, record the abnormal propagation path set from the abnormal service s′ ₀ to the abnormal service s′ _t as Path(s′ ₀ , s′ _t )={o ₁ , o ₂ ,..., o _n }, where o _k is an abnormal propagation path, ok _k ={s′ ₀ ,..., s _t ′}. Initialization, s′ _i =s′ ₀ , o _k =φ, Path(s′ ₀ , s′ _t )=φ, and then perform the following recursive steps

(1) If s′ _i ∈ o _k , then return to the superior call.

(2) Add s′ _i to _ok .

(3) If s' _i == s' _t , add o _k to Path(s' ₀ , s' _t ).

令s′_i＝s′_j，并执行步骤(1)。(4) If s′ _i has an adjacent abnormal service node s′ _j , for each s′ _j , if

Set s' _i =s' _j , and execute step (1).

(5) Return to the superior call.

Steps (1) to (5) are recursively executed until no new anomaly propagation path is added to Path(s′ ₀ , s′ _t ).

32) Calculating the abnormal root cause score: For s′ ₀ , use the abnormal propagation graph G′ and the abnormal propagation path Path(s′ ₀ , s′ _t ) to calculate the abnormal root cause score for each s′ _t ∈ R. The calculation formula is shown in formula (3).

Among them, score(s′ _t ) represents the abnormal root cause score of abnormal service s′ _t , M represents the number of abnormal propagation paths from s′ ₀ to s′ _t , N represents the number of hops in the propagation path o _k , w _{k, i} represents the dependency weight between node i and node i-1 in the kth propagation path, and s _i represents the value of node i, which is the reciprocal of the moment when the abnormality occurs. The design principle of the root cause score calculation formula here is that the earlier the anomaly occurs, the more likely it is the root cause, and the more likely the anomaly with more propagation paths is the root cause.

33) Report abnormal root cause and propagation path: arrange the abnormal root cause in descending order according to the score, and report the corresponding abnormal propagation path.

Fig. 2 is a structural block diagram of an abnormal root cause location system based on a dynamic service dependency graph provided by the present invention.

The present invention provides a system for realizing an abnormal root cause location method based on a dynamic service dependency graph. The system uses configuration information, operation logs, key performance indicators, and abnormal collections as inputs, and includes a dynamic service dependency graph construction module and an abnormal propagation graph construction module, abnormal root cause location module;

The different modules are described in detail below.

S1) Dynamic service dependency graph building blocks

The function of the dynamic service dependency graph building module is to build a dynamic service dependency graph based on service configuration information, operation logs, and key performance indicators. This module contains four submodules:

S11) Deployment time service dependency finder

The deployment-time service dependency finder mines deployment-time dependencies between microservices that are deployed at the same location or depend on the same resource based on service configuration information.

S12) Runtime Service Dependency Finder

The runtime service dependency discoverer mines the runtime dependencies between microservices based on service running logs and key performance indicators.

S13) Dynamic service dependency graph builder

The dynamic service dependency graph builder dynamically builds a service dependency graph based on service deployment-time dependencies and runtime dependencies.

S14) service dependency graph memory

The service dependency graph storage stores the service dependency graph in the form of a matrix, and provides high-performance query on the service dependency graph.

S2) Exception Propagation Graph Building Blocks

The function of the exception propagation graph building block is to build an exception propagation graph according to the service dependency graph and the input exception collection. This module contains two submodules:

S21) Exception Propagation Graph Builder

The exception propagation graph builder marks exceptions on the service dependency graph and generates subgraphs according to the service dependency graph and the input exception set, and constructs the exception propagation graph.

S22) abnormal propagation map memory

The exception propagation graph memory stores the exception propagation graph in the form of a matrix, and provides high-performance query on the exception propagation graph.

S3) abnormal root cause location module

The function of the abnormal root cause location module is to find all possible abnormal root cause nodes according to the abnormal propagation graph, construct their abnormal propagation paths, calculate the root cause score, and generate an abnormal root cause report. This module is divided into three submodules:

S31) Exception propagation path builder

The abnormal propagation path builder uses the abnormal propagation graph to find all possible abnormal root cause nodes and constructs the abnormal propagation path.

S32) abnormal root cause calculator

The anomaly root cause calculator uses the anomaly propagation path to calculate the root cause score of the anomaly based on the number of anomaly propagation paths, the weight on the path, and the time when the anomaly occurs.

S32) abnormal root cause reporter

The anomaly root cause reporter sorts anomalies in descending order of scores and reports the corresponding anomaly propagation paths.

It should be noted that the purpose of the disclosed embodiments is to help further understand the present invention, but those skilled in the art can understand that various replacements and modifications are possible without departing from the spirit and scope of the present invention and the appended claims of. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the protection scope of the present invention is subject to the scope defined in the claims.

Claims (7)

1. An abnormal root cause positioning method based on a dynamic service dependency graph is characterized by comprising the steps of constructing the dynamic service dependency graph, constructing an abnormal propagation graph and positioning the abnormal root cause; the method comprises the following specific steps:

1) Constructing a dynamic service dependency graph, and specifically executing the following steps:

11 Extract deployment-time service dependencies;

12 Extract runtime service dependencies;

13 Build a dynamic service dependency graph G: step 11) and step 12) are executed each time the abnormal root cause positioning task is executed, service dependence during deployment and service dependence during operation are dynamically obtained, and a service dependence graph G = < V, E >; wherein G is a directed graph, V represents a service, and is a node of G; e represents a dependency, being an edge of G;

2) Constructing an abnormal propagation diagram, and specifically executing the following steps:

constructing an abnormal propagation map: exception set A = { a) to be input ₁ ，a ₂ ，...，a _n Marking the service dependency graph G, taking out the service nodes marked as abnormal to form a subgraph G' of the G, and simultaneously reserving edges among the nodes, wherein the values of the nodes are the reciprocal of the abnormal occurrence time;

3) And (3) positioning the abnormal root cause, and specifically executing the following steps:

31 Construct an exception propagation path: randomly selecting one abnormal service s 'on the abnormal propagation graph G' ₀ Starting, searching candidate root cause nodes and abnormal propagation paths thereof by depth-first search;

32 Compute an abnormal root score: to s' ₀ Using the abnormal propagation map G 'and the abnormal propagation Path Path (s' ₀ ，s′ _t ) Calculate each s' _t E.g. abnormal root score of R;

33 Report exception root cause and propagation path: and sorting the abnormal root causes according to the scores in a descending order, and reporting the abnormal propagation path corresponding to the abnormal root causes.

2. The dynamic service dependency graph-based anomaly root cause locating method according to claim 1, wherein in step 11) a set of attributes K = (K) for configuration information to be collected is defined ₁ ，k ₂ ，...，k _n ) So that the collected configuration information c _i ＝(k _i ，b _i ) Must have k _i E.g. K, wherein K _i Attribute representing configuration information, v _i Value representing configuration information, remembering a service s _i Is C _i ＝(c ₁ ，c ₂ ，...，c _n ) If service s _i And service s _j There is an intersection of the configuration sets of, i.e. C _i ∩C _j Not equal to phi, then service s _i And service s _j There is a deployment dependency between.

3. The dynamic service dependency graph-based abnormal root cause locating method as claimed in claim 1, wherein step 12) employs a PC algorithm to judge causal relationships between variables through conditional independence test, and then determines direction between causal relationships using d-separation condition, and the conditional independence test method used is G ² The conditional cross entropy measure is as shown in formula (1) subject to χ with degree of freedom D ² Distribution, as shown in formula (2):

D＝(N _X -1)(N _Y -1)Π _Z′∈Z N _Z′ formula (2)

wherein ,

z is the set of Z' and m is the number of samples.

4. The dynamic service dependency graph-based anomaly root cause positioning method according to claim 1, wherein in step 13) the construction rule is that if the service s is a service s _i And service s _j If there is a runtime dependency between them, then add s _i And s _j With directed edges in between, the weight of the edge being s _i And s _j G between ² Value, if service s _i And service s _j There is a deployment-time dependency between, then s is added _i To s _j And s _j To s _i The weights of the edges are all the mean values of the run-time dependent edge weights.

5. The dynamic service dependency graph-based abnormal root cause locating method as claimed in claim 1, wherein in step 2), a _i ＝(s _i ，t _i ) Indicates the ith exception, s _i Indicates the occurrence of an anomaly a _i Service of t _i Indicating the occurrence of an anomaly a _i Standardizing the time when the abnormality occurs, and setting the time when the abnormality occurs firstly as 1; the other exceptions are in the order of occurrence, and the occurrence time is increased on the basis of 1.

6. The method for locating abnormal root cause based on dynamic service dependency graph as claimed in claim 1, wherein the calculation formula in step 32) is shown in formula (3):

wherein, score (s' _t ) Represents abnormal service s' _t M represents s' ₀ To s' _t N represents the propagation path o _k Number of hops in, w _k，i Represents the dependency weight between node i and node i-1 in the k-th propagation path, s _i The value of node i, i.e. the inverse of the time of occurrence of the anomaly.

7. The abnormal root cause positioning system based on the dynamic service dependency graph is characterized by comprising a dynamic service dependency graph building module, an abnormal propagation graph building module and an abnormal root cause positioning module, wherein:

the dynamic service dependency graph building module comprises a service dependency relationship finder during deployment, a service dependency relationship finder during operation, a dynamic service dependency graph builder and a service dependency graph memory; the service dependency relationship discovering device discovers the dependency relationship between the micro services when deployed by using the service configuration information; the runtime service dependency relationship finder finds runtime dependency relationships among the microservices by using the service running logs and the key performance indexes; the dynamic service dependency graph builder uses the service dependency relationship during deployment and the service dependency relationship during operation to dynamically construct a service dependency graph; the service dependency graph memory is used for storing a service dependency graph;

the abnormal propagation graph constructing module comprises an abnormal propagation graph constructor and an abnormal propagation graph memory, wherein the abnormal propagation graph constructor constructs an abnormal propagation graph by using the service dependency graph and the input abnormal set; the exception propagation map memory is used for storing an exception propagation map;

the abnormal root cause positioning module comprises an abnormal propagation path builder, an abnormal root cause calculator and an abnormal root cause reporter, wherein the abnormal propagation path builder uses an abnormal propagation graph to search all possible abnormal root cause nodes and build an abnormal propagation path; the abnormal root cause calculator uses the abnormal propagation path and calculates the abnormal root cause score according to the number of the paths, the weight on the upper side of the path and the abnormal occurrence time; and the abnormal root cause reporter sorts the abnormal according to the descending order of the scores and reports the abnormal propagation path corresponding to the abnormal root cause reporter.

Images