WO2021032175A1

WO2021032175A1 - Fault injection method and device, and service system

Info

Publication number: WO2021032175A1
Application number: PCT/CN2020/110351
Authority: WO
Inventors: 崔成; 魏凌
Original assignee: 华为技术有限公司
Priority date: 2019-08-20
Filing date: 2020-08-20
Publication date: 2021-02-25
Also published as: CN110674028A

Abstract

A fault injection method, comprising: a first application generates a service message, the service message being used for accessing a second application; a fault injection module obtains the service message, and performs a fault injection operation indicated by fault attribute parameters on the service message; the fault injection module sends the service message having experienced the fault injection operation to the second application. The method improves the accuracy of reliability testing according to an injected fault.

Description

Fault injection method and device and business service system

Technical field

This application relates to the computer field, in particular to a fault injection method and device, and business service system.

Background technique

With the rapid development of cloud computing technology, the use of cloud services has become more and more common. In addition, for cloud service providers, building their own cloud services (hereinafter referred to as cloud services B) on the basis of other cloud services (hereinafter referred to as cloud services A) can effectively improve the development efficiency of cloud services. However, this will cause the cloud service B to have a certain dependency on cloud service A, which will cause cloud service B to be unavailable when the implementation system of cloud service A fails. Therefore, test the implementation system of cloud service The reliability of the implementation system of the cloud service on which it depends is necessary when the system fails.

In related technologies, the implementation of the reliability test of the implementation system of cloud service B is to inject faults into the implementation system of cloud service A to simulate the failure of the implementation system of cloud service A, and then use the faulty The realization system continues to provide cloud services to the realization system of cloud service B, and then observes the performance of the realization system of cloud service B after receiving the cloud service provided by the failed realization system, and then evaluates the realization system of cloud service B based on the performance Reliability. Among them, the current injected faults mainly include: the shutdown fault of the server that implements the cloud service, the network disconnection fault of the server that implements the cloud service, and the forced exit fault of the cloud service process.

However, the above faults are all global faults, resulting in low accuracy of reliability testing based on the faults.

Summary of the invention

The present application provides a fault injection method, a device thereof, and a business service system, which can solve the problem of low accuracy of the current reliability test.

In the first aspect, a fault injection method is provided, which is applied to a business service system in which a first application, a second application, and a fault injection module are running. The method includes: the first application generates a business message , The business message is used to access the second application; the fault injection module obtains the business message, and performs the fault injection operation indicated by the fault attribute parameter on the business message; the fault injection module sends the business message after the fault injection operation is performed to the second application .

In the fault injection method provided by the embodiment of the present application, after the first application generates a business message for accessing the second application, the business message is acquired through the fault injection module, and the fault indicated by the fault attribute parameter is executed on the business message. Inject operation and send the business message after the fault injection operation is performed to the second application, which can inject faults in the process of sending business messages. Compared with the global faults in related technologies, the granularity of the injected faults is refined and effective This improves the accuracy of reliability testing based on injected faults. Moreover, since the injecting fault is realized by intercepting the business message during the sending process of the business message, there is no need to adapt the object in which the fault is injected, and the faultless injection can be realized.

As an achievable way for the fault injection module to obtain business messages, an address conversion module is also running in the business service system. Before the fault injection module obtains the business message, the method further includes: the address conversion module obtains the business message and transfers the business message The destination address is modified to the address of the fault injection module, and the business message with the modified destination address is sent.

In one implementation, the fault injection module performs the fault injection operation indicated by the fault attribute parameter on the business message, including: the fault injection module performs the fault injection operation indicated by the fault attribute parameter on the business message when the business message meets the filter condition . By using filter conditions to filter business messages, and when the business message meets the filter conditions, faults are injected into the business messages, so that fault injection can be carried out in a targeted manner for business requests, which can avoid the introduction of irrelevant tests in the test process. Accurate injection of faults.

Among them, when the business message is a business request, the filtering conditions involve one or more of the following content: request type, address requested to be accessed, request header keywords, and request body keywords; when the business message is a response to the business request, Filter conditions involve one or more of the following: response status code, response header keywords, and response body keywords.

Optionally, the fault injection operation indicated by the fault attribute parameter may be a packet loss operation, a delay operation, and a packet error operation. Accordingly, the fault injection operation indicated by the fault attribute parameter may be performed in the following situations:

The business message includes multiple data packets. When the fault injection operation indicated by the fault attribute parameter is a packet loss operation, the fault injection module performs the fault injection operation indicated by the fault attribute parameter on the business message, including: the fault injection module is based on the fault attribute parameter , To discard some or all of the multiple packets.

When the fault injection operation indicated by the fault attribute parameter is a delayed operation, the fault injection module performs the fault injection operation indicated by the fault attribute parameter on the business message, including: the fault injection module is based on the fault attribute parameter and is specified in the delayed fault attribute parameter. After a period of time, a business message is sent.

When the fault injection operation indicated by the fault attribute parameter is an error packet operation, the fault injection module performs the fault injection operation indicated by the fault attribute parameter on the business message, including: the fault injection module modifies the message in the business message based on the fault attribute parameter .

In a second aspect, a business service system is provided. The business service system includes: a first server, a second server, and a fault injection module; the first server is used to generate business messages, and the business messages are used to access the second server; and the fault injection module It is used to obtain the business message and perform the fault injection operation indicated by the fault attribute parameter on the business message; the fault injection module is also used to send the business message after the fault injection operation is performed to the second server.

Optionally, the business service system further includes: an address conversion module; the address conversion module is used to obtain business messages, modify the destination address of the business message to the address of the fault injection module, and send the business message with the modified destination address; the fault injection module is specific It is used to receive the business message sent by the address conversion module, and perform the fault injection operation indicated by the fault attribute parameter on the business message.

Optionally, the fault injection module is specifically configured to perform the fault injection operation indicated by the fault attribute parameter on the business message when the business message meets the filter condition.

Optionally, when the business message is a business request, the filtering conditions involve one or more of the following content: request type, address to be accessed, request header keywords, and request body keywords; when the business message is a response to the business request When the filter condition involves one or more of the following content: response status code, response header keywords, and response body keywords.

Optionally, when the fault injection operation indicated by the fault attribute parameter is a packet loss operation, the fault injection module is specifically used to: when the business message includes a data packet, the fault injection module discards a data packet based on the fault attribute parameter; When the business message includes multiple data packets, the fault injection module discards some or all of the multiple data packets based on the fault attribute parameters.

Optionally, when the fault injection operation indicated by the fault attribute parameter is a delayed operation, the fault injection module is specifically configured to send a business message after delaying the time specified by the fault attribute parameter based on the fault attribute parameter.

Optionally, when the fault injection operation indicated by the fault attribute parameter is an error packet operation, the fault injection module is specifically configured to modify the message in the business message based on the fault attribute parameter.

In a third aspect, a fault injection device is provided. The fault injection device includes: a first monitoring unit and a fault injection unit; the first monitoring unit is used to obtain a business message generated by a first application, and the business message is used to access a second Application; the fault injection unit is used to obtain the business message from the first monitoring unit, perform the fault injection operation indicated by the fault attribute parameter on the business message, and send the business message after the fault injection operation is executed to the second application.

Optionally, the fault injection device further includes: a filtering unit; the filtering unit is configured to receive the business message sent by the first monitoring unit, filter the business message according to the filtering condition, and send the business message that meets the filtering condition to the fault injection unit .

Optionally, the fault injection device further includes: a second monitoring unit; the fault injection unit is specifically configured to send the business message after the fault injection operation is performed to the second monitoring unit; the second monitoring unit is used to send the service message after the fault injection operation is performed The business message is sent to the second application.

Optionally, the fault injection device further includes: an address conversion unit; the address conversion unit is used to modify the destination address of the service message to be sent by the second monitoring unit to the address of the second application.

Description of the drawings

FIG. 1 is a schematic structural diagram of a fault injection module provided by an embodiment of the present application;

2 is a schematic diagram of an application scenario involved in a fault injection method provided by an embodiment of the present application;

3 is a schematic diagram of an application scenario involved in another fault injection method provided by an embodiment of the present application;

4 is a schematic diagram of an application scenario involved in another fault injection method provided by an embodiment of the present application;

FIG. 5 is a flowchart of a fault injection method provided by an embodiment of the present application;

Fig. 6 is a schematic diagram of a user interface for managing an application in a client provided by an embodiment of the present application.

detailed description

In order to make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

In related technologies, when testing the reliability of the implementation system of cloud services, the injected faults mainly include: downtime failure of the server that implements the cloud service, network disconnection failure of the server that implements the cloud service, and forced exit failure of the cloud service process. Sexual failure. Since the granularity of the injected faults are all server-level faults, after the injected faults, the function implementation of other unrelated services will be affected and unrelated tests will be introduced, resulting in low accuracy of reliability testing based on the injected faults. Moreover, when injecting faults into the implementation system of the cloud service on which the cloud service under test depends, a lot of adaptation work is usually required, and it is impossible to inject faults without perception.

The embodiments of the present application provide a fault injection method, which can improve the accuracy of reliability testing based on injected faults. The following first describes the application scenarios involved in the fault injection method provided in the embodiments of the present application.

This application scenario involves the first application, the second application and the fault injection module. The first application program and the second application program may be application programs that have a dependency relationship in function realization. That is, the implementation of the functions of the dependent end in the first application and the second application depends on the implementation of the functions of the dependent end in the first application and the second application. Among them, the dependency relationship between the two can be expressed as: the dependent end can send a business request to the dependent end, the dependent end can process the business request, and send a business response carrying the processing result to the dependent end, and the dependent end will be based on The business response implements the function of the dependent end. In the normal working process of the first application and the second application, the business message generated by the first application for accessing the second application can be directly sent from the first application to the second application.

In the embodiment of the present application, the business message generated by the first application program for accessing the second application program needs to be sent to the fault injection module first. The fault injection module executes the fault attribute on the business message according to the fault attribute parameter. After the fault injection operation indicated by the parameter, the service message after the fault injection is sent to the second application. Wherein, the fault attribute parameter may be pre-stored in the fault injection module, or the fault injection module may receive a fault injection request, the fault injection request is used to indicate a fault injecting a business message, and the fault injection request may carry The fault attribute parameter.

In an achievable manner, the first application can be the dependent end, and the second application can be the dependent end, that is, the business message can be a business request sent by the first application to the second application. Faults are injected into the process of sending business requests. From the perspective of the first application, it can be regarded as a fault in the second application. Correspondingly, the response of the first application after receiving the business response sent by the second application The analysis can realize the reliability analysis of the first application when the second application fails.

In another implementation, the second application can be the dependent end, and the first application can be the dependent end, that is, the business message can be the business response sent by the first application to the second application. Faults are injected during the sending of business responses. From the perspective of the second application, it can be considered that the first application has a fault. Correspondingly, the response of the second application after receiving the faulty service response The analysis can realize the reliability analysis of the second application when the first application fails.

Optionally, the function of forwarding the business message generated by the first application program for accessing the second application program to the fault injection module may be implemented by the address conversion module. For example, the business message generated by the first application program can be obtained through the address conversion module, the destination address of the business message can be modified to the address of the fault injection module, and the business message with the modified destination address can be sent, so that the original need to be sent The business message to the second application is sent to the fault injection module.

Optionally, the function of the fault injection module can be realized by multiple functional units. For example, as shown in FIG. 1, the fault injection module may include: a first monitoring unit C01 and a fault injection unit C02. The first monitoring unit C01 is used to obtain a business message generated by a first application, and to transfer the business message Send to fault injection unit C02. The fault injection unit C02 is configured to perform the fault injection operation indicated by the fault attribute parameter on the service message based on the fault attribute parameter, and send the service message after the fault injection to the second application, so that the second application can Business messages are processed. At this time, since the business message generated by the first application program is obtained through the first monitoring unit 01, the address conversion module modifies the destination address of the business message to the address of the fault injection module, essentially modifying the destination address of the business message to this The address of the first listening unit C01.

In addition, the fault injection module can inject faults into business messages with specific attributes for targeted reliability testing. At this point, the business messages can be screened, and faults can be injected into the business messages that pass the screening. In an implementable manner, the function of screening service messages can be implemented by the filtering unit C03. That is, as shown in FIG. 1, the multiple functional units may further include: a filtering unit C03, the filtering unit C03 is used to filter business messages according to filtering conditions, and send business messages that meet the filtering conditions to the fault The injection unit C02 sends the business message that does not meet the filtering conditions to the second application. By injecting faults into business messages with specific attributes, it is possible to avoid the introduction of irrelevant tests during the testing process, and to achieve precise fault injection.

Further, as shown in FIG. 1, the multiple functional units may further include: a second monitoring unit C04. At this time, the fault injection unit C02 may send the service message after the fault is injected to the second monitoring unit C04, so as to send the service message after the fault injection to the second application through the second monitoring unit C04. In addition, the filtering unit C03 may bypass the fault injection unit C02 and send the business messages that do not meet the filtering conditions to the second monitoring unit C04, so as to directly send the business messages that do not meet the filtering conditions to the second application through the second monitoring unit C04 program.

In an achievable manner, in order to ensure that the service message after the fault injection can be sent to the second application, as shown in FIG. 1, the fault injection module may further include: an address conversion module C05, the address conversion unit C05 is used to modify the destination address of the service message after the fault is injected to the address of the second application. For example, after the fault injection unit C02 sends the service message after the fault injection to the second monitoring unit C04, the address conversion unit C05 may modify the destination address of the service message after the fault injection to the address of the second application, so that The second monitoring unit 04 sends the service message after the fault injection to the second application according to the modified destination address. Alternatively, after the filtering unit C03 sends the business message that does not meet the filtering conditions to the second monitoring unit 04, the address conversion unit C05 may modify the destination address of the business message that does not meet the filtering conditions to the address of the second application. The second monitoring unit 04 sends the business messages of the business messages that do not meet the filtering conditions to the second application according to the modified destination address.

It should be noted that the functions of the above fault injection module can be implemented in whole or in part by software, hardware, or a combination of software and hardware. Moreover, when the function of the fault injection module is realized by software, the function of the fault injection module may be realized in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on the computer, the processes or functions described in the embodiments of the present application are fully or partially realized. The computer can be a general-purpose computer, a dedicated computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server, or data center through a wired (For example: coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wireless (for example: infrared, wireless, microwave, etc.) to transmit to another website, computer, server or data center. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center integrated with one or more available media. The usable medium can be magnetic medium (for example: floppy disk, hard disk, magnetic tape), optical medium (for example: Digital Versatile Disc (DVD)), or semiconductor medium (for example: Solid State Disk (SSD)) Wait.

When software is used to implement the function of the fault injection module, the application scenarios involved in the fault injection method provided in the embodiments of the present application may have multiple deployment modes, and the following are described as examples.

As shown in FIGS. 2 and 3, the application scenario includes: a first server 110 and a second server 120. Both the first server 110 and the second server 120 may be one server, or a server cluster composed of several servers, or a cloud computing service center. The first server 110 and the second server 120 may be connected through a wired network or a wireless network.

In addition, as shown in FIGS. 2 and 3, the first server 110 includes a first processor 1101, a first communication interface 1102, and a first memory 1103. The first processor 1101, the first communication interface 1102, and the first memory 1103 are connected to each other through a first bus 1104. The second server 120 includes a second processor 1201, a second communication interface 1202, and a second memory 1203. The second processor 1201, the second communication interface 1202, and the second memory 1203 are connected to each other through a second bus 1204. Wherein, the first memory 1103 and the second memory 1203 are both used to store a computer program, and the computer program may be an application program. When the first processor 1101 calls the application program in the first memory 1103, it can realize the function of the application program When the second processor 1201 calls the application in the second memory 1203, it can implement the function of the application.

In the application scenario shown in FIG. 2, the first server 110 may run a first application program, an address conversion module, and a fault injection module, and the second server 120 may run a second application program. At this time, as shown in FIG. 2, the first application program 1103a, the address conversion module 1103b, and the fault injection module 1103c may be stored in the first memory 1103, and the second application program 1203a may be stored in the second memory 1203.

In the application scenario shown in FIG. 3, the first server 110 may run a first application program and an address conversion module, and the second server 120 may run a second application program and a fault injection module. At this time, as shown in FIG. 3, the first memory 1103 may store the first application program 1103a and the address conversion module 1103b, and the second memory 1203 may store the second application program 1203a and the fault injection module 1203b.

FIG. 4 is a schematic diagram of an application scenario involved in another fault injection method provided by an embodiment of the present application. As shown in FIG. 4, the application scenario includes: a first server 110, a second server 120, and a third server 130. The first server 110, the second server 120, and the third server 130 may all be one server, or a server cluster composed of several servers, or a cloud computing service center. The first server 110 and the second server 120, the first server 110 and the third server 130, and the second server 120 and the third server 130 may all be connected through a wired network or a wireless network.

In addition, as shown in FIG. 4, the first server 110 includes a first processor 1101, a first communication interface 1102, and a first memory 1103. The first processor 1101, the first communication interface 1102, and the first memory 1103 are connected to each other through a first bus 1104. The second server 120 includes a second processor 1201, a second communication interface 1202, and a second memory 1203. The second processor 1201, the second communication interface 1202, and the second memory 1203 are connected to each other through a second bus 1204. The third server 130 includes a third processor 1301, a third communication interface 1302, and a third memory 1303. The third processor 1301, the third communication interface 1302, and the third memory 1303 are connected to each other through a third bus 1304. The first memory 1103, the second memory 1203, and the third memory 1303 are all used to store computer programs, and the computer programs may be application programs. When a processor in a server calls an application program in the server’s memory, it can be implemented The functionality of the application.

In the application scenario shown in FIG. 4, the first application program and the address conversion module may be running in the first server 110, the second application program may be running in the second server 120, and the third server 130 may be running faulty. Inject the module. At this time, as shown in FIG. 4, the first application program 1103a and the address conversion module 1103b can be stored in the first memory 1103, the second application program 1203a is stored in the second memory 1203, and the fault injection is stored in the third memory 1303. Module 1303a.

When hardware is used to implement the function of the fault injection module, the application scenarios involved in the fault injection method provided in the embodiments of the present application may include: a first server, a second server, and a fault injection module. Both the first server and the second server may be one server, or a server cluster composed of several servers, or a cloud computing service center. The first server and the second server, the first server and the fault injection module, and the fault injection module and the second server may be connected through a wired network or a wireless network. And in this application scenario, for the deployment situation of the first application in the first server and the deployment situation of the second application in the second server, please refer to the deployment situation in FIG. 3 accordingly, and will not be repeated here.

It should be noted that when the first application program and the address conversion module are both deployed in the first server 110, the address conversion module may be integrated in the first application program. For example, the address conversion module may be a small program in the first application program. When a fault injection request is received, the address conversion module may obtain the business message generated by the first application program for accessing the second application program, and Modify the destination address of the business message to the address of the fault injection module so that the fault injection module can obtain the business message.

Or, when the first application program and the address conversion module are both deployed in the first server 110, the address conversion module and the first application program may be independently deployed in the first server 110, for example, the foregoing FIGS. 2 to 4 Both are exemplary illustrations of the independent deployment of the first application program and the address translation module. When the address translation module is deployed independently of the first application program, from the perspective of the first application program, after the first application program generates a business message for accessing the second application program, the process of sending the business message by the first application program It is the same as the business message sending process when the reliability test is not carried out, so that there is no need to change the first application program during the test process, which can solve the problem of a large amount of adaptation work in the related technology before the test can be carried out. No perception injection.

In FIGS. 2 to 4, any one of the first bus 1104, the second bus 1204, and the third bus 1304 can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in FIGS. 2 to 4 to indicate, but it does not mean that there is only one bus or one type of bus.

In FIGS. 2 to 4, any one of the first processor 1101, the second processor 1201, and the third processor 1301 may be a hardware chip, and the hardware chip may be an application-specific integrated circuit (application-specific integrated circuit). , ASIC), programmable logic device (programmable logic device, PLD) or a combination thereof. The above-mentioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. Or, it may also be a general-purpose processor, for example, a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP.

In FIGS. 2 to 4, any one of the first memory 1103, the second memory 1203, and the third memory 1303 may include a volatile memory (volatile memory), such as random-access memory (RAM). ); It can also include non-volatile memory, such as flash memory, hard disk drive (HDD), or solid-state drive (SSD); it can also include the above A combination of types of storage.

The following takes the deployment situation shown in FIG. 2 and the multiple functional units of the fault injection module include: a first monitoring unit, a fault injection unit, a filtering unit, a second monitoring unit, and an address conversion unit as an example, the embodiment of the present application is provided The implementation process of the fault injection method is explained. When the first application, the second application, the address conversion module, and the fault injection module are deployed according to other conditions, please refer to the implementation process of step 501 to step 507 below for the implementation process of the fault injection method. As shown in Figure 5, the method includes:

Step 501: The first monitoring unit receives the fault injection request, and sends an address conversion request to the address conversion module based on the fault injection request.

The address conversion request is used to request that the destination address of the business message generated by the first application program for accessing the second application program be modified to the address of the first monitoring unit. The fault injection request is used to request for fault injection during the process of sending a business message from the first application to the second application. In addition, the fault injection request may carry a fault attribute parameter, and the fault attribute parameter is used to indicate the fault to be injected. At this time, after receiving the fault injection request, the first monitoring unit also needs to send the fault attribute parameter to the fault injection unit. Alternatively, the fault attribute parameter may also be pre-stored in the fault injection unit, which is not specifically limited in the embodiment of the present application.

Moreover, the fault injection request is sent to the fault injection module, and the fault injection request can be received by any functional unit in the fault injection module, for example, it can be received by the first listening unit or by the filtering unit, Or received by the fault injection unit, or received by the second monitoring module. Figure 5 is an example of receiving the fault injection request by the first listening unit.

When it is necessary to perform reliability analysis on the implementation system of the application program, a fault injection request can be triggered on the management client to request the implementation system of the reliability to be analyzed to inject faults. For example, the tester can submit a fault injection task in the management client to trigger the fault injection request. Or, the fault injection request can be triggered by invoking a software development kit (SDK). And the task of injecting faults submitted in the management client can be operated in the user interface of the application in the management client, or in the user interface of the browser in the management client.

Among them, the fault attribute parameter can be input in the user interface of the management client. Optionally, the fault attribute parameter may include: the fault identifier of the injected fault. For example, the first server may store program instructions for injecting different faults. After receiving the fault injection request carrying the fault identifier, the first server may call the program instructions of the fault indicated by the fault identifier according to the fault identifier, And by running the called program instructions, to achieve the fault injection indicated by the fault identifier.

In an implementable manner, the fault attribute parameter may further include one or more of the following: the probability of occurrence of the fault and the duration of the fault. By setting the duration of the failure, the duration of the failure injection can be effectively controlled, so that the failure state of the dependent end in the first application and the second application can be simulated within a specified time period. When the fault attribute parameter includes the probability of occurrence of the fault, by injecting the fault according to the fault attribute parameter, the fault can be controlled to randomly appear fault according to the probability of occurrence, so as to simulate the situation that the dependent end is in a sub-healthy state and realize The reliability test of the dependent end when the end is in a sub-health state. Among them, the dependent end is in a sub-healthy state refers to the state in which the dependent end can provide services with probabilistic failure. For example, suppose the dependent end is the second application, and when the injection fault is a packet loss fault, if the probability of occurrence of the fault is 100% (that is, the packet loss rate is 100%), it can be simulated that the second application receives data The packet is the failure state of discarding. If the probability of occurrence of the failure is 50% (that is, the packet loss rate is 50%), it can be simulated that the second application randomly discards the sub-health state of the received data packet with a probability of 50%.

In another achievable way, faults can be injected in a targeted manner. At this time, the fault attribute parameters can also include one or more of the following: port number and Ip of the injected fault, and filter conditions for filtering business messages . When performing fault injection into the service message, the port number and Ip of the injected fault are the port number and Ip of the port used by the second application to receive the service message. At this time, by injecting the fault according to the fault attribute parameter, the situation of the port failure can be simulated. By setting filter conditions for filtering business messages, faults can be injected into business messages with specific attributes, so that faults can be injected into business requests in a targeted manner, and irrelevant tests can be avoided in the testing process and other tests that do not need to be tested. Service affects, and accurate injection of faults is achieved. It should be noted that, when the fault injection request carries a filter condition, after the first monitoring unit receives the fault injection request, the filter condition needs to be sent to the filter unit.

For example, FIG. 6 is a schematic diagram of a user interface for managing an application in a client provided by an embodiment of the application. The tester can input fault attribute parameters in the user interface, and click on the user interface after the parameter input is completed. "Inject now" button to trigger a fault injection request. Among them, the fault attribute parameters that need to be input in the user interface shown in Figure 6 include: the environment name of the test environment, the name of the environment node used to deploy the test environment, and the Internet Protocol address of the server under test running the first application ( internet protocol address, IP), the network data exchange rules (protocol) in the test environment, the IP and port number (server_address) of the server on which the business depends, the type of the injected fault (drop_type, also called the fault identifier of the injected fault), the fault The probability of occurrence (drop_rate), the duration of the failure (timeout), and the filter conditions for filtering business messages (filter_keyword_content).

Figure 6 is a user interface diagram of the injection failure as a packet loss failure. As shown in Figure 6, the environment name of the packet loss failure is: ECS_cui, the name of the environment node is: Apigateway_001, and the IP of the server under test is: 172.168.200.41 , The network data exchange rule is: Hypertext Transfer Protocol Secure (https), the IP and port number of the server on which the business depends is: 172.168.200.42:8080, and the fault identifier of the injected fault is: req_drop (the identifier Used to indicate that the injection failure is a packet loss failure), the probability of occurrence of the failure (that is, the packet loss rate) is: 100%, the duration of the failure is: 3600 seconds, and the filter condition for filtering business messages is: all (all), The all indicates that all business messages are filtered, that is, fault injection is performed on all business messages.

Step 502: The first application program generates a business message for accessing the second application program.

When the user needs the first application to provide business services (for example, cloud services), the client can send a business request instruction to the first application through the client, and the first application can generate a corresponding business message after receiving the business request instruction (At this time, it is also called a business request). And because the function realization of the first application depends on the function realization of the second application, the first application can generate a service request for accessing the second application, so as to request the second application to send the first application to the first application. The application program provides the corresponding service, so that the first application program provides the service indicated by the service request instruction to the client according to the first application program. Or, when the user needs the second application to provide business services (for example, cloud services), he can send a business request instruction to the second application through the client, and the second application can generate the corresponding service request instruction after receiving the business request instruction. Service request. After the second application sends a service request to the first application level, the first application can generate a service message for accessing the second application based on the service request (also called service response at this time) to facilitate the second application. The application program provides the client with the service indicated by the service request instruction according to the service response.

Step 503: The address conversion module obtains the service message, based on the address conversion request, modifies the destination address of the service message to the address of the first monitoring unit, and sends the modified service message.

After the address conversion module receives the address conversion request, it can monitor whether the first application program has generated a business message for accessing the second application program, and after monitoring that the first application program has generated a business message for accessing the second application program At this time, the business message is acquired, the destination address of the business message is modified to the address of the first monitoring unit, and the business message with the modified destination address is sent to the first monitoring unit.

For example, suppose the port number and IP of the first application are: 172.168.200.41:4000, the port number and IP of the first listening unit are: 172.168.200.41:5000, and the port number and IP of the second listening unit are: 1.2 .3.4:8080, the port number and IP of the second application are: 172.168.200.42:8080, the service request generated by the first application is: http://172.168.200.42:8080/v1/xxxx, which is not required During fault injection, the sender address of the business message generated by the first application program for accessing the second application program is 172.168.200.41:4000, and the destination address of the business message is 172.168.200.42:8080, in the first application program After sending the service message, the router and other devices in the network will send the service message to the second application according to the destination address of the service message.

When fault injection is required, the address translation module can modify the destination address of the business message to the address of the first listening unit, so that the destination address of the business message becomes 172.168.200.41:5000, that is, the destination address of the business message is changed from 172.168. 200.42:8080 is modified to 172.168.200.41:5000. At this time, after the address translation module sends out the service message for modifying the destination address, the gateway and other devices in the network will send the service message to the first listener according to the modified destination address unit. Therefore, by modifying the destination address of the business message, the fault injection module can intercept the business message that is expected to be sent to the second application, and after injecting a fault into the business message, send the injected business message to the second application.

Among them, the destination address of the business message can be modified through the iptables command (a kind of program instruction). For example, the command to modify the destination address of the business message from 172.168.200.42:8080 to 172.168.200.41:5000 can be: iptables–t nat– A OUTPUT–d 172.168.200.42–p tcp–m tcp–dport 8080–j DNAT–to_destination 172.168.200.41:5000.

It should be noted that when it is necessary to stop injecting faults, the address replacement module can be controlled to stop working, so that the first application can directly issue business messages for accessing the second application, so that the first application and the Second, the application works in the original way, so that the business returns to normal.

Step 504: The first monitoring unit receives the service message with the modified destination address, and sends the modified service message to the filtering unit.

Step 505: The filtering unit filters the business messages according to the filtering conditions, and when the business messages meet the filtering conditions, sends the business messages to the fault injection unit.

During reliability testing, targeted testing can be performed. At this time, faults can be injected into business messages with specific attributes, and then reliability testing can be completed according to the business messages after the faults are injected. The implementation method of injecting faults into business messages with specific attributes can be: using a filtering unit to filter the business messages obtained by the fault injection unit according to the filtering conditions, and when the business messages meet the filtering conditions, send the business messages to the fault injection Unit, and inject faults into business messages through the fault injection unit. This can avoid the introduction of irrelevant tests in the test process, and achieve accurate fault injection.

Among them, the filter conditions can be set according to actual needs. For example, when the service message is a service request, the filter condition may involve one or more of the following content: request type, address requested to be accessed, request header keyword, and request body keyword. When the service message is a response to a service request, the filter condition may involve one or more of the following content: response status code, response header keywords, and response body keywords.

For example, the request type may include: a request to change information to the receiver (ie a POST request), a request to query data (ie a GET request), a request to request the header of the page (ie a HEAD request), allowing the client to view the server Performance requests (ie OPTIONS requests), and requests from the client to the server to replace the content of the specified document (PUT requests) and other request types. The address requested for access can be represented by a uniform resource identifier (URI). For example, the address requested for access can be /v1/createVM. The request header keyword can be the content-Type of the request. The keyword of the request body can be the request success (success). The response header keyword can be the content-Type of the response. The response body keyword can be response success (success). The response status code can be 200, 404, 500, etc. The response status code 200 indicates that the request is successful, the response status code 404 indicates that the requested resource does not exist, and the response status code 500 indicates that the server has an unexpected error.

It should be noted that, in the application scenario shown in FIG. 2, if the filtering unit determines that the service message does not meet the filtering conditions, it can send the service message to the second monitoring unit to send the service message through the second monitoring unit To the second application. In the application scenario shown in FIG. 3 or FIG. 4, when the second monitoring unit is not set, if the filtering unit determines that the service message does not meet the filtering condition, the service message can be directly sent to the second application.

Step 506: The fault injection unit performs the fault injection operation indicated by the fault attribute parameter on the service request, and sends the service message after the fault injection operation is performed to the second monitoring unit.

The faults injected into the service request can be network faults, such as packet loss faults, delay faults, and error packet faults. The packet loss fault refers to the situation that the data packet used to carry the service message is dropped during the process of sending the service message. That is, the fault injection unit injects the packet loss fault into the service request: when the service request includes a data packet, the fault injection unit discards the data packet according to the fault attribute parameter. When the service request includes multiple data packets, the fault The injection unit discards all or part of the multiple data packets according to the fault attribute parameter. Delay failure refers to a situation in which the data packet used to carry the service message is delayed and forwarded during the process of sending the service message. That is, the fault injection unit injects the delay fault into the service request refers to: the fault injection unit sends the data packet of the service message after delaying the time specified by the fault attribute parameter. The error of packet error refers to the situation that the content of the data packet used to carry the business message is modified in the process of sending the business message, for example, the message header, message body and status code in the data packet appear One or more circumstances have been modified. That is, the fault injection unit injects the wrong packet fault into the service request means that the fault injection unit modifies the message in the service message according to the fault attribute parameter.

In addition, the injected faults may also be system resource faults, node faults, database faults, container faults, and other types of faults. For example, the system resource failures may include: central processing unit (CPU) boost failure, memory leak failure, hard disk failure, network failure, abnormal process exit failure, file abnormal failure, file system failure, and system management Failure etc. The node faults may include: abnormal shutdown of the node and abnormal restart of the node. In the embodiments of the present application, the injected fault is not limited to the above-described fault, and may also be other faults, that is, the injected fault is expandable. The program instructions for injecting faults can be written in advance and stored in the storage medium. When faults need to be injected, the faults to be injected can be determined by the fault injection method provided in the embodiments of the present application, and the instructions for injecting corresponding The program instructions of the fault to achieve the corresponding fault injection.

It should be noted that when the fault injection module includes the second monitoring unit, after the fault injection unit injects the fault into the business message, the fault injection unit can send the service message after the fault is injected to the second monitoring unit to inject the fault through the second monitoring unit The subsequent business message is sent to the second application. When the fault injection module does not include the second monitoring unit, after the fault injection unit injects the fault into the service message, the service message after the fault injection may be directly sent to the second application program, that is, step 507 may not be executed.

In addition, before the second monitoring unit sends the fault-injected service message to the second application, the address conversion unit in the fault injection module can also modify the destination address of the service message to be sent by the second monitoring unit Is the address of the second application. For example, still taking the example in step 503 as an example, the address conversion unit may modify the destination address of the service message from 1.2.3.4:8080 to 172.168.200.42:8080. Among them, the command to modify the destination address of the business message from 1.2.3.4:8080 to 172.168.200.42:8080 can be: iptables–t nat–A OUTPUT–d 1.2.3.4–p tcp–m tcp–dport 8080–j DNAT –To_destination 172.168.200.42:8080.

It should also be noted that, when setting the fault attribute parameter, the receiving end of the service message after the injection failure can be set as the second application, so that the functional unit used to send the service message after the injection failure to the second application The receiving end of the service message after the fault is injected can be obtained according to the fault attribute parameter. Therefore, the fault injection module may not include an address conversion unit.

It should also be noted that when the fault injected into the business message is a packet loss fault with a packet loss rate of 100%, there is no need to send the business message to the second application again.

Step 507: The second monitoring unit sends the service message after the failure is injected to the second application, so that the second application process the service message after the failure is injected.

When the business message is a business request, after the second application receives the business request after the fault is injected, it will provide business services based on it, and send a business response to the first application based on the result of the business service provided. It will react after receiving the business response. At this point, it is equivalent to simulating the failure of the second application program. By analyzing the response of the first application program after receiving the business response, the reliability of the first application program when the second application program fails can be realized. Analysis.

When the business message is a business response, the second application will respond after receiving the business response after the fault is injected. At this point, it is equivalent to simulating the failure of the first application program. By analyzing the response of the second application program after receiving the business response, the reliability of the second application program when the first application program fails can be realized. Analysis.

To sum up, in the fault injection method provided by the embodiment of the present application, after the first application generates a business message for accessing the second application, the business message is obtained through the fault injection module, and the fault attribute is executed on the business message. The fault injection operation indicated by the parameter and sending the business message after the fault injection operation is performed to the second application can inject faults into the business message during the sending process of the business message. Compared with the global fault in the related technology, it is more detailed The granularity of injected faults is improved, and the accuracy of reliability testing based on injected faults is effectively improved.

Moreover, since the injecting fault is realized by intercepting the business message during the sending process of the business message, there is no need to adapt the object in which the fault is injected, and the faultless injection can be realized.

At the same time, by using filter conditions to filter business messages, and when the business messages meet the filter conditions, faults are injected into the business messages, so that faults can be injected into the business messages in a targeted manner, and irrelevant tests can be avoided in the test process. It affects other services that do not need to be tested, and realizes accurate fault injection.

The sequence of steps of the fault injection method provided in the embodiments of the present application can be appropriately adjusted, and the steps can also be increased or decreased according to the situation. For example, it is possible to choose whether to execute step 507 according to the situation. Any person familiar with the technical field can easily think of a method of change within the technical scope disclosed in this application, which should be covered by the protection scope of this application, and therefore will not be repeated.

The embodiment of the present application also provides a fault injection device, which can be deployed on a server or computer equipment. The fault injection device may include the fault injection module provided in the embodiment of the present application. For example, the fault injection device may include: a first monitoring unit and a fault injection unit; the first monitoring unit is used to obtain a business message generated by a first application, and the business message is used to access a second application; The fault injection unit is used to obtain the service message from the first monitoring unit, perform the fault injection operation indicated by the fault attribute parameter on the service message, and send the service message after the fault injection operation is performed to the second application.

Optionally, the fault injection device may further include: a filtering unit; the filtering unit is configured to receive the business message sent by the first monitoring unit, filter the business message according to filtering conditions, and filter The conditional business message is sent to the fault injection unit.

Optionally, the fault injection device may further include: a second monitoring unit; at this time, the fault injection unit is specifically configured to send the service message after the fault injection operation is performed to the second monitoring unit; The monitoring unit is configured to send the service message after the fault injection operation is performed to the second application.

Optionally, the fault injection device may further include: an address conversion unit; the address conversion unit is configured to modify the destination address of the service message to be sent by the second monitoring unit to the address of the second application.

Those skilled in the art can clearly understand that, for convenience and concise description, the specific working process of each unit in the fault injection device can refer to the description of the corresponding unit in the foregoing system embodiment, which will not be repeated here.

The embodiment of the present application also provides a storage medium. The storage medium is a non-volatile computer-readable storage medium. When the instructions in the storage medium are executed by the processor, the fault injection module or address in the embodiment of the present application is implemented. The function realized by the conversion module.

The embodiments of the present application also provide a computer program product containing instructions. When the computer program product runs on a computer, the computer executes the functions implemented by the fault injection module or the address conversion module in the embodiments of the present application.

Those of ordinary skill in the art can understand that all or part of the steps in the foregoing embodiments can be implemented by hardware, or by a program instructing relevant hardware to be completed. The program can be stored in a computer-readable storage medium. The storage medium mentioned can be a read-only memory, a magnetic disk or an optical disk, etc.

In the embodiments of the present application, the terms "first", "second" and "third" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance. The term "at least one" refers to one or more, and the term "plurality" refers to two or more, unless specifically defined otherwise.

The term "and/or" in this application is merely an association relationship that describes associated objects, indicating that there can be three types of relationships. For example, A and/or B can mean that there is A alone, and both A and B exist. There are three cases of B. In addition, the character "/" in this text generally indicates that the associated objects before and after are in an "or" relationship.

The above are only optional embodiments of this application and are not intended to limit this application. Any modification, equivalent replacement, improvement, etc. made within the concept and principle of this application shall be included in the protection of this application Within range.

Claims

A fault injection method, characterized in that the fault injection method is applied to a business service system in which a first application program, a second application program and a fault injection module are running, and the method includes:

The first application program generates a business message, and the business message is used to access the second application program;

The fault injection module acquires the business message, and performs the fault injection operation indicated by the fault attribute parameter on the business message;

The fault injection module sends the business message after performing the fault injection operation to the second application.
The method according to claim 1, characterized in that an address conversion module is also operated in the business service system, and before the fault injection module obtains the business message, the method further comprises:

The address conversion module obtains the business message, modifies the destination address of the business message to the address of the fault injection module, and sends the modified business message.
The method according to claim 1 or 2, wherein the fault injection module performing the fault injection operation indicated by the fault attribute parameter on the service message comprises:

The fault injection module performs the fault injection operation indicated by the fault attribute parameter on the service message when the service message meets the filter condition.
The method according to any one of claims 1 to 3, wherein when the service message is a service request, the filter condition involves one or more of the following content: request type, address requested to be accessed, request header Keywords and request body keywords;

When the service message is a response to a service request, the filter condition involves one or more of the following content: response status code, response header keyword, and response body keyword.
The method according to any one of claims 1 to 4, wherein the service message includes multiple data packets, and when the fault injection operation indicated by the fault attribute parameter is a packet loss operation, the fault injection module Performing the fault injection operation indicated by the fault attribute parameter on the business message includes:

The fault injection module discards part or all of the multiple data packets based on the fault attribute parameter.
The method according to any one of claims 1 to 5, wherein when the fault injection operation indicated by the fault attribute parameter is a time-delay operation, the fault injection module performs fault attribute parameter analysis on the service message. The indicated fault injection operations include:

Based on the fault attribute parameter, the fault injection module sends the service message after delaying the time specified by the fault attribute parameter.
The method according to any one of claims 1 to 6, wherein when the fault injection operation indicated by the fault attribute parameter is an error packet operation, the fault injection module performs fault attribute parameter analysis on the service message. The indicated fault injection operations include:

The fault injection module modifies the service message based on the fault attribute parameter.
A business service system, characterized in that the business service system includes: a first server, a second server and a fault injection module;

A first application program running on the first server is used to generate a business message, and the business message is used to access a second application program running on the second server;

The fault injection module is used to obtain the service message, perform the fault injection operation indicated by the fault attribute parameter on the service message; send the service message after the fault injection operation is performed to the second server running on the second server application.
The system according to claim 8, wherein the business service system further comprises: an address conversion module;

The address conversion module is used to obtain the business message, modify the destination address of the business message to the address of the fault injection module, and send the modified business message.
The system according to claim 8 or 9, wherein:

The fault injection module is specifically configured to perform the fault injection operation indicated by the fault attribute parameter on the service message when the service message meets the filter condition.
The system according to any one of claims 8 to 10, wherein when the service message is a service request, the filter condition involves one or more of the following content: request type, address requested to be accessed, request header Keywords and request body keywords;

When the service message is a response to a service request, the filter condition involves one or more of the following content: response status code, response header keyword, and response body keyword.
The system according to any one of claims 8 to 11, wherein the service message includes multiple data packets, and when the fault injection operation indicated by the fault attribute parameter is a packet loss operation, the fault injection module Specifically, the fault injection module discards part or all of the multiple data packets based on the fault attribute parameter.
The system according to any one of claims 8 to 12, wherein when the fault injection operation indicated by the fault attribute parameter is a delayed operation, the fault injection module is specifically configured to be based on the fault attribute parameter, After delaying the time specified by the fault attribute parameter, the service message is sent.
The system according to any one of claims 8 to 13, wherein when the fault injection operation indicated by the fault attribute parameter is an error packet operation, the fault injection module is specifically configured to be based on the fault attribute parameter, Modify the message in the service message.
A fault injection device, characterized in that the device comprises: a first monitoring unit and a fault injection unit;

The first monitoring unit is used to obtain a business message generated by a first application, and the business message is used to access a second application;

The fault injection unit is configured to obtain the business message from the first monitoring unit, perform the fault injection operation indicated by the fault attribute parameter on the business message, and send the business message after the fault injection operation is performed to the The second application.
The device according to claim 15, wherein the device further comprises: a filtering unit;

The filtering unit is configured to receive the business message sent by the first monitoring unit, filter the business message according to the filtering condition, and send the business message that meets the filtering condition to the fault injection unit.
The device according to claim 15 or 16, wherein the device further comprises: a second listening unit;

The fault injection unit is specifically configured to send the business message after the fault injection operation is performed to the second monitoring unit;

The second monitoring unit is configured to send the service message after the fault injection operation is performed to the second application.
The device according to any one of claims 15 to 17, wherein the device further comprises: an address conversion unit;

The address conversion unit is configured to modify the destination address of the service message to be sent by the second monitoring unit to the address of the second application program.