CN112506729A

CN112506729A - Fault simulation method and device

Info

Publication number: CN112506729A
Application number: CN201910870621.8A
Authority: CN
Inventors: 邹翔; 于超; 李志勇
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2019-09-16
Filing date: 2019-09-16
Publication date: 2021-03-16

Abstract

The application discloses a fault simulation method and a device, wherein the method comprises the following steps: the terminal responds to the fault simulation initiating instruction and sends a fault simulation request to the server; the server sends the fault association parameter and the fault identification to a fault issuing object; the fault issuing object determines a target fault execution script command corresponding to the fault identification in a locally deployed fault agent component; and the fault issuing object locally simulates the target fault based on the target fault execution script command and the fault associated parameters. By utilizing the technical scheme provided by the application, the fault simulation based on software can be realized, the fault simulation efficiency and the density of fault simulation operation are improved, the simulation requirements of various faults are flexibly met, and the richness of a fault simulation scene is effectively improved.

Description

Fault simulation method and device

Technical Field

The present application relates to the field of internet communication technologies, and in particular, to a fault simulation method and apparatus.

Background

With the rapid development of computer and internet technologies, people's daily life has a higher and higher degree of dependence on computers and the internet, and the severity of the influence caused by various equipment faults and network faults is gradually increased. The fault simulation can effectively analyze the fault and also becomes one of important means for solving various faults.

At present, when fault simulation is carried out, fault simulation needs to be carried out by combining hardware, the fault simulation by combining the hardware often needs a user to manually control related simulation operation, the problems of long time consumption, scattered fault simulation operation and the like are caused, and due to the fact that faults are various, the richness of the fault simulation cannot be guaranteed due to the fact that the fault simulation by combining the hardware is often caused by the reasons of cost and the like. Therefore, there is a need to provide a more reliable or efficient solution.

Disclosure of Invention

The application provides a fault simulation method and device, which can realize software-based fault simulation, improve the fault simulation efficiency and the density of fault simulation operation, flexibly meet the simulation requirements of various faults and effectively improve the richness of fault simulation scenes.

In one aspect, the present application provides a fault simulation method, including:

receiving a fault simulation request sent by a terminal, wherein the fault simulation request comprises a fault issuing object, a fault identifier of a target fault and a fault association parameter;

sending the fault correlation parameters and the fault identification to the fault issuing object so that the fault issuing object determines a target fault execution script command corresponding to the fault identification in a locally deployed fault agent component, wherein the fault agent component comprises a fault execution script command simulating multiple faults; and locally simulating the target fault based on the target fault execution script command and the fault associated parameters.

In some embodiments, the target fault includes at least one of:

disk failures, stand-alone failures, component failures, network failures, and resource failures.

Another aspect provides a fault simulation apparatus, the apparatus comprising:

the fault simulation system comprises a fault simulation request receiving module, a fault simulation module and a fault analysis module, wherein the fault simulation request receiving module is used for receiving a fault simulation request sent by a terminal, and the fault simulation request comprises a fault issuing object, a fault identifier of a target fault and a fault correlation parameter;

the data sending module is used for sending the fault association parameters and the fault identifications to the fault issuing object so that the fault issuing object determines a target fault execution script command corresponding to the fault identifications in a locally deployed fault agent component, and the fault agent component comprises a fault execution script command simulating multiple faults; and locally simulating the target fault based on the target fault execution script command and the fault associated parameters.

Another aspect provides a fault simulation server comprising a processor and a memory, the memory having stored therein at least one instruction, at least one program, set of codes, or set of instructions, which is loaded and executed by the processor to implement a fault simulation method as described above.

In another aspect, the present application provides a fault simulation method, including:

receiving a fault associated parameter and a fault identifier sent by a server when receiving a fault simulation request which is sent by a terminal and comprises a fault sending object, a fault identifier of a target fault and a fault associated parameter;

determining a target fault execution script command corresponding to the fault identification in a locally deployed fault agent component, wherein the fault agent component comprises a fault execution script command simulating multiple faults;

and locally simulating the target fault based on the target fault execution script command and the fault correlation parameter.

In some embodiments, the target fault includes at least one of:

In some embodiments, when the target fault comprises network delay jitter in a network fault, the target fault execution script command comprises a script command that invokes a flow control tool to execute a network delay jitter simulation, and the fault-associated parameters comprise an ingress and egress flow threshold and a network flow rate;

the locally simulating the target fault based on the target fault execution script command and the fault associated parameters comprises:

calling a flow control tool corresponding to the network delay jitter, and binding a preset network card;

setting an access flow threshold value and a network flow rate in the preset network card based on the fault correlation parameters in the fault correlation parameters including the access flow threshold value and the network flow rate;

when the target fault comprises a network packet loss in the network fault, the target fault execution script command comprises a script command for calling a firewall tool to execute network packet loss simulation, and the fault associated parameter comprises data packet transmission quantity;

locally simulating the target fault based on the target fault execution script command and the fault associated parameters comprises:

and calling a firewall tool corresponding to the network packet loss, and controlling the transmission quantity of the data packet of the fault issuing object to be the transmission quantity of the data packet in the fault correlation parameter.

In some embodiments, the fault proxy component further comprises a plurality of failed fault recovery script commands;

the method further comprises the following steps:

receiving a timing recovery instruction of the target fault, which is sent by the server, wherein the timing recovery instruction comprises recovery time;

after locally simulating the target fault based on the target fault execution command and the fault-associated parameters, the method further comprises:

determining a fault recovery script command corresponding to the target fault in the fault agent component;

when the recovery time is up, performing fault recovery processing on the target fault based on the fault recovery script command;

the method further comprises the following steps:

receiving a recovery instruction of the target recovery fault sent by a server, and determining a fault recovery script command corresponding to the target recovery fault in the fault proxy component after locally simulating the target fault based on the target fault execution command and the fault associated parameter; and performing fault recovery processing on the target recovery fault based on the fault recovery script command.

Another aspect provides a fault simulation apparatus, the apparatus comprising:

the data sending module is used for receiving the fault associated parameters and the fault identifications sent by the server when receiving a fault simulation request which is sent by the terminal and comprises a fault sending object, a fault identification of a target fault and the fault associated parameters;

the target fault execution script command determining module is used for determining a target fault execution script command corresponding to the fault identifier in a locally deployed fault agent component, and the fault agent component comprises a fault execution script command simulating multiple faults;

and the target fault simulation module is used for executing a script command and the fault correlation parameters based on the target fault and locally simulating the target fault.

Another aspect provides a fault simulation apparatus comprising a processor and a memory, the memory having stored therein at least one instruction, at least one program, set of codes, or set of instructions, which is loaded and executed by the processor to implement a fault simulation method as described above.

Another aspect provides a computer readable storage medium having stored therein at least one instruction, at least one program, set of codes, or set of instructions that is loaded and executed by a processor to implement a fault simulation method as described above.

The fault simulation method and the fault simulation device have the following technical effects:

when fault simulation is needed, the server forwards the fault association parameters and the target fault to the fault issuing object with the fault agent component, so that the fault issuing object can flexibly meet the simulation requirements of various faults by executing script commands on the faults of various faults in the fault agent component, the richness of a fault simulation scene is effectively improved, the software-based fault simulation is realized, and the fault simulation efficiency and the fault simulation operation intensity are improved. And by means of component deployment, remote copy can be directly carried out within the reachable range of the network, and convenient and fast terminal deployment is realized.

Drawings

In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a schematic diagram of an application environment provided by an embodiment of the present application;

fig. 2 is a schematic flow chart of a fault simulation method according to an embodiment of the present application;

FIG. 3 is a schematic interface diagram of a fault simulation platform provided herein;

FIG. 4 is a schematic diagram of a selection interface for a class type fault according to an embodiment of the present application;

FIG. 5 is a schematic interface diagram of another fault simulation platform provided in an embodiment of the present application;

fig. 6 is a schematic diagram of a fault resource pool provided in an embodiment of the present application;

FIG. 7 is a schematic flow chart diagram of another fault simulation method provided in the embodiments of the present application;

fig. 8 is a schematic flowchart of a fault issuing object locally simulating the target fault based on the target fault execution script command and the fault associated parameter according to the embodiment of the present application;

FIG. 9 is a schematic diagram of a fault routing inspection report provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of a fault report of a kick plate fault according to an embodiment of the present disclosure;

FIG. 11 is a schematic flow chart diagram of another fault simulation method provided in the embodiments of the present application;

FIG. 12 is a schematic interface diagram of a fault simulation platform for displaying fault records according to an embodiment of the present disclosure;

fig. 13 is a schematic structural diagram of a fault simulation apparatus according to an embodiment of the present application;

FIG. 14 is a schematic structural diagram of another fault simulation apparatus provided in an embodiment of the present application;

fig. 15 is a hardware block diagram of a server in a fault simulation method according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or server that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Referring to fig. 1, fig. 1 is a schematic diagram of an application environment according to an embodiment of the present application, and as shown in fig. 1, the application environment at least includes a terminal 01, a server 02, and a failure issuing object 03.

In this embodiment, the terminal 01 may include a smart phone, a desktop computer, a tablet computer, a notebook computer, a digital assistant, an Augmented Reality (AR)/Virtual Reality (VR) device, a smart wearable device, and other types of physical devices, and may also include software running in the physical devices, such as a virtual machine. The operating system running on the network node in the embodiment of the present application may include, but is not limited to, an android system, an IOS system, linux, Unix, windows, and the like. The terminal 01 includes a User Interface (UI) layer, and the terminal 01 provides a fault simulation issue service to the outside through the UI layer, and sends data required for simulating a fault to the server 02 based on an Application Programming Interface (API).

In this embodiment, the server 02 may include a server operating independently, or a distributed server, or a server cluster composed of a plurality of servers. The server 02 may comprise a network communication unit, a processor and a memory, etc. Specifically, the server 02 may be configured to deploy a fault agent component to a fault delivery object, and forward data required for simulating a fault to the fault delivery object configured with the fault agent component. Specifically, the server may be further configured to record running data in the fault simulation process.

In this embodiment, the fault issuing object 03 may be a device for simulating a fault, specifically, the fault issuing object includes entity devices of a smart phone, a desktop computer, a tablet computer, a notebook computer, a digital assistant, an Augmented Reality (AR)/Virtual Reality (VR) device, an intelligent wearable device, and the like, and may also include software running in the entity devices, such as a virtual machine and the like. It may also comprise a server running independently, or a distributed server, or a server cluster consisting of a plurality of servers. Specifically, the fault delivery object executes a specific fault simulation task based on data required for deploying the fault agent component and simulating the fault.

While a specific embodiment of a fault simulation method of the present application is described below, fig. 2 is a schematic flow chart of a fault simulation method provided by an embodiment of the present application, and the present specification provides the method operation steps as described in the embodiment or the flow chart, but may include more or less operation steps based on conventional or non-inventive labor. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. In practice, the system or server product may be implemented in a sequential or parallel manner (e.g., parallel processor or multi-threaded environment) according to the embodiments or methods shown in the figures. Specifically, as shown in fig. 2, the method may include:

s201: and the terminal responds to the fault simulation initiating instruction and sends a fault simulation request to the server.

In this embodiment of the present specification, the fault simulation request may include a fault issuing object, a fault identifier of a target fault, and a fault association parameter. Specifically, the target failure may include at least one of: disk failures, stand-alone failures, component failures, network failures, and resource failures. In this embodiment of the present specification, the identification information of the fault may be used as a unique identifier of the fault, and is used to distinguish different faults, specifically, the identification information may include, but is not limited to, a character string such as a fault name.

Specifically, the disk failure may include at least one of: SCSI (Small Computer System Interface) kicking (a disk with SCSI Interface protocol removed), OFFLINE kicking (a disk with dropped lines removed), EIO (Error Input/Output, inconsistent disk Input/Output), Hang disk (disk false death), slow disk (Input/Output processing time on disk is too long); specifically, the single machine fault may include at least one of: equipment power failure, equipment restart and system disk error; specifically, the component failure may include at least one of: a process kill, a shared memory kill, and a file read-only; specifically, the network failure may include at least one of: network packet loss and network delay jitter; specifically, the resource failure may include at least one of: CPU overhigh, memory overhigh and time jump.

In embodiments of the present description, the fault-related parameters may include data required to simulate the target fault. In a specific embodiment, taking a network packet loss in a network failure as an example, the data required for simulating the network packet loss may include: the network packet loss rate is determined according to the network packet loss rate, the network packet type, the source end or the destination end, the network packet loss rate and the network packet loss machine IP.

In practical application, the terminal can provide a UI interface, and a user triggers a fault simulation initiation instruction based on the UI interface, so that the sex server sends a fault simulation request. Specifically, as shown in fig. 3, fig. 3 is an interface schematic diagram of a fault simulation platform provided in the present application. As can be seen from fig. 3, a menu area for selecting a fault, a fault resource pool, a menu area for triggering a function, a setting area for target device information, and a setting area for fault-related parameters are set in the interface of the fault simulation platform.

Specifically, the fault displayed in the menu area for fault selection may be a primary fault type, and after reaching the primary fault type through mouse hover, a fault of a related secondary subtype may occur, and specifically, a selection interface of the secondary subtype fault is shown in fig. 4, and the related fault may be selected in the selection interface of the secondary subtype fault.

Specifically, the menu area triggered by the function may include information of a task ID (identification) of a single failure, a batch task ID of a batch failure, and buttons for adding a failure pool, recovering a failure, recovering a batch failure, initiating a failure, reporting a failure, and the like. Specifically, the information of the task ID and the batch task ID may be directly input and set by a user, or may be automatically generated after the fault simulation request is sent. The button to join the failure pool is used to join one or more failures to be simulated to the failure resource pool. The failure recovery button may be used to trigger the recovery of a certain failure; the bulk fail-over button may be used to trigger the recovery of a variety of failures. A fault initiation button may be used to trigger a fault simulation request and a fault report button may be used to trigger a query for a fault report.

In particular, the pool of failed resources may be used to expose one or more failures that need to be simulated.

Specifically, the setting area of the target device information may be used to set the relevant information of the fault issue object (target device), and specifically, the relevant information of the fault issue object (target device) may include a target device IP, a target device port, a target device account, and a target device password.

Specifically, the setting region of the fault-related parameter may be used for setting data required for performing the fault simulation. Specifically, the related setting may be performed in combination with the selected different faults.

In practical application, after a user selects a fault through a menu area provided with fault selection in an interface of a fault simulation platform, relevant information and parameters are set in a setting area of target equipment information and a setting area of fault associated parameters, then, a fault to be simulated is added into a fault pool based on clicking of a button for adding into the fault pool of a menu area triggered by a function, and finally, a fault simulation initiating instruction is triggered through clicking of a button for initiating the fault of the menu area triggered by the function, and a fault simulation request is sent to a server.

Further, as shown in fig. 3, the interface of the fault simulation platform may further be provided with a fault record area for displaying fault simulation pipeline data. Specifically, the fault simulation flow data may include a task ID, a fault type (i.e., a fault name), target device information (i.e., a target device IP, a target device port, a target device account, and a target device password), fault information (i.e., fault association parameters in the fault simulation process), fault initiation time, fault modification time (time for completing fault simulation after a fault simulation request is issued and returning a successful or failed fault simulation flow record to the terminal), a fault operator, and time consumed by a task (time for fault simulation).

Further, as shown in fig. 3, the interface of the fault simulation platform may also be provided with a setting area of a fault operator, and the fault operator issues a fault simulation user manually.

Further, as shown in fig. 5, after a certain fault (for example, network packet loss) is selected on the selection interface of the second-class subtype fault, a parameter related to the network packet loss is displayed in a setting area of a fault-related parameter, and a specific parameter value may be preset or may be input by a user; in addition, by clicking a button for joining the failure pool, a failure (network packet loss) can be joined to the failure resource pool.

Specifically, as shown in fig. 6, after a plurality of faults are added to the fault resource pool in sequence, the simulation order of the faults may be adjusted by dragging a certain fault in the fault resource pool, or a certain fault may be deleted by clicking a fork, specifically, in fig. 6, the fault resource pool includes three faults, namely para-svr _1 (device forced power down) para-process-kill _2 (process kill) and para-iscsi _3 (sci kick panel).

Further, as shown in fig. 5, after the fault is added to the fault resource pool, a button initiated by the fault may be clicked to trigger sending of a fault simulation request to the server.

S203: and the server sends the fault association parameters and the fault identification to the fault issuing object.

In this embodiment of the present description, after receiving a fault simulation request sent by a terminal, a server may determine whether a fault issuing object has deployed a fault proxy component, and when the fault issuing object deploys the fault proxy component, may send a fault association parameter and a fault identifier to the fault issuing object. On the contrary, when the failure delivery object does not deploy the failure agent component, as shown in fig. 7, the method further includes:

s209: and the server deploys the fault agent component on the fault issuing object.

Specifically, in practical application, the deployment of the fault agent component may copy the fault agent component to the fault delivery object remotely through paramiko. Specifically, paramiko is a device that connects to a remote server based on SSH (Secure Shell protocol) and performs related operations (SSHClient component and SFTPClinet component, i.e. one is remote connection and one is upload and download service) to perform command or file operations on the remote server. Specifically, the SSHClient component and the target device information (information of the fault delivery object) may be used to establish a remote connection with the fault delivery object, and then the SFTPClinet component may be used to upload the fault agent component to the fault delivery object. Subsequently, when the server sends a related command to the fault delivery object, the fault delivery object can be executed by using the fault agent component.

In embodiments of the present description, the fault agent component may include fault execution script commands that simulate various faults. In practical application, during fault simulation, for example, single machine fault, component fault, network fault resource fault and the like, corresponding execution script commands can be directly relied on; other fault simulations, such as disk failures and/or network failures, may require reliance on third party tools. Correspondingly, the fault execution script command can also comprise a script command for calling a third-party tool to execute fault simulation.

S205: and the fault issuing object determines a target fault execution script command corresponding to the fault identification in the locally deployed fault agent component.

In this embodiment of the present specification, the target fault execution script command may be an execution script command for simulating a target fault, or may be a script command for scheduling a target fault execution tool corresponding to the target fault to execute target fault simulation.

In practical application, the fault execution script command simulating multiple faults in the fault agent component can be provided with identification information of corresponding faults so as to distinguish the fault execution script commands.

S207: and the fault issuing object locally simulates the target fault based on the target fault execution script command and the fault associated parameters.

In an embodiment of this specification, when the target failure includes a disk failure, the target failure execution script command includes a script command for calling a kernel monitoring and tracking tool to execute disk failure simulation, and the failure associated parameter includes a disk failure associated parameter;

correspondingly, as shown in fig. 8, the locally simulating the target fault by the fault issuing object based on the target fault execution script command and the fault associated parameter includes:

s2071: the fault issuing object calls the kernel monitoring and tracking tool to acquire kernel key information in an original script command corresponding to the fault issuing object and the target fault;

s2073: modifying parameters in the kernel key information by the fault issuing object based on the disk fault associated parameters;

s2075: and the fault issuing object executes the original script command after the kernel key information is modified.

In a specific embodiment, a Hang disk in a disk failure is taken as an example, and the Hang disk refers to a phenomenon of 'false death' caused by too high load of a device, so that if the device cannot process IO and an IO request is continuously accumulated, the phenomenon of an IO Hang can be simulated. In this embodiment, the function (nvme _ process _ cq) that controls the processing completion queue may exit before writing the register, specifically, when the register pointer head is not moved and phase (io completion phase, disk state cqe. Specifically, kernel key information nvmeq in an original script command corresponding to the Hang disk is obtained, cqe.

When the target fault comprises network delay jitter in the network fault, the target fault execution script command comprises a script command for calling a flow control tool to execute network delay jitter simulation, and the fault associated parameters comprise an access flow threshold value and a network flow rate;

the fault issuing object executes a script command and the fault associated parameters based on the target fault, and locally simulating the target fault comprises:

the fault issuing object calls a flow control tool corresponding to the network delay jitter and binds a preset network card;

and the fault issuing object sets the access flow threshold value and the network flow rate in the preset network card based on the fault associated parameters in the fault associated parameters including the access flow threshold value and the network flow rate.

and the fault issuing object calls a firewall tool corresponding to the network packet loss, and the transmission quantity of the data packet of the fault issuing object is controlled to be the transmission quantity of the data packet in the fault correlation parameter.

When the target faults comprise component faults, single machine faults and resource faults, the simulation of the target faults can be realized without depending on a third-party tool, and the corresponding target fault execution script commands can comprise target fault execution script commands.

Specifically, when the target fault includes a process kill in the component fault, the target fault execution script command may include a process kill command, the fault association parameter may include process entry identification information, and accordingly, a process that needs kill may be found based on the process entry identification information, and then, the process kill command is used to kill the corresponding process.

Specifically, when the target failure includes a shared memory kill in a component failure, the target failure execution script command may include an ipcs command (the ipcs command is used to write some information about the active interprocess communication facility to the standard output) and an iprm (shared memory delete) command; the fault associated parameter may include a key value of the shared memory, specifically, all the shared memories in the current environment may be checked by using an ipcs command, and the iprm command is executed by using the key value of the shared memory in the fault associated parameter, thereby implementing deletion and cleaning of the shared memories.

Specifically, when the target failure includes that a file in the component failure is read only, the target failure execution script command may include a chmod (which is a computer function, and functions to change read-write permission settings of the file) command; the fault-associated parameters may include identification information of the target read-only file. Correspondingly, the target read-only file can be found based on the identification information of the target read-only file in the fault correlation parameter, and the target read-only file is set to be read-only through the chmod command, so that the fault of file write failure is realized.

Specifically, when the target failure includes a device power failure in a single machine failure, the target failure execution script command may include a forced shutdown command (ordinary shutdown, requiring a program to be stopped first and then shutdown); the fault-associated parameters may include power port information. Correspondingly, a forced shutdown command can be executed through the power port corresponding to the power port information in the fault correlation parameter.

Specifically, when the target failure includes a device restart in a single machine failure, the target failure execution script command may include a reboot command. Accordingly, the restart operation may be performed through reboot.

Specifically, when the target failure includes a system disk error in a single machine failure, the target failure execution script command may include a dd (a command on a Unix and Unix-like system, and a main function of the command is to convert and copy a file) command, the failure-related parameter may correspond to the system disk identification information, and the corresponding system disk may be queried according to the system disk identification information in the failure-related parameter, and then the system disk write coverage is performed by using the dd command, thereby causing a system disk failure in which the system disk metadata is written out.

When the target fault includes that the CPU in the resource fault is too high, the total core number of the CPU of the current physical machine may be calculated, and then the core number of the CPU that needs to be operated may be calculated according to the percentage of the CPU that is full in the fault-related parameter, for example, if the CPU needs to be full at 20%, the core number of the CPU that needs to be operated is 20% of the total core number, and then, for the core number of the CPU that needs to be operated, data is read from/dev/zero through dd command, and written into/dev/null, and the CPU core that is currently bound is full through this way.

When the target fault includes that the memory in the resource fault is too high, taking an operating system with a fault issuing object as linux as an example, the current physical total memory can be calculated firstly through free (a function of applying for memory space and releasing memory space), then the percentage of the memory required to be filled in the fault association parameter is calculated, the memory value required to be filled is calculated, and the memory is continuously consumed through a script of a memory malloc (dynamic memory allocation, so-called free linked list) command for connecting available memory blocks into a long list until the memory value reaches an expected value.

As can be seen from the technical solutions provided by the embodiments of the present specification, in the embodiments of the present specification, when fault simulation is required, the server forwards the fault association parameter and the target fault to the fault issuing object deployed with the fault agent component, so that the fault issuing object can flexibly meet the simulation requirements of various faults by executing script commands on multiple faults in the fault agent component, the richness of a fault simulation scenario is effectively improved, software-based fault simulation is implemented, and the fault simulation efficiency and the density of fault simulation operations are improved. And by means of component deployment, remote copy can be directly carried out within the reachable range of the network, and convenient and fast terminal deployment is realized.

Further, in practical application, as shown in fig. 3, the menu area for selecting a fault may also be provided with a fault polling button, and correspondingly, the area for triggering a function may also be provided with a fault polling button; specifically, the fault routing inspection may check which faults are simulated by some fault delivery objects which have already simulated faults, and whether the faults have already been recovered. Correspondingly, the method further comprises the following steps:

1) responding to a fault routing inspection instruction, and sending a fault routing inspection request to a server, wherein the fault routing inspection request comprises a task identifier of a target routing inspection fault;

2) the server determines fault simulation flow data of the target inspection fault based on the task identifier of the target inspection fault;

3) the server generates a fault polling report based on the fault simulation flow data of the target polling fault;

4) and the server sends the fault patrol inspection report to the terminal.

In this embodiment of the present description, the target inspection fault may include one or more simulated faults, and accordingly, the task identifier of the target inspection fault may include a task identifier of a single fault to be simulated, or may be a task identifier of a fault to be simulated in batch. Specifically, as shown in fig. 3, after the user selects the fault inspection through the menu area provided with the fault selection in the interface of the fault simulation platform, the user inspects the task identifier of the fault at the input target of the menu area triggered by the function, and clicks the button of the fault inspection to trigger the fault inspection instruction, and sends the fault inspection request to the server. In a specific embodiment, as shown in fig. 9, fig. 9 is a schematic diagram of a fault inspection report provided in an embodiment of the present application.

In other embodiments, to facilitate analysis of the fault simulation situation by the associated personnel, the method further comprises:

1) a terminal sends a generation request of a fault analysis report to a server, wherein the generation request comprises a task identifier of a target fault;

2) the server determines fault simulation flow data of the target fault based on the task identification;

3) the server carries out fault analysis on the target fault based on the fault simulation running water data to obtain a fault analysis report;

4) and the server sends the fault analysis report to the terminal.

In a specific embodiment, taking a certain kick plate fault as an example, as shown in fig. 10, the fault analysis report may include an execution condition of each step in the fault simulation process, and each step may be checked by clicking a refresh. Specifically, the fault issuing object may be a distributed system, where a cell is a slave node in the distributed system, and a master is a master node in the distributed system. In a specific embodiment, the distributed system may be a blockchain system, and in particular, the blockchain system may be formed by a plurality of nodes (any form of computing devices in an access network, such as servers and user terminals) and clients, a Peer-To-Peer (P2P) network is formed between the nodes, and the P2P Protocol is an application layer Protocol running on top of a Transmission Control Protocol (TCP). In a distributed system, any machine, such as a server or a terminal, can join to become a node, and the node comprises a hardware layer, a middle layer, an operating system layer and an application layer.

In other embodiments, the fault proxy component may further include a plurality of failed fault recovery script commands; correspondingly, the method may further include:

the server sends a timing recovery instruction of the target fault to the fault issuing object, wherein the timing recovery instruction comprises recovery time;

after the fault delivery object locally simulates the target fault based on the target fault execution command and the fault associated parameter, the method further comprises:

1) the fault issuing object determines a fault recovery script command corresponding to the target fault in the fault agent component;

2) and when the recovery time is up, the fault issuing object carries out fault recovery processing on the target recovery fault based on the fault recovery script command.

Specifically, a task scanning tool can be set through a crontab (command for periodically executing a program) timer, the task scanning tool can continuously scan a fault task of the current equipment (fault issuing object), and a fault recovery script command can be executed after the fault reaches the default recovery time, so that the fault can be recovered;

in addition, in some embodiments, if the user needs to recover in advance, the fault needing to be recovered can be specified through the task identification of the terminal. Correspondingly, the method further comprises the following steps:

1) and the terminal responds to the fault recovery instruction and sends a fault recovery request to the server, wherein the fault recovery request comprises a task identifier of the target recovery fault.

2) And the server determines a fault issuing object corresponding to the target recovery fault based on the task identifier and sends a recovery instruction of the target recovery fault to the fault issuing object.

1) the fault issuing object determines a fault recovery script command corresponding to the target recovery fault in the fault proxy component;

2) and the fault issuing object carries out fault recovery processing on the target recovery fault based on the fault recovery script command.

Specifically, the target recovery fault may include one or more simulated faults, and the task identifier of the target recovery fault may include a task identifier of a single fault to be simulated, or may be a task identifier of a fault to be simulated in batches.

In other embodiments, in order to facilitate subsequent trace analysis of the fault simulation, after the terminal sends the fault simulation request to the server, as shown in fig. 11, the method may further include:

s211: generating fault sending pipeline data by the terminal;

s213: the terminal sends the failure issuing stream data to the server;

correspondingly, after the fault issuing object executes the script command and the fault associated parameter based on the target fault and locally simulates the target fault, with reference to fig. 11, the method further includes:

s215: the fault issuing object records fault execution flow data;

s217: the fault issuing object sends the fault execution stream data to the server;

s219: the server generates fault simulation running water data of the target fault based on the fault issuing running water data and the fault execution running water data

S221: and the server sends the fault simulation running water data to the terminal.

Specifically, the failure issuing pipeline data may include, but is not limited to, a failure operator, a failure initiating time, a task ID, and a failure type. Specifically, the fault execution pipeline data may include, but is not limited to, task ID, fault type, fault modification time, task elapsed time, target device information, and fault information.

In practical application, when the terminal displays the fault simulation running water data, the fault information in the fault simulation running water data can be obtained, and when the value of the fault associated parameter in the fault information is consistent with the value of the set fault associated parameter when the fault simulation request is issued, the fault simulation can be determined to be successful; otherwise, the fault simulation fails. Correspondingly, for the failure simulation success and failure, different colors can be adopted to display the failure simulation running water data and the like for distinguishing.

In a specific embodiment, as shown in fig. 12, fig. 12 is a schematic interface diagram of a fault simulation platform for displaying fault records according to an embodiment of the present application. Specifically, after the fault simulation is issued, a relevant flow is displayed in the fault record, as can be seen from fig. 12, for the fault batch task, a batch task ID and an individual task ID may be set for each task, where the fault simulation task with the task ID of 190812 is a hang disk fault, and it may be determined that the fault issuing device is not passed through or the server is abnormal because the fault modification time of the fault simulation pipeline is empty, so that the fault simulation request is not issued to the fault issuing device. The task ID is 19075, the failure simulation task with the batch task ID of 201963 is hang disk failure, and the failure simulation task is successful, which takes 3 seconds. The fault simulation task with task ID of 190754, batch task ID of 201963 was a process kill fault, and the fault simulation task failed.

In addition, it should be noted that information related to the latest fault simulation can be generally displayed in the parameter setting.

According to the technical scheme provided by the embodiment of the specification, when fault simulation is needed, the server forwards the fault association parameters and the target fault to the fault issuing object with the fault agent component, so that the fault issuing object can flexibly meet the simulation requirements of various faults by executing script commands on the faults of various faults in the fault agent component, the richness of a fault simulation scene is effectively improved, the software-based fault simulation is realized, and the fault simulation efficiency and the fault simulation operation density are improved. And by means of component deployment, remote copy can be directly carried out within the reachable range of the network, and convenient and fast terminal deployment is realized. In addition, multiple functions of convenient and quick fault tracking (routing inspection), fault recovery, fault analysis and the like can be realized. And the user can realize fault simulation by combining the UI and the API, and the operation is convenient and fast.

An embodiment of the present application further provides a fault simulation apparatus, as shown in fig. 13, the apparatus includes:

a fault simulation request receiving module 1310, configured to receive a fault simulation request sent by a terminal, where the fault simulation request includes a fault issuing object, a fault identifier of a target fault, and a fault association parameter;

a data sending module 1320, configured to send the fault association parameter and the fault identifier to the fault issue object, so that the fault issue object determines a target fault execution script command corresponding to the fault identifier in a locally deployed fault agent component, where the fault agent component includes a fault execution script command for simulating multiple faults; and locally simulating the target fault based on the target fault execution script command and the fault associated parameters.

In some embodiments, the apparatus further comprises:

and the fault agent component deployment module is used for deploying the fault agent component in the fault delivery object when the fault delivery object is not deployed with the fault agent component.

the device further comprises:

a timing recovery instruction sending module, configured to send a timing recovery instruction of the target fault to the fault issue object, where the timing recovery instruction includes recovery time, so that the fault issue object determines a fault recovery script command corresponding to the target fault in the fault agent component after locally simulating the target fault based on the target fault execution command and the fault associated parameter; when the recovery time is up, performing fault recovery processing on the target fault based on the fault recovery script command;

the device further comprises:

a fault recovery request receiving module, configured to receive a fault recovery request sent by the terminal, where the fault recovery request includes a task identifier of a target recovery fault;

a recovery instruction sending module, configured to determine, based on the task identifier, a fault issue object corresponding to the target recovery fault, and send a recovery instruction of the target recovery fault to the fault issue object, so that the fault issue object determines, after locally simulating the target fault based on the target fault execution command and the fault associated parameter, a fault recovery script command corresponding to the target recovery fault in the fault agent component; and performing fault recovery processing on the target recovery fault based on the fault recovery script command.

In some embodiments, the apparatus further comprises:

the first running water data receiving module is used for receiving the running water data under the fault sent by the terminal;

the second running water data receiving module is used for receiving fault execution running water data sent by the fault issuing object;

the fault simulation running water data generation module is used for generating fault simulation running water data of the target fault based on the fault issuing running water data and the fault execution running water data;

and the fault simulation flow data sending module is used for sending the fault simulation flow data to the terminal.

In some embodiments, the apparatus further comprises:

a generation request receiving module, configured to receive a generation request of a fault analysis report sent by the terminal to a server, where the generation request includes a task identifier of a target fault;

the first flow data determining module is used for determining fault simulation flow data of the target fault based on the task identification;

the fault analysis module is used for carrying out fault analysis on the target fault based on the fault simulation running water data to obtain a fault analysis report;

and the fault analysis report sending module is used for sending the fault analysis report to the terminal.

In some embodiments, the apparatus further comprises:

the fault routing inspection request receiving module is used for receiving the sent fault routing inspection request, and the fault routing inspection request comprises a task identifier of a target routing inspection fault;

the second running water data determining module is used for determining fault simulation running water data of the target inspection fault based on the task identifier of the target inspection fault;

the fault routing inspection report generating module is used for generating a fault routing inspection report based on the fault simulation flow data of the target routing inspection fault;

and the fault inspection report sending module is used for sending the fault inspection report to the terminal.

In some embodiments, the target fault includes at least one of:

The device and method embodiments in the device embodiment are based on the same application concept.

The embodiment of the present application provides a fault simulation server, which includes a processor and a memory, where the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the fault simulation method provided in the above method embodiment.

An embodiment of the present application further provides another fault simulation apparatus, as shown in fig. 14, the apparatus includes:

the data sending module 1410 is configured to receive the fault association parameter and the fault identifier sent by the server when receiving the fault simulation request which is sent by the terminal and includes the fault sending object, the fault identifier of the target fault, and the fault association parameter;

a target fault execution script command determining module 1420, configured to determine a target fault execution script command corresponding to the fault identifier in a locally deployed fault agent component, where the fault agent component includes a fault execution script command simulating multiple faults;

and a target fault simulation module 1430, configured to locally simulate the target fault based on the target fault execution script command and the fault associated parameter.

In some embodiments, the target fault includes at least one of:

In some embodiments, when the target failure comprises a disk failure, the target failure execution script command comprises a script command that invokes a kernel monitoring and tracking tool to execute disk failure simulation, and the failure associated parameters comprise disk failure associated parameters;

the target fault simulation module 1430 includes:

the first tool calling module is used for calling the kernel monitoring and tracking tool and acquiring kernel key information in an original script command corresponding to the fault issuing object and the target fault;

the parameter modification module is used for modifying the parameters in the kernel key information based on the disk fault associated parameters;

and the script execution module is used for executing the original script command after the key information of the kernel is modified.

the target fault simulation module 1430 includes:

the second tool calling module is used for calling the flow control tool corresponding to the network delay jitter and binding a preset network card;

the parameter setting module is used for setting an access flow threshold value and a network flow rate in the preset network card based on the fault associated parameters in the fault associated parameters including the access flow threshold value and the network flow rate;

the target fault simulation module 1430 includes:

and a third tool calling module, configured to call a firewall tool corresponding to the network packet loss, and control the packet transmission amount of the fault issuing object to be the packet transmission amount in the fault association parameter.

the device further comprises:

a timing recovery instruction receiving module, configured to receive a timing recovery instruction of the target fault, where the timing recovery instruction includes recovery time, and the timing recovery instruction is sent by the server;

the device further comprises:

a first fault recovery script command determining module for determining a fault recovery script command corresponding to the target fault in the fault proxy component after locally simulating the target fault based on the target fault execution command and the fault associated parameter;

the first fault recovery processing module is used for carrying out fault recovery processing on the target fault based on the fault recovery script command when the recovery time is up;

the device further comprises:

the recovery instruction is a module used for receiving a recovery instruction of the target recovery fault sent by the server;

a second fault recovery script command determining module for determining a fault recovery script command in the fault proxy component corresponding to the target recovery fault after locally simulating the target fault based on the target fault execution command and the fault associated parameter;

and the second fault recovery processing module is used for carrying out fault recovery processing on the target recovery fault based on the fault recovery script command.

The present application provides a fault simulation device, which includes a processor and a memory, where the memory stores at least one instruction, at least one program, a set of codes, or a set of instructions, and the at least one instruction, the at least one program, the set of codes, or the set of instructions is loaded and executed by the processor to implement the fault simulation method provided by the above method embodiment.

In the embodiments of the present disclosure, the memory may be used to store software programs and modules, and the processor executes various functional applications and data processing by operating the software programs and modules stored in the memory. The memory can mainly comprise a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs needed by functions and the like; the storage data area may store data created according to use of the apparatus, and the like. Further, the memory may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory may also include a memory controller to provide the processor access to the memory.

The method provided by the embodiment of the application can be executed in a mobile terminal, a computer terminal, a server or a similar operation device. Taking the example of running on a server, fig. 15 is a hardware structure block diagram of a server of a fault simulation method provided in the embodiment of the present application. As shown in fig. 15, the server 1500 may have a relatively large difference due to different configurations or performances, and may include one or more Central Processing Units (CPUs) 1510 (the processor 1510 may include but is not limited to a Processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1530 for storing data, and one or more storage media 1520 (e.g., one or more mass storage devices) for storing applications 1523 or data 1522. The memory 1530 and storage media 1520 may be, among other things, transient storage or persistent storage. The program stored in the storage medium 1520 may include one or more modules, each of which may include a series of instruction operations in a server. Still further, the central processor 1510 may be disposed in communication with the storage medium 1520, and perform a series of instruction operations in the storage medium 1520 on the server 1500. The Server 1500 may also include one or more power supplies 1560, one or more wired or wireless network interfaces 1550, one or more input-output interfaces 1540, and/or one or more operating systems 1521, such as a Windows Server^TM，Mac OS X^TM，Unix^TM,Linux^TM，FreeBSD^TMAnd so on.

The input/output interface 1540 can be used to receive and transmit data over a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the server 1500. In one example, i/o Interface 1540 includes a Network adapter (NIC) that may be coupled to other Network devices through a base station to communicate with the internet. In one example, the input/output interface 1540 can be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.

It will be understood by those skilled in the art that the structure shown in fig. 15 is only an illustration and is not intended to limit the structure of the electronic device. For example, server 1500 may also include more or fewer components than shown in FIG. 15, or have a different configuration than shown in FIG. 15.

Embodiments of the present application further provide a storage medium, which may be disposed in a server to store at least one instruction, at least one program, a code set, or a set of instructions related to implementing a fault simulation method in the method embodiments, where the at least one instruction, the at least one program, the code set, or the set of instructions are loaded and executed by the processor to implement the fault simulation method provided by the method embodiments.

Alternatively, in this embodiment, the storage medium may be located in at least one network server of a plurality of network servers of a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.

According to the fault simulation method, the fault simulation device, the server or the storage medium, when fault simulation is needed, the server forwards the fault correlation parameters and the target fault to the fault issuing object with the fault proxy component, so that the fault issuing object can flexibly meet the simulation requirements of various faults by executing script commands on multiple faults in the fault proxy component, the richness of a fault simulation scene is effectively improved, software-based fault simulation is realized, and the fault simulation efficiency and the fault simulation operation density are improved. And by means of component deployment, remote copy can be directly carried out within the reachable range of the network, and convenient and fast terminal deployment is realized. In addition, multiple functions of convenient and quick fault tracking (routing inspection), fault recovery, fault analysis and the like can be realized.

It should be noted that: the sequence of the embodiments of the present application is only for description, and does not represent the advantages and disadvantages of the embodiments. And specific embodiments thereof have been described above. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the device, server, terminal and storage medium embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and in relation to the description, reference may be made to some parts of the description of the method embodiments.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware to implement the above embodiments, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disk, or the like.

The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of fault simulation, the method comprising:

2. The method of claim 1, further comprising:

and when the fault issuing object is not deployed with the fault agent component, deploying the fault agent component on the fault issuing object.

3. The method of claim 1, wherein the fault proxy component further comprises a plurality of failed fault recovery script commands;

the method further comprises the following steps:

sending a timing recovery instruction of the target fault to the fault issuing object, wherein the timing recovery instruction comprises recovery time, so that the fault issuing object determines a fault recovery script command corresponding to the target fault in the fault agent component after locally simulating the target fault based on the target fault execution command and the fault associated parameter; when the recovery time is up, performing fault recovery processing on the target fault based on the fault recovery script command;

the method further comprises the following steps:

receiving a fault recovery request sent by the terminal, wherein the fault recovery request comprises a task identifier of a target recovery fault;

determining a fault issuing object corresponding to the target recovery fault based on the task identifier, and sending a recovery instruction of the target recovery fault to the fault issuing object, so that the fault issuing object determines a fault recovery script command corresponding to the target recovery fault in the fault proxy component after locally simulating the target fault based on the target fault execution command and the fault associated parameter; and performing fault recovery processing on the target recovery fault based on the fault recovery script command.

4. The method of claim 1, further comprising:

receiving the failure issuing pipeline data sent by the terminal;

receiving fault execution pipeline data sent by a fault issuing object;

generating fault simulation running water data of the target fault based on the fault issuing running water data and the fault execution running water data;

and sending the fault simulation running water data to the terminal.

5. The method of claim 1, further comprising:

receiving a generation request of a fault analysis report sent by the terminal to a server, wherein the generation request comprises a task identifier of a target fault;

determining fault simulation flow data of the target fault based on the task identification;

performing fault analysis on the target fault based on the fault simulation running water data to obtain a fault analysis report;

and sending the fault analysis report to the terminal.

6. The method of claim 1, further comprising:

receiving the sent fault inspection request, wherein the fault inspection request comprises a task identifier of a target inspection fault;

determining fault simulation flow data of the target inspection fault based on the task identifier of the target inspection fault;

generating a fault patrol inspection report based on the fault simulation flow data of the target patrol inspection fault;

and sending the fault patrol inspection report to the terminal.

7. A method of fault simulation, the method comprising:

8. The method of claim 7, wherein when the target failure comprises a disk failure, the target failure execution script command comprises a script command that invokes a kernel monitoring and tracking tool to perform disk failure simulation, and the failure associated parameters comprise disk failure associated parameters;

calling the kernel monitoring and tracking tool to acquire kernel key information in an original script command corresponding to the fault issuing object and the target fault;

modifying parameters in the kernel key information based on disk fault associated parameters;

and executing the original script command after modifying the key information of the kernel.

9. A fault simulation apparatus, characterized in that the apparatus comprises:

10. A fault simulation apparatus, characterized in that the apparatus comprises: