CN114500349B - A cloud platform chaos testing method and device - Google Patents

Abstract

The invention discloses a cloud platform chaos testing method and device, wherein the method includes: after injecting a fault into the cloud platform to be tested, obtaining the test result of the kth round of concurrent testing of the cloud platform to be tested, and counting the actual results of the kth round of concurrent testing Error rate; use the actual error rate of the k-th round of concurrent testing to estimate the predicted error rate of the k+1-th round of concurrent testing; determine whether the predicted error rate of the k+1-th round of concurrent testing is greater than the preset threshold; when greater than the preset threshold , then reduce the amount of concurrent requests; when it is less than the preset threshold, increase the amount of concurrent requests; when it is equal to the preset threshold, determine the amount of concurrent requests of the k-th round of concurrent testing as the criticality of the cloud platform under test under failure. The present invention adjusts the amount of concurrent requests for the current test by using historical test results, and self-adaptively adjusts the amount of concurrent requests after the fault injection, thereby accurately testing the performance criticality of the cloud platform after the fault is injected, and determining the performance of the fault after the fault Cloud platform performance degradation.

Description

A cloud platform chaos testing method and device

technical field

The invention relates to the technical field of testing, in particular to a cloud platform chaos testing method and device.

Background technique

In recent years, cloud computing has been a hot research direction in the ICT field, and the quality control of cloud platform performance has also attracted much attention. At present, the mainstream open source cloud platform software (such as: OpenStack) generally adopts the microservice architecture. The microservice architecture splits a single software into multiple software services with distinct functions that can be run and deployed independently. It has the characteristics of good scalability, easy deployment, and easy development. The adoption of the microservice architecture is beneficial to reduce the cost of software development and facilitates the combination with the Devops (Development and operations) working mode, but it also introduces new challenges. The software of the microservice architecture is distributed, and the copies of microservices that provide the same function are generally located in different hosts, virtual machines or containers. Infrastructure failures and unexpected start and stop of microservice copies may reduce or even interrupt the ability of the software to provide services to the outside world. For cloud service providers, the performance degradation and response failure of the control plane will cause huge economic losses.

Chaos experiment is an emerging research direction in the field of software testing in recent years. The chaos experiment is mainly used to observe whether the microservice software system has the ability to deal with failures in the case of random failure injection. The execution of chaos experiments is an important link that needs to be automated. Existing chaos experiment tools, such as: ChaosBlade, ChaosMonkey, etc., can meet the needs of artificially simulating CPU, memory and other faults, but the inventor found that if the cloud platform test uses a constant number of concurrent requests, it can only be based on the request success rate To determine whether the fault has an impact on the result, but not to determine how much the fault will degrade the performance of the cloud platform. An assessment of the performance impact can only be given by performing multiple experiments.

Contents of the invention

Therefore, the present invention aims to solve the technical problem in the prior art that it is impossible to determine the performance degradation of the cloud platform due to faults, thereby providing a cloud platform chaos testing method and device.

An aspect of the embodiments of the present invention provides a cloud platform chaos testing method, comprising the following steps: after injecting a fault into the cloud platform to be tested, obtaining the test result of the kth round of concurrent testing of the cloud platform to be tested, and counting The actual error rate of the kth round of concurrent testing, where k is 1, 2, 3...; the predicted error rate of the k+1th round of concurrent testing is estimated by using the actual error rate of the kth round of concurrent testing; Whether the prediction error rate of the k+1 round of concurrent testing is greater than the preset threshold; when the prediction error rate of the k+1th round of concurrent testing is greater than the preset threshold, it will be reduced on the basis of the kth round of concurrent testing The amount of concurrent requests is obtained by obtaining the concurrent request amount of the k+1th round of concurrent testing; when the prediction error rate of the k+1th round of concurrent testing is less than the preset threshold, then on the basis of the kth round of concurrent testing Increase the amount of concurrent requests to obtain the amount of concurrent requests for the k+1th round of concurrent testing; when the prediction error rate of the k+1th round of concurrent testing is equal to the preset threshold, the concurrency of the kth round of concurrent testing The request amount is determined as the criticality of the cloud platform under test under the fault.

Optionally, estimating the predicted error rate of the k+1th round of concurrent testing by using the actual error rate of the k-th round of concurrent testing includes: obtaining the predicted error rate of the k-th round of concurrent testing; using pre-configured weights and calculating the predicted error rate and the actual error rate of the kth round of concurrent testing to obtain the predicted error rate of the k+1th round of concurrent testing.

Optionally, the prediction error rate of the k+1th round of concurrent testing is calculated by the following formula:

e′ _k+1 ＝αe _k +(1-α)e′ _k

Among them, e _k represents the actual error rate of the k-th round of concurrent testing, e' _k represents the predicted error rate of the k-th round of concurrent testing, and α represents the smoothing coefficient.

Optionally, the reducing the amount of concurrent requests on the basis of the kth round of concurrent testing to obtain the amount of concurrent requests of the k+1th round of concurrent testing includes: using the prediction error of the k+1th round of concurrent testing The rate is used as an attenuation coefficient to calculate the amount of concurrent requests for the k+1th round of concurrent testing.

Optionally, the amount of concurrent requests for the k+1th round of concurrent testing is calculated by the following formula:

Among them, e' _k+1 represents the prediction error rate of the k+1th round of concurrent testing, C _k represents the amount of concurrent requests for the kth round of concurrent testing, Represents the round-up operator.

Optionally, said increasing the amount of concurrent requests on the basis of the k-th round of concurrent testing to obtain the amount of concurrent requests for the k+1 round of concurrent testing includes: using a preset floating coefficient and the k-th round of concurrent testing The amount of concurrent requests to be increased is determined by the amount of concurrent requests in the test, and added to the amount of concurrent requests in the kth round of concurrent testing to obtain the amount of concurrent requests in the k+1th round of concurrent testing.

Optionally, the amount of concurrent requests for the k+1th round of concurrent testing is calculated by the following formula:

Among them, e' _k+1 represents the prediction error rate of the k+1-th round of concurrent testing, C _k represents the amount of concurrent requests for the k-th round of concurrent testing, and β represents the floating coefficient, Represents the round down operator.

Another aspect of the present invention also provides a cloud platform chaos testing device, including: an acquisition module, used to acquire the test of the kth round of concurrent testing of the cloud platform to be tested after injecting a fault into the cloud platform to be tested As a result, the actual error rate of the kth round of concurrent testing is counted, k is 1, 2, 3...; the estimation module is used to estimate the actual error rate of the kth round of concurrent testing to obtain the k+1th round of concurrent testing Prediction error rate; judging module, used to judge whether the prediction error rate of the k+1th round of concurrent testing is greater than a preset threshold; a first calculation module, used when the prediction error rate of the k+1th round of concurrent testing If it is greater than the preset threshold, the amount of concurrent requests is reduced on the basis of the kth round of concurrent testing to obtain the amount of concurrent requests for the k+1th round of concurrent testing; the second calculation module is used for when the k+th round If the prediction error rate of one round of concurrent testing is less than the preset threshold, the amount of concurrent requests is increased on the basis of the kth round of concurrent testing to obtain the amount of concurrent requests for the k+1th round of concurrent testing; the determination module is used to When the prediction error rate of the k+1th round of concurrent testing is equal to the preset threshold, the concurrent request amount of the kth round of concurrent testing is determined as the criticality of the cloud platform under test under the failure.

Another aspect of the present invention also provides a computer device, including: at least one processor; and a memory connected to the at least one processor in communication; wherein, the memory stores information that can be used by the at least one processor Executed instructions, the instructions are executed by the at least one processor, so as to execute the above cloud platform chaos testing method.

Another aspect of the present invention also provides a computer-readable storage medium, which is characterized in that the computer-readable storage medium stores computer instructions, and the computer instructions are used to make a computer execute the above-mentioned cloud platform chaos testing method .

The technical solution of the present invention has the following advantages:

According to the embodiment of the present invention, by using the historical test results to adjust the concurrent request volume of the current test, and adaptively adjust the concurrent request volume after the fault injection, so as to accurately test the performance criticality of the cloud platform after the fault injection, and determine The performance degradation of the cloud platform after a failure.

According to the embodiment of the present invention, by using the error rate of the last round of concurrent testing, that is, using historical data to dynamically adjust the current error rate, and then achieve the purpose of adjusting the amount of concurrent requests, and improve the accuracy of the evaluation of the performance degradation level after a fault sex.

Description of drawings

In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following will briefly introduce the accompanying drawings that need to be used in the specific implementation or description of the prior art. Obviously, the accompanying drawings in the following description The drawings show some implementations of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

Fig. 1 is the frame schematic diagram of the testing system of the embodiment of the present invention;

Fig. 2 is the flowchart of a concrete example of cloud platform chaos test method in embodiment 1 of the present invention;

Fig. 3 is the sequence diagram of fault injection of cloud platform chaos test in the embodiment of the present invention;

FIG. 4 is a schematic diagram of a deployment architecture of a test system according to an embodiment of the present invention;

Fig. 5 is the functional block diagram of a specific example of cloud platform chaos testing device in embodiment 2 of the present invention;

FIG. 6 is a schematic structural diagram of a computer device according to an embodiment of the present invention.

Detailed ways

The technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Apparently, the described embodiments are part of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" etc. The indicated orientation or positional relationship is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, or in a specific orientation. construction and operation, therefore, should not be construed as limiting the invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and should not be construed as indicating or implying relative importance.

In the description of the present invention, it should be noted that unless otherwise specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection. Connected, or integrally connected; it can be mechanically or electrically connected; it can be directly connected, or indirectly connected through an intermediary, or it can be the internal communication of two components, which can be wireless or wired connect. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention in specific situations.

In addition, the technical features involved in the different embodiments of the present invention described below may be combined with each other as long as there is no conflict with each other.

The cloud platform chaos testing method and device provided by the embodiments of the present invention can dynamically adjust the amount of concurrent requests after a fault in the automated testing process of the cloud platform, and can also control the number of iterations or time when fault injection and release occur, realizing service Simulation of failures such as start-stop, network congestion, high memory load, etc. The device proposed by the invention can automatically execute and record the fault injection of the chaos experiment, improves the diversity of the steady-state indicators of the chaos experiment, and improves the evaluation of the performance degradation caused by the fault.

Before introducing the test method of the embodiment of the present invention, the test resource and environment provided by the embodiment of the present invention are introduced first.

The testing process of the embodiment of the present invention needs to provide the following resources: chaos tasks, users, items, and roles. A chaos task is an execution of a chaos configuration on the cloud platform to be tested. A project is the smallest unit for managing chaos configurations and chaos tasks. Users, projects, and roles are associated in the way of role-based access control (RBAC, Role-based Access Control). Users access resources under a project by virtue of roles.

The embodiment of the present invention also provides a test system for performing chaos testing, as shown in Figure 1, the system is mainly composed of the following modules: user authority management module, task management module, cloud platform test suite, remote fault injection tools etc. In addition to the above modules, the device depends on the code hosting platform and continuous integration platform.

User rights management module: Provide functions for creating, deleting, modifying, and querying users, projects, and roles. Supports permission control through RBAC.

Code hosting platform: You can choose code hosting warehouses such as gitlab or gerrit. The user submits the yaml configuration about fault injection to the code hosting platform. The configuration parameters of the yaml file include but are not limited to: fault type, number of injection iterations, number of release iterations, total number of iterations, etc. The configuration is reviewed and merged in on the code hosting platform.

Task management module: The task management module operates in the form of HTTP service, which provides the functions of adding, deleting, modifying and querying chaotic tasks. When adding a chaotic task, the task management module first verifies the request, including but not limited to: verifying whether the start time is greater than the current time, whether the fault injection configuration file exists, and other verification items. After the verification is completed, save the ID, name, remarks, start time, configuration file content and other information to the database. At this time, the task is in the pending state. When the task start time is reached, the task management module triggers the pipeline on the continuous integration platform for fault injection and testing of the cloud platform, and updates the task status to running. After the continuous integration platform pipeline is completed, the task management module updates the task as finished and saves the test results to the database. When updating a chaotic task, the task management module first checks the status, and the task name and remark field in any state can be updated, but only the task in the pending state can update the start time. After the update, the task management module saves the information to the database.

Continuous integration platform pipeline: When the continuous integration platform pipeline is triggered, the continuous integration platform slave node will first download the cloud platform test tool, fault injection tool, and fault injection configuration file from the code hosting platform. Then install fault injection tools and cloud platform testing tools in sequence. Finally, the basic configuration of the cloud platform test is executed, and the fault injection configuration file is specified to run the test.

Cloud platform testing tool: The continuous integration platform runs the cloud platform testing tool on the slave node. After the test command is issued, the cloud platform test tool starts two threads, one is used to execute the test task, and the other is used to handle fault injection. The test task and the number of iterations executed are managed in the first-in-first-out queue, and the thread executing the test puts the iteration number and other single-run information into the queue before each execution. If the thread that handles fault injection is triggered by a specified number of iterations, it will listen to the queue of iterations, and compare the number of iterations sequentially from the queue. If it is triggered at a specified time, the timer will be started after the first number of iterations are fetched in the queue.

Cloud platform fault injection: The cloud platform fault injection module provides the capabilities of service start and stop, docker start and stop, server start and stop, network packet loss, network delay, memory load injection, and CPU load injection. The fault injection module provides an API for other program calls. In the present invention, the fault injection module is invoked by the cloud platform test tool, and after the fault injection processing thread in the test tool is triggered, the fault injection API is called. After the fault injection module logs in through the management network SSH or the out-of-band network IPMI, it uses systemctl, tc and other tools to complete the fault injection task.

A kind of cloud platform chaos test method provided by the embodiment of the present invention, as shown in Figure 2, includes the following steps:

Step S101, after injecting faults into the cloud platform to be tested, obtain the test result of the k-th round of concurrent testing on the cloud platform to be tested, and count the actual error rate of the k-th round of concurrent testing, where k is 1, 2, 3... .

The k-th round of concurrent testing described in the embodiments of the present invention refers to that during the testing process of the k-th iteration, for the k-th round of concurrent testing, the cloud platform testing tool can be used to initiate a request for a corresponding amount of concurrent requests to the cloud platform , to obtain the corresponding test results. Wherein, the test result may refer to a single concurrent response result, if there is no response, an error will be reported; if the response is successful, it means that the test is passed. Among them, the ratio of the number of error reports to the amount of concurrent requests can be used as the actual error rate. In the embodiment of the present invention, the value of k may be 1, 2, 3..., which may refer to the test round after the fault is injected, or may refer to the test round in the entire test task. For example, the test is iterated 1000 times, and the value of k is 1, 2, 3...1000, and the fault is injected at the 200th time and recovered at the 800th time. The embodiment of the present invention mainly protects how to adjust the amount of concurrent requests after fault injection.

Certainly, for those skilled in the art, after reading the embodiments of the present invention, it can be known that the adjustment of the amount of concurrent requests can be carried out throughout the entire testing process. The embodiment of the present invention emphasizes how to determine the degraded criticality of the performance of the cloud platform to be tested in a fault state after fault injection.

In the embodiment of the present invention, the message delivery of test execution and fault injection is completed through queue 1, and queue 1 stores the number of iterations (that is, k). Before each execution of the test, the test process stores the number of iterations in the queue 1 and then opens a new thread to perform the test task. When the test thread being executed reaches the number of concurrent requests, the test process waits for the test thread to finish executing, reads the test result, and counts the actual error rate. Wherein, each test thread corresponds to a test case, and a test case corresponds to a concurrent test request.

Step S102, using the actual error rate estimation of the kth round of concurrent testing to obtain the predicted error rate of the k+1th round of concurrent testing.

In this embodiment, after the actual error rate is calculated, the actual error rate is smoothly updated to estimate the predicted error rate of the next round of concurrent testing. Specifically, in the embodiment of the present invention, the predicted error rate of the k+1-th round of concurrent testing can be directly estimated by using the actual error rate of the k-th round of concurrent testing, for example, the actual error rate is multiplied by a coefficient; on the other hand, It is also possible to use the actual error rate of the k-th round of concurrent testing and the predicted error rate of the k-th round of concurrent testing to calculate the predicted error rate of the k+1-th round of concurrent testing, for example, the actual error rate of the k-th round of concurrent testing and The prediction error rate is weighted and summed to obtain the prediction error rate of the k+1th round of concurrent testing.

As an optional implementation manner, the estimating the predicted error rate of the k+1th round of concurrent testing by using the actual error rate of the k-th round of concurrent testing includes: obtaining the predicted error rate of the k-th round of concurrent testing; using The pre-configured weights and the predicted error rate and actual error rate of the kth round of concurrent testing are calculated to obtain the predicted error rate of the k+1th round of concurrent testing.

In the embodiment of the present invention, the prediction error rate of the k-th round of concurrent testing can be calculated after the end of the k-1 round of concurrent testing; if k=1, the prediction error rate is the initial value, that is, 0. The pre-configured weight refers to the relative weight of the predicted error rate of the k-th round of concurrent testing and the actual error rate in the calculation process when calculating the predicted error rate of the k+1-th round of concurrent testing.

According to the embodiment of the present invention, by using the error rate of the last round of concurrent testing, that is, using historical data to dynamically adjust the current error rate, and then achieve the purpose of adjusting the amount of concurrent requests, and improve the accuracy of the evaluation of the performance degradation level after a fault sex.

As a further optional implementation manner, the prediction error rate of the k+1th round of concurrent testing can be calculated by the following formula:

e′ _k+1 ＝αe _k +(1-α)e′ _k

Among them, e _k represents the actual error rate of the k-th round of concurrent testing, e' _k represents the predicted error rate of the k-th round of concurrent testing, and α represents the smoothing coefficient. Wherein, the value of α is a fixed value, which may be greater than 0.5 and less than 1, for example, 0.8. The higher the smoothing coefficient, the higher the proportion of new observations. Since the testing process in the embodiment of the present invention is an iterative process, the above formula is also an iterative process. As the number of iterations increases, the longer the error rate, the smaller the proportion.

Step S103, judging whether the prediction error rate of the k+1th round of concurrent testing is greater than a preset threshold.

Step S104, when the prediction error rate of the k+1th round of concurrent testing is greater than the preset threshold, then reduce the amount of concurrent requests on the basis of the k+1th round of concurrent testing to obtain the k+1th round of concurrent testing The amount of concurrent requests.

Step S105, when the prediction error rate of the k+1th round of concurrent testing is less than the preset threshold, increase the amount of concurrent requests on the basis of the k+1th round of concurrent testing to obtain the k+1th round of concurrent testing The amount of concurrent requests.

Step S106, when the prediction error rate of the k+1th round of concurrent testing is equal to the preset threshold, determine the amount of concurrent requests of the kth round of concurrent testing as the cloud platform under test under the failure critical.

In the embodiment of the present invention, when it is judged that the prediction error rate of the k+1 round of concurrent testing is less than the preset threshold, it means that the cloud platform to be tested has not reached the critical point of concurrent request processing, therefore, the amount of concurrent requests for the next round can be increased ; If the prediction error rate of the k+1th round of concurrent testing is greater than the preset threshold, it means that the cloud platform to be tested has exceeded the limit of concurrent request processing, and it is necessary to reduce the amount of concurrent requests for the next round of testing; the prediction error rate is equal to the preset threshold. If the threshold is set, it can be considered that the current amount of concurrent requests is the limit of the cloud platform under test under the fault, and there is no need to adjust the amount of concurrent requests.

In the embodiment of the present invention, regardless of the relationship between the prediction error rate and the preset threshold, the next round of iterative testing can be returned. Among them, if it is greater than or less than the preset threshold, use the adjusted concurrent request amount to perform the next round of concurrent testing, that is, add 1 to the value of k, return to execute the corresponding concurrent testing, that is, execute steps S101-S103 , Make corresponding judgments, and then make follow-up actions according to the judgment results. When the value of k reaches the maximum value, the chaos test is completed. On the other hand, the loop exit condition can also be set as: when the prediction error rate of the k+1th round of concurrent testing is equal to the preset threshold, because the concurrency limit of the cloud platform to be tested under the failure has been determined at this time .

According to the embodiment of the present invention, by using the historical test results to adjust the concurrent request volume of the current test, and adaptively adjust the concurrent request volume after the fault injection, so as to accurately test the performance criticality of the cloud platform after the fault injection, and determine The performance degradation of the cloud platform after a failure.

As an optional implementation manner, the reducing the amount of concurrent requests on the basis of the kth round of concurrent testing to obtain the concurrent request amount of the k+1th round of concurrent testing includes: using the k+1th round of concurrent testing The prediction error rate of the test is used as an attenuation coefficient to calculate the concurrent request amount of the k+1th round of concurrent testing.

Since the predicted error rate is calculated based on the actual error rate of the previous round, it is used as the attenuation coefficient to adjust the amount of concurrent requests so that the adjusted amount of concurrent requests is more in line with the test results of the previous round. It can also approach the criticality of the cloud platform under failure faster.

Specifically, in this embodiment of the present invention, the amount of concurrent requests for the k+1th round of concurrent testing (that is, the reduced amount of concurrent requests) can be calculated by the following formula:

Among them, e' _k+1 represents the prediction error rate of the k+1th round of concurrent testing, C _k represents the amount of concurrent requests for the kth round of concurrent testing, Represents the round-up operator.

On the other hand, in the embodiment of the present invention, the method similar to the above can also be used to increase the amount of concurrency, and the prediction error rate is used as the floating coefficient. The calculation method can be similar to the above formula, and the effect is also similar, and will not be repeated here.

As an optional implementation manner, the increase of the concurrent request amount on the basis of the k-th round of concurrent testing to obtain the concurrent request amount of the k+1-th round of concurrent testing includes: using a preset floating coefficient and the The amount of concurrent requests to be increased is determined from the amount of concurrent requests in the kth round of concurrent testing, and added to the amount of concurrent requests in the kth round of concurrent testing to obtain the amount of concurrent requests in the k+1th round of concurrent testing.

That is, in the embodiment of the present invention, a fixed floating coefficient can be set to calculate the increased concurrent request amount. Specifically, the amount of concurrent requests for the k+1th round of concurrent testing can be calculated by the following formula:

Among them, e' _k+1 represents the prediction error rate of the k+1th round of concurrent testing, C _k represents the amount of concurrent requests for the kth round of concurrent testing, and β represents the floating coefficient, which can be set according to experience. Represents the round down operator.

Combining the above calculation formulas, we get the following:

Among them, ∈ is the preset threshold. If the prediction error rate is greater than the preset threshold, the test management process decays the amount of concurrent requests in the test management process according to the error rate. If the error rate is lower than the preset threshold, for example, when the current error rate is 0, the amount of concurrent requests cannot reach the critical limit of the cloud platform. At this time, try to increase the amount of concurrent requests of the test management process appropriately.

In the embodiment of the present invention, the fault injection thread has two working modes, one is the timing mode, and the other is the iteration number mode. The configuration of a chaos experiment in timed mode specifies the fault injection time after the start of the test. The number of iterations mode determines when a fault is injected based on the number of times a test is executed. Both of them adopt the method of polling.

The following introduces the embodiment of the present invention through the workflow of the chaos test shown in Figure 3, as shown in Figure 3, including:

Step 1: The user submits the chaos experiment configuration to the code hosting platform.

Step 2: Merge into the code base after being reviewed by the administrator.

Step 3: The user creates a chaos experiment task through the Restful API interface, and specifies the configuration and start time of the chaos experiment task when creating.

Step 4: After the task management module detects that the time of the chaos experiment task has arrived, it triggers the continuous integration platform pipeline.

Step 5: The continuous integration platform downloads the chaos experiment configuration, fault injection tool, and cloud platform test tool from the node.

Step 6: The continuous integration platform installs fault injection tools and cloud platform testing tools sequentially from the nodes.

Step 7: Run the cloud platform test. During the test, the cloud platform test tool and fault injection tool manage fault injection and recovery.

Step 8: Generate test results and return task status.

Figure 4 is a chaos testing platform according to the embodiment of the present invention, and its deployment structure is as follows: the code hosting platform is deployed on server 1, the user rights management module, task management module, and database are deployed on server 2, and the Jenkins master is deployed on server 3 , Server 4 acts as a Jenkins slave. Servers 1 to 4 are all connected to the cloud platform management network, and server 4 is connected to the out-of-band management network of all management nodes and computing nodes of the cloud platform.

The cloud platform to be tested uses Openstack and consists of three management nodes and two computing nodes. The management nodes are deployed with keystone, nova-api, nova-scheduler, nova-conductor, placement, cinder-api, cinder-scheduler, glance, neutron-server, neutron-dhcp-agent, etc. The computing nodes are deployed with services such as nova-compute and cinder-volume.

The specific operation steps for users include:

Step 1: The user submits the chaos experiment configuration to the code hosting platform on server 1 through git.

Step 2: The configuration is merged after review.

Step 3: The user holds the credential information to apply for a token from the user authority management module, and holds the token to initiate a request to the task management module to create a chaos experiment task.

Step 4: The Jenkins pipeline is triggered.

Step 5: Jenkins slave starts downloading configuration and related software.

Step 6: The execution of the chaos experiment task is on the Jenkins slave node (server 4), and the fault injection tool and the cloud platform testing tool are installed on the server 4.

Step 7: The cloud platform test tool executes the test. In this embodiment, the chaos experiment configuration specifies to run the test of creating a cloud host, and specifies a total number of iterations of 1000, and a concurrency of 20. The fault type is that the nova-api service of a random management node is down, the occurrence time is 200 iterations, and the release time is 800 iterations. When the test tool runs the test case 200 times, the fault injection thread in the test tool detects that the specified number of times has been reached, remotely logs in to management node 1 (selected at random) through the cloud platform management network, executes the command systemctl stop openstack-nova-api, and then the service stops . Resume service when iteration count 800 is reached. After 200 iterations, start to adjust the number of concurrency adaptively according to the statistics of the error rate, and keep approaching the concurrency limit of the system's normal processing requests.

Step 8: After the chaos experiment is over, return the test result and the state of the chaos experiment.

Example 2

This embodiment provides a cloud platform chaos test device, which can be used to execute the test method in the above-mentioned embodiment 1, as shown in Figure 5, the device includes:

Obtaining module 501, is used for after injecting fault to cloud platform to be tested, obtains the test result of the kth round concurrent test of described cloud platform to be tested, counts the actual error rate of kth round concurrent test, k is 1, 2, 3...;

An estimation module 502, configured to estimate the predicted error rate of the k+1th round of concurrent testing by using the actual error rate of the kth round of concurrent testing;

A judging module 503, configured to judge whether the prediction error rate of the k+1th round of concurrent testing is greater than a preset threshold;

The first calculation module 504 is configured to reduce the amount of concurrent requests on the basis of the kth round of concurrent testing when the prediction error rate of the k+1th round of concurrent testing is greater than the preset threshold, and obtain the k+th round of concurrent testing. The amount of concurrent requests for one round of concurrent testing;

The second calculation module 505 is configured to increase the amount of concurrent requests on the basis of the kth round of concurrent testing when the prediction error rate of the k+1th round of concurrent testing is less than the preset threshold, and obtain the k+th round of concurrent testing. The amount of concurrent requests for one round of concurrent testing;

A determining module 506, configured to determine the concurrent request amount of the kth round of concurrent testing as the cloud platform under test when the prediction error rate of the k+1th round of concurrent testing is equal to the preset threshold Criticality under the above fault.

According to the embodiment of the present invention, by using the error rate of the last round of concurrent testing, that is, using historical data to dynamically adjust the current error rate, and then achieve the purpose of adjusting the amount of concurrent requests, and improve the accuracy of the evaluation of the performance degradation level after a fault sex.

For specific descriptions of the apparatus embodiments, reference may be made to the foregoing method embodiments, and details are not repeated here.

Example 3

In an embodiment of the present invention, a computer device is also provided, and its internal structure diagram may be as shown in FIG. 6 . The computer equipment includes a processor, a memory, a network interface connected through a system bus, and may also include a display screen and an input device. Wherein, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with external computer devices via a network connection. When the computer program is executed by the processor, the data deduplication method for traffic playback or the test method of the business system can be realized. The computer device can also include a display screen and an input device, and the display screen can be a liquid crystal display or an electronic ink display. The input device of the computer equipment may be a touch layer covered on the display screen, or a button, a trackball, or a touch pad provided on the casing of the computer equipment.

On the other hand, the computer equipment may not include a display screen and an input device. Those skilled in the art can understand that the structure shown in FIG. A specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor , the instructions are executed by the at least one processor to implement the following steps:

After injecting faults into the cloud platform to be tested, obtain the test result of the kth round of concurrent testing of the cloud platform to be tested, and count the actual error rate of the kth round of concurrent testing, where k is 1, 2, 3...;

Using the actual error rate estimation of the kth round of concurrent testing to obtain the predicted error rate of the k+1th round of concurrent testing;

Judging whether the prediction error rate of the k+1th round of concurrent testing is greater than a preset threshold;

When the prediction error rate of the k+1th round of concurrent testing is greater than the preset threshold, the amount of concurrent requests is reduced on the basis of the kth round of concurrent testing to obtain the amount of concurrent requests for the k+1th round of concurrent testing ;

When the prediction error rate of the k+1th round of concurrent testing is less than the preset threshold, the amount of concurrent requests is increased on the basis of the kth round of concurrent testing to obtain the amount of concurrent requests for the k+1th round of concurrent testing ;

When the prediction error rate of the k+1th round of concurrent testing is equal to the preset threshold, the concurrent request amount of the kth round of concurrent testing is determined as the criticality of the cloud platform under test under the failure.

In one embodiment, a readable storage medium is provided, the computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute:

After injecting faults into the cloud platform to be tested, obtain the test result of the kth round of concurrent testing of the cloud platform to be tested, and count the actual error rate of the kth round of concurrent testing, where k is 1, 2, 3...;

Using the actual error rate estimation of the kth round of concurrent testing to obtain the predicted error rate of the k+1th round of concurrent testing;

Judging whether the prediction error rate of the k+1th round of concurrent testing is greater than a preset threshold;

When the prediction error rate of the k+1th round of concurrent testing is greater than the preset threshold, the amount of concurrent requests is reduced on the basis of the kth round of concurrent testing to obtain the amount of concurrent requests for the k+1th round of concurrent testing ;

When the prediction error rate of the k+1th round of concurrent testing is less than the preset threshold, the amount of concurrent requests is increased on the basis of the kth round of concurrent testing to obtain the amount of concurrent requests for the k+1th round of concurrent testing ;

When the prediction error rate of the k+1th round of concurrent testing is equal to the preset threshold, the concurrent request amount of the kth round of concurrent testing is determined as the criticality of the cloud platform under test under the failure.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above-mentioned embodiments can be completed by instructing related hardware through computer programs, and the computer programs can be stored in a non-volatile computer-readable memory In the medium, when the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any references to memory, storage, database or other media used in the various embodiments provided in the present application may include non-volatile and/or volatile memory. Nonvolatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Chain Synchlink DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

Apparently, the above-mentioned embodiments are only examples for clear description, rather than limiting the implementation. For those of ordinary skill in the art, other changes or changes in different forms can be made on the basis of the above description. It is not necessary and impossible to exhaustively list all the implementation manners here. And the obvious changes or changes derived therefrom are still within the scope of protection of the present invention.

Claims (10)

1. a cloud platform chaos testing method, is characterized in that, comprises the steps: After injecting faults into the cloud platform to be tested, obtain the test result of the kth round of concurrent testing of the cloud platform to be tested, and count the actual error rate of the kth round of concurrent testing, where k is 1, 2, 3...; Using the actual error rate estimation of the kth round of concurrent testing to obtain the predicted error rate of the k+1th round of concurrent testing; Judging whether the prediction error rate of the k+1th round of concurrent testing is greater than a preset threshold; When the prediction error rate of the k+1th round of concurrent testing is greater than the preset threshold, the amount of concurrent requests is reduced on the basis of the kth round of concurrent testing to obtain the amount of concurrent requests for the k+1th round of concurrent testing ; When the prediction error rate of the k+1th round of concurrent testing is less than the preset threshold, the amount of concurrent requests is increased on the basis of the kth round of concurrent testing to obtain the amount of concurrent requests for the k+1th round of concurrent testing ; When the prediction error rate of the k+1th round of concurrent testing is equal to the preset threshold, the concurrent request amount of the kth round of concurrent testing is determined as the criticality of the cloud platform under test under the failure. 2. cloud platform chaos test method according to claim 1, is characterized in that, described utilization of the actual error rate estimation of described k round concurrent test obtains the prediction error rate of k+1 round concurrent test, comprising: Obtain the prediction error rate of the k-th round of concurrent testing; The predicted error rate of the k+1th round of concurrent testing is calculated by using the preconfigured weights and the predicted error rate and actual error rate of the kth round of concurrent testing. 3. cloud platform chaos test method according to claim 2, is characterized in that, calculates and obtains the forecast error rate of described k+1 round concurrent test by following formula: e′ _k+1 ＝αe _k +(1-α)e′ _k Among them, e _k represents the actual error rate of the k-th round of concurrent testing, e' _k represents the predicted error rate of the k-th round of concurrent testing, and α represents the smoothing coefficient. 4. cloud platform chaos test method according to claim 1, is characterized in that, described on the basis of described k the concurrent test of round reduces concurrent request amount, obtains the concurrent request amount of k+1 round concurrent test, include: Using the prediction error rate of the k+1th round of concurrent testing as an attenuation coefficient, the amount of concurrent requests for the k+1th round of concurrent testing is calculated. 5. cloud platform chaos test method according to claim 4, is characterized in that, obtains the concurrent request volume of described k+1 round concurrency test by following formula calculation: Among them, e' _k+1 represents the prediction error rate of the k+1th round of concurrent testing, C _k represents the amount of concurrent requests for the kth round of concurrent testing, Represents the round-up operator. 6. cloud platform chaos test method according to claim 1, is characterized in that, described on the basis of described k the concurrent test of round increases concurrent request amount, obtains the concurrent request amount of k+1 round concurrent test, include: Use the preset floating coefficient and the concurrent request amount of the kth round of concurrent testing to determine the amount of concurrent requests to be increased, and add the concurrent request amount of the kth round of concurrent testing to obtain the k+1th round of concurrent requests The amount of concurrent requests tested. 7. cloud platform chaos test method according to claim 6, is characterized in that, obtains the concurrency request amount of described k+1 round concurrency test by following formula calculation: Among them, C _k represents the amount of concurrent requests for the k-th round of concurrent testing, and β represents the floating coefficient. Represents the round down operator. 8. A cloud platform chaos testing device, characterized in that it comprises: The acquisition module is used to obtain the test result of the kth round of concurrent testing of the cloud platform to be tested after injecting faults into the cloud platform to be tested, and count the actual error rate of the kth round of concurrent testing, where k is 1, 2, 3 ...; An estimating module, configured to estimate the predicted error rate of the k+1th round of concurrent testing by using the actual error rate of the kth round of concurrent testing; A judging module, configured to judge whether the prediction error rate of the k+1th round of concurrent testing is greater than a preset threshold; The first calculation module is configured to reduce the amount of concurrent requests on the basis of the k+1th round of concurrent testing when the prediction error rate of the k+1th round of concurrent testing is greater than the preset threshold, to obtain the k+1th round The number of concurrent requests for rounds of concurrent testing; The second calculation module is configured to increase the amount of concurrent requests on the basis of the k+1th round of concurrent testing when the prediction error rate of the k+1th round of concurrent testing is less than the preset threshold to obtain the k+1th round The number of concurrent requests for rounds of concurrent testing; A determining module, configured to determine the amount of concurrent requests of the kth round of concurrent testing as the cloud platform under test when the prediction error rate of the k+1th round of concurrent testing is equal to the preset threshold Criticality under failure. 9. A computer device, characterized in that it comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, The instructions are executed by the at least one processor, so as to execute the cloud platform chaos testing method according to any one of claims 1-7. 10. A computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to make a computer execute the cloud platform according to any one of claims 1-7 Chaos testing method.