CN116302975A - Service system change testing method and device - Google Patents

Service system change testing method and device Download PDF

Info

Publication number
CN116302975A
CN116302975A CN202310120093.0A CN202310120093A CN116302975A CN 116302975 A CN116302975 A CN 116302975A CN 202310120093 A CN202310120093 A CN 202310120093A CN 116302975 A CN116302975 A CN 116302975A
Authority
CN
China
Prior art keywords
change
test
experiment
message
change event
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310120093.0A
Other languages
Chinese (zh)
Inventor
李冰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba China Co Ltd
Original Assignee
Alibaba China Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba China Co Ltd filed Critical Alibaba China Co Ltd
Priority to CN202310120093.0A priority Critical patent/CN116302975A/en
Publication of CN116302975A publication Critical patent/CN116302975A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3672—Test management
    • G06F11/3684—Test management for test design, e.g. generating new test cases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3672—Test management
    • G06F11/3688—Test management for test execution, e.g. scheduling of test suites
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/546—Message passing systems or structures, e.g. queues
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Software Systems (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The embodiment of the application provides a method and a device for testing service system change, wherein the method comprises the following steps: the message queue is monitored to obtain the change message corresponding to the change event of the service system from the message queue, and the message queue is used for writing the change message corresponding to the change event when the change event exists in the service system. And acquiring a target test case corresponding to the change event from a plurality of preset test cases according to the change message. And executing test operation on the changed service system according to the target test case so as to obtain a test result corresponding to the change event. According to the technical scheme, the testing efficiency of the changed service system can be effectively improved.

Description

Service system change testing method and device
Technical Field
The embodiment of the application relates to computer technology, in particular to a testing method and device for service system change.
Background
The configuration change is a common experimental mode in the service system, and the validity of the configuration change can be effectively verified by testing the changed service system.
In the current common implementation manner, after the release of the configuration change of the service system is manually observed, the configuration change is manually triggered and a test flow is executed to verify the validity of the configuration change, so as to determine whether the service system after the configuration change achieves the expected effect.
However, manual triggering and implementation of the execution of the test procedure are time-consuming and relatively inefficient.
Disclosure of Invention
The embodiment of the application provides a testing method and device for service system change, which are used for solving the problem of lower testing efficiency for service system change.
In a first aspect, an embodiment of the present application provides a method for testing a service system change, including:
monitoring a message queue to acquire a change message corresponding to a change event of the service system from the message queue, wherein the message queue is used for writing the change message corresponding to the change event when the change event exists in the service system;
according to the change message, acquiring a target test case corresponding to the change event from a plurality of preset test cases;
and executing test operation on the changed service system according to the target test case so as to obtain a test result corresponding to the change event.
In a second aspect, an embodiment of the present application provides a testing apparatus for service system modification, including:
the processing module is used for monitoring a message queue to acquire a change message corresponding to a change event of the service system from the message queue, wherein the message queue is used for writing the change message corresponding to the change event when the change event exists in the service system;
The acquisition module is used for acquiring a target test case corresponding to the change event from a plurality of preset test cases according to the change message;
and the test module is used for executing test operation on the changed service system according to the target test case so as to obtain a test result corresponding to the change event.
In a third aspect, an embodiment of the present application provides an electronic device, including:
a memory for storing a program;
a processor for executing the program stored by the memory, the processor being adapted to perform the method of the first aspect and any of the various possible designs of the first aspect as described above when the program is executed.
In a fourth aspect, embodiments of the present application provide a computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the method of the first aspect above and any of the various possible designs of the first aspect.
In a fifth aspect, embodiments of the present application provide a computer program product comprising a computer program which, when executed by a processor, implements a method as described in the first aspect and any of the various possible designs of the first aspect.
The embodiment of the application provides a method and a device for testing service system change, wherein the method comprises the following steps: the message queue is monitored to obtain the change message corresponding to the change event of the service system from the message queue, and the message queue is used for writing the change message corresponding to the change event when the change event exists in the service system. And acquiring a target test case corresponding to the change event from a plurality of preset test cases according to the change message. And executing test operation on the changed service system according to the target test case so as to obtain a test result corresponding to the change event. When a change event exists in the service system, writing the change message of the change event into a message queue, then automatically acquiring the change message from the message queue by monitoring the message queue, completing automatic test operation on the changed service system according to the change message, and sending a test result to target equipment corresponding to the change event, so that the full-automatic processing flow of the service system after the change can be realized, and the test efficiency of the service system after the change can be effectively improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief description will be given below of the drawings that are needed in the embodiments or the prior art descriptions, it being obvious that the drawings in the following description are some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort to a person skilled in the art.
Fig. 1 is a schematic view of a scenario of a configuration change of a service system according to an embodiment of the present application;
fig. 2 is a flowchart of a testing method for service system modification according to an embodiment of the present application;
fig. 3 is a second flowchart of a testing method for service system modification according to an embodiment of the present application;
fig. 4 is a schematic implementation diagram of a preset mapping relationship provided in an embodiment of the present application;
FIG. 5 is a schematic diagram of an implementation of filtering a change message according to an embodiment of the present application;
FIG. 6 is a flow chart of a testing method for service system change provided in the present application;
fig. 7 is a schematic structural diagram of a testing device for service system modification according to an embodiment of the present application;
fig. 8 is a schematic hardware structure of an electronic device according to an embodiment of the present application.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
For better understanding of the technical solutions of the present application, the related art related to the present application is described in further detail below.
The service products have a great amount of configuration change capability, wherein the configuration change capability can be understood as changing service configuration or service data in a service system of the product to realize the expected effect, and the configuration change only needs to simply change corresponding configuration, so that the effect of switching new and old functions can be realized without release.
In order to realize orderly overall configuration change of the service system, the configuration change of the service system can be managed in an experimental mode, for example, an experiment can be issued for the service system, the content of the experiment is the configuration change of the service system, and then the experiment is tested to determine the validity of the configuration change. The validity of the configuration change can have two meanings, namely that the service system after the configuration change can normally and stably run, and that the service system after the configuration change can realize expected effects or functions.
For example, it can be understood that, when a developer needs to change a service system, for example, an a/B experiment can be deployed in the service system to verify how to effect corresponding to different changing modes. The scheme for maximizing the local index value can be accurately selected through an A/B experiment, wherein the A/B time delay is enabled to be stably and effectively applied to a service system, and the scheme is an important one in quality assurance work. How to reduce the verification cost and efficiency improvement of experimental change under a diversified test scene is a key of quality assurance.
Based on the above description, it may be determined that, for an experiment issued by the service system, it is actually some changes to the service system, where the content of the experiment may be, for example, adding some functions to the service system, or modifying some deployments in the service system, where specific experimental content may be selected and set according to actual requirements, and this embodiment is not limited to this.
Based on the above description, for example, the configuration change of the service system may be understood with reference to fig. 1, and fig. 1 is a schematic view of the configuration change of the service system provided in the embodiment of the present application.
In this embodiment, at least one service line is divided in the service system, and at least one application is included in each service line, one of which is understood as a service. For example, in a map scenario, where there is a service to route from location 1 to location 2, such a route planning service may be understood as an application, where an application may be understood as existing as a software element in a business system to provide corresponding service support for the associated product. And at least one experiment configured in each application, the experiment being a change to some configuration of the application.
For example, as shown in fig. 1, a service line a, a service line B, and a service line C are included in the service system, and an application a and an application B may be included in the service line, and an application C may be included in the service line B, for example. And referring to fig. 1, experiments a, B, and C may be configured in application B, for example.
Taking experiment a as an example, experiment a is for example directed to a change of configuration in application B in service line a. In order to determine the effect after a configuration change, it is necessary to test experiment a and then determine whether the configuration change is valid based on the test results.
In the actual implementation process, the specific service line division, the specific application setting and the specific experiment configuration can be selected and set according to the actual requirements, and the embodiment is not limited to this.
Based on the above description, it can be determined that it is very important to test an experiment after the experiment is issued for the service system, and based on the test result, how the effect of the configuration change corresponding to the experiment is determined. In the prior art, when verifying the validity of the configuration change, the test is usually manually observed to release the test, and then the test flow for the changed service system is manually triggered and executed. However, such manual triggering and implementation of the execution of the test procedure is time-consuming and relatively inefficient.
And it can be understood that when an experiment is newly added to the service system, the experiment needs to be tested, and when an existing experiment is adjusted, a further test is needed to be performed on the adjusted experiment, if the experiment after each adjustment needs to be manually observed and released and manually triggered, the test efficiency of the experiment is definitely reduced greatly.
Aiming at the problems in the prior art, the application provides the following technical conception: a set of automatic processes can be designed to automatically trigger the execution of the subsequent test processes after the experiment release is completed, and after the test result is obtained, the test result can be directly reported to corresponding personnel, so that the automatic test of the configuration change of the service system can be realized in an all-around manner without manual intervention.
Based on the above description, a detailed description of a method for testing a service system change provided in the present application is provided below in connection with a specific embodiment. It should be noted that, the execution subject of each embodiment in the present application may be a device having a data processing function, such as a server, a processor, a computer, and a chip, and the embodiment is not limited thereto, and any device having a data processing function may be used as the execution subject of each embodiment in the present application.
The following description will first be made with reference to fig. 2, where fig. 2 is a flowchart of a method for testing a service system change provided in an embodiment of the present application.
As shown in fig. 2, the method includes:
s201, monitoring a message queue to obtain a change message corresponding to a change event of the service system from the message queue, wherein the message queue is used for writing the change message corresponding to the change event when the change event exists in the service system.
In this embodiment, a developer may, for example, perform a configuration change of the service system, and after the service system performs a configuration change, for example, an experiment may be issued for the service system, and the experiment may be, for example, a new experiment or an adjustment for an existing experiment.
Meanwhile, it can be determined that after the service system performs a configuration change, a change event can be considered to exist, and in combination with the above description, the change event can be a new experiment or the change event can also be a change for an existing experiment, and the embodiment does not limit a specific implementation manner of the change event, and all events that change the configuration of the service system can be considered to be change events in the embodiment.
For example, in connection with the above-described example, if a developer makes a configuration change for a business system to add an experiment D, it can be considered that there is a change event corresponding to the new experiment D. Or if the developer performs configuration change on the service system, so as to adjust the original experiment A to obtain an experiment A ', a change event can be considered to exist, and the change event corresponds to the adjustment of the experiment A to the experiment A'.
In one possible implementation, after the service system makes the configuration change, a change message corresponding to the change event may be generated and written into the message queue, where the change message is generated and written into the message queue, which may be accomplished, for example, through an experiment platform.
The processing device in this embodiment may monitor the message queue, and when it is monitored that there is a message in the message queue, it may consume the message in the message queue, so as to obtain a change message corresponding to a change event of the service system. The change message corresponding to the change event may include related event data, and specific data content of the event data may be selected and set according to actual requirements.
S202, acquiring a target test case corresponding to a change event from a plurality of preset test cases according to the change message.
In this embodiment, a plurality of test cases may be preset, and after a change message corresponding to a change event is obtained, a target test case corresponding to the change event may be obtained from the plurality of preset test cases according to the change message.
In one possible implementation manner, the change message may include event information corresponding to the change event, for example, a mapping relationship between the event information and a preset test case may be preset, and for example, matching may be performed according to the mapping relationship, so as to obtain a target test case corresponding to the change event from a plurality of preset test cases.
Or, the matching degree can be determined according to the event information in the change message and the case information of the test case, so as to determine the target test case corresponding to the change event. The embodiment does not limit the specific implementation manner of determining the target test case, as long as the target test case is determined in a plurality of preset test cases and is the test case corresponding to the change event.
S203, executing test operation on the changed service system according to the target test case so as to obtain a test result corresponding to the change event.
In this embodiment, after the service system performs the configuration change, the changed service system needs to be tested to determine the validity of the configuration change. Therefore, after the target test case corresponding to the change event is determined, the test operation can be automatically executed on the changed service system according to the target test case, so that the test result corresponding to the change event is obtained, wherein the test result corresponding to the change event is actually the test result of the experiment corresponding to the change event.
The test result can indicate that the service system after the change can stably run and can reach the expected purpose, and then the test result corresponding to the change event can be determined to be successful. Or the test result can indicate that the service system after the change cannot stably run or cannot achieve the expected purpose, and then the test result corresponding to the change event can be determined to be failure.
After determining the test result corresponding to the change event, the test result may be sent to the target device corresponding to the change event, for example. Or the target equipment can actively acquire the test result, and a specific test result reporting mode can be selected and set according to actual requirements.
The target device corresponding to the change event may be, for example, a device of an initiator of the experiment corresponding to the change event, or may also be a device of a responsible person of the experiment corresponding to the change event, which is not limited in this embodiment.
The obtained test results are sent to the target device corresponding to the change event in the embodiment, so that a series of test procedures can be automatically executed after the configuration change of the service system is monitored, and finally the obtained test results are automatically reported to corresponding personnel, so that the quick and effective test for the configuration change of the service system is realized.
The method for testing the service system change provided by the embodiment of the application comprises the following steps: the message queue is monitored to obtain the change message corresponding to the change event of the service system from the message queue, and the message queue is used for writing the change message corresponding to the change event when the change event exists in the service system. And acquiring a target test case corresponding to the change event from a plurality of preset test cases according to the change message. And executing test operation on the changed service system according to the target test case so as to obtain a test result corresponding to the change event. When a change event exists in the service system, writing the change message of the change event into a message queue, then automatically acquiring the change message from the message queue by monitoring the message queue, and completing automatic test operation on the changed service system according to the change message, so that the full-automatic processing flow of the service system after the change can be realized, and the test efficiency of the service system after the change can be effectively improved.
Based on the description, it can be determined that after the modification message is acquired from the monitoring message queue, the target test case corresponding to the modification event needs to be acquired from a plurality of preset test cases according to the modification message, and based on the description, in the service system modification test method provided by the application with reference to fig. 3 to 5, the implementation manner of the target test case is determined to be described in further detail. Fig. 3 is a flowchart two of a testing method for service system change provided in the embodiment of the present application, fig. 4 is a schematic implementation diagram of a preset mapping relationship provided in the embodiment of the present application, and fig. 5 is a schematic implementation diagram of a filtering change message provided in the embodiment of the present application.
As shown in fig. 3, the method includes:
s301, judging whether a changed service line identifier exists in the service line identifier set, if so, executing S302, and if not, executing S307.
Based on the above description, it may be determined that the change event may be a new experiment or may be an adjustment to an existing experiment. And each experiment belongs to a certain application, and each application belongs to a certain service line, so in this embodiment, the change message corresponding to the change event may include, for example: a change business line identifier corresponding to a change event, a change application identifier corresponding to the change event and a change experiment identifier corresponding to the change event.
The change experiment corresponding to the change event is subordinate to the change application, and the change application corresponding to the change event is subordinate to the change business line, that is, the change business line identifier, the change application identifier and the change experiment identifier corresponding to the change event follow the membership of the specific experiment, application and business line.
For example, taking the example described in fig. 1 as an example, assuming that the current change event is an adjustment for experiment a, it may be determined that the change experiment identifier corresponding to the change event is experiment a, the change application identifier corresponding to the change event is application B, and the change business line identifier corresponding to the change event is business line a.
After the change message corresponding to the change event is acquired, the target test case corresponding to the change event is acquired from a plurality of preset test cases according to the change message.
In one possible implementation manner, for example, a service line identifier set may be preset, where the service line identifier set includes all service lines in the service system, and in order to ensure that a target test case corresponding to a change event can be effectively determined, it may be first determined whether a change service line identifier included in a change message exists in the service line identifier set.
In one possible implementation manner, if the service line identifier set does not have a change service line identifier, it may be determined that the service line corresponding to the current change event is not a service line included in the service system, in such a scenario, there may be an event information error of the change event itself, or an abnormality of the system, in any case, the change service line identifier corresponding to the change event cannot be matched in such a scenario, and then the change message corresponding to the current change event may be directly ignored. And further sending notification information to the target device to notify that the service line corresponding to the current change event is not the service line existing in the service system.
S302, determining whether a test case corresponding to the changed experiment identifier exists according to a preset mapping relation, if so, executing S303, and if not, executing S304.
In one possible implementation manner, if the service line identifier set includes a change service line identifier, the service line corresponding to the current change event may be considered to be the service line existing in the service system, and the matching operation of the subsequent target test case may be continuously performed.
In this embodiment, for example, a preset mapping relationship may be set, where the preset mapping relationship is used to indicate a corresponding relationship among a preset test case, an experiment identifier, and an application identifier, and the corresponding relationship may be expressed as an application- > experiment- > test case, for example. For example, as can be understood with reference to fig. 4, as shown in fig. 4, it is assumed that the corresponding relationship between the application B, the experiment a, and the test case 1 and the corresponding relationship between the application B, the experiment B, and the test case 2 are configured in the preset mapping relationship, and other implementation manners are similar and are not repeated here. In the actual implementation process, the corresponding relation among the test cases, the experiments and the applications included in the preset mapping relation can be selected and set according to the actual requirements.
The change message in this embodiment includes a change experiment identifier corresponding to the change event, and on the basis of determining that the change service line identifier exists in the service line identifier set, it may further determine whether a test case corresponding to the change experiment identifier exists according to the above-described preset mapping relationship.
S303, determining the test case corresponding to the change experiment identifier in the preset mapping relation as a target test case.
In one possible implementation manner, if the preset mapping relationship includes a correspondence between the modification experiment identifier and the corresponding test case, it may be determined that the test case corresponding to the modification experiment identifier exists according to the preset mapping relationship, that is, the test case corresponding to the experiment to which the current modification event belongs is preconfigured. The test case corresponding to the change experiment identifier in the preset mapping relation can be directly determined as the target test case.
For example, if the change experiment identifier corresponding to the change event in the change message is experiment a, then according to the preset mapping relationship shown in fig. 4, it may be determined that the test case 1 corresponding to the experiment a is the target test case.
S304, determining whether a test case corresponding to the changed application identifier exists according to a preset mapping relation, if so, executing S305, and if not, executing S306.
In another possible implementation manner, if the preset mapping relationship does not include the correspondence between the modification experiment identifier and the corresponding test case, it may be determined that the test case corresponding to the modification experiment identifier does not exist according to the preset mapping relationship, that is, the test case corresponding to the experiment to which the current modification event belongs is not configured in advance.
In this case, whether the test case corresponding to the changed application identifier exists may be further determined according to the preset mapping relationship.
The change message in this embodiment includes a change application identifier corresponding to the change event, and on the basis that it is determined that the change service line identifier exists in the service line identifier set and that the test case corresponding to the change experiment identifier does not exist, it may be further determined whether the test case corresponding to the change application identifier exists according to the described preset mapping relationship.
S305, determining the test case corresponding to the change application identifier in the preset mapping relation as a target test case.
In one possible implementation manner, if the preset mapping relationship includes a correspondence between the change application identifier and the corresponding test case, it may be determined that the test case corresponding to the change application identifier exists according to the preset mapping relationship, that is, the test case corresponding to the application to which the current change event belongs is preconfigured. The test case corresponding to the change application identifier in the preset mapping relation can be directly determined as the target test case.
If only one test case corresponding to the changed application identifier is in the preset mapping relation, the test case corresponding to the changed application identifier can be directly determined to be the target test case. Or if the plurality of test cases corresponding to the changed application identifier in the preset mapping relation are said, one or more test cases can be selected as target test cases from the plurality of test cases. The selection may be, for example, random selection, or may be selected according to priority, which is not limited in this embodiment.
The present change event is described herein with reference to a specific example, for example, the experiment D is newly added to the application B, and referring to the above-described processing procedure, it can be determined that the experiment D is affiliated with the application B, and that the application B is affiliated with the service line a. It may first be determined that traffic line a is present in the traffic line identification set.
And referring to fig. 4 described above, it may be determined that the preset mapping relationship does not include the correspondence between the experiment D and the corresponding test case. Therefore, it may be further removed, and it may be determined whether there is a test case corresponding to the application B according to the preset mapping relationship, and referring to fig. 5, it may be determined that the test case corresponding to the current application B includes the test case 1, the test case 2, and the test case 3. One or more of these 3 test cases may be selected and determined as target test cases, for example.
After determining the target test case, for example, notification information may be sent to the target device, where the notification information is used to indicate that the experiment corresponding to the current changed experiment identifier is changed, and the automated test has already been performed, so that the related personnel synchronize the test progress in time.
It should be noted that, in the case of the current step, the experiment D is a newly added experiment, and the corresponding relationship between the experiment D and the corresponding test case does not exist in the preset mapping relationship, which indicates that the test case is not configured for the experiment D in advance. In this case, in order to ensure that the automated test for experiment D can be successfully completed, the next test case corresponding to application B to which experiment D belongs may be selected for testing, and since the test data corresponding to one application is similar, the test cases can be tested for experiment D, and the suitability of the test case determined in the present case is slightly inferior to that of the test case specifically configured for experiment D.
It will be appreciated that if a corresponding test case is configured for each experiment, then the necessary test results are best, since this ensures that a specific test case is determined adaptively for each experiment. However, in some special cases, for some experiments, corresponding test cases are not configured, and the test cases corresponding to the applications to which the experiments belong are adopted for testing, so that smooth performance of automatic testing can be ensured.
S306, sending notification information to the target equipment, wherein the notification information is used for indicating that the target test case corresponding to the change event does not exist.
In another possible implementation manner, if it is determined that there is no test case corresponding to the change application identifier according to the second mapping relationship, it may be considered that the test case corresponding to the change event cannot be automatically matched. In this case, notification information may be sent to the target device, where the notification information is used to indicate that there is no target test case corresponding to the change event, and then the developer may perform further processing.
S307, the change message is ignored.
According to the testing method for service system change, whether the service line corresponding to the change event is the service line in the service system is determined, and then subsequent matching processing is performed on the basis of affirmation, so that subsequent processing on abnormal conditions can be effectively avoided, and resources consumed for abnormal data processing are saved. And in the subsequent matching process, when determining that the test case corresponding to the experiment to which the change event belongs exists, preferentially determining the test case corresponding to the experiment as a target test case, so that the test case corresponding to the change business line and the change experiment can be accurately matched. And under the condition that the test case corresponding to the change experiment does not exist, the test case corresponding to the application to which the one-step matching change event belongs is returned, and then the test case is determined as the target test case for subsequent automatic test, so that the success rate of matching the target test case can be effectively improved.
On the basis of the above description, after the change message of the change event is obtained from the message queue, for example, the change message may be filtered, where the filtering process may filter the message that does not need to be automatically tested, so as to improve the execution efficiency and effectiveness of the automatic test flow. The following is a detailed description with reference to fig. 5, and fig. 5 is a schematic message filtering diagram of a testing method for service system modification according to an embodiment of the present application.
As shown in fig. 5, after the change message is obtained from the message queue, the change message whose message type is not the preset type may be filtered out.
In one possible implementation, the change message may include a message type, for example, or may also include a message type carried by the change message, and the message type of the change message may be obtained.
When the message type of the change message is not the preset type, the change message can be considered to be not the message which needs to trigger the automatic test, and the change message can be directly discarded.
Or when the message type of the change message is a preset type, the change message can be considered as a message needing to trigger the automatic test, and the subsequent processing can be performed according to the change message.
In an actual implementation process, the message types may include an experiment change class, a pre-change class, an initiation flow class, a reset state class, and the like, for example, only the experiment change class needs to trigger a subsequent automatic test, and the preset type may be set as the experiment change class. In the actual implementation process, the specific message types can be selected and set according to the actual requirements, and all message types needing to trigger the automatic test can be used as preset types in the embodiment.
And in one possible implementation manner, as shown in fig. 5, in this embodiment, for example, test start information is configured, where the test start information is used to indicate whether an automation test corresponding to the change event is started. The test start information is, for example, information of a switch class, and when the test start information indicates that the automation test corresponding to the change event is not started, the current change event can be considered that the automation test does not need to be triggered, and the change message can be discarded.
Or when the test start information indicates that the automatic test corresponding to the change event is started, the automatic test corresponding to the change event can be considered to be required to be triggered, and subsequent processing can be performed according to the change message.
The test-on information may be configured globally, for example, the test-on information may be configured globally to be off, that is, all change events may not trigger an automated test. Or, the test start information may be configured for the application, for example, the test start information is configured for the application a to be closed, and it may be determined that all the change events corresponding to the experiments under the application a will not trigger the automatic test. Or the test starting information can also be specific to a service line or experimental configuration, etc., and the embodiment does not limit a specific implementation manner. In short, according to the test opening information and the corresponding information corresponding to the change event, it can be determined whether the test opening information specifically indicates that the automated test corresponding to the change event is opened or closed.
When the test start information indicates that the automated test corresponding to the change event is closed, after the change event occurs, for example, indication information can be sent to the target device corresponding to the change event to inform that the change event occurs currently, and meanwhile, the automated test flow of the corresponding experiment is not triggered.
And in one possible implementation manner, in order to improve the freedom degree of experiment configuration, for example, a blacklist mechanism can be further provided, and by adding some experiments into the blacklist, a change event corresponding to the experiments in the blacklist can be realized without triggering an automatic test flow. Then after the change message is acquired, if it is determined that the change message belongs to an experiment in the experiment blacklist, the change message may be selected for discarding.
Based on the description and fig. 5, it can be determined that, in this embodiment, the filtering of the obtained change message may be implemented by determining the message type, determining whether the automation test corresponding to the change event is started or not by testing the start information, and implementing the experimental blacklist, where the three filtering modes are not sequential, and may be set according to actual requirements, and by filtering the message in the message queue, it may be ensured that the automation test is triggered only for the change message that needs to trigger the automation test, thereby effectively avoiding the consumption of the server caused by frequently calling the corresponding interface, and also avoiding the query for the database. The implementation mode of the configuration test starting information can avoid interface calling errors or error data caused by test case execution when service is fused or degraded, wherein the configuration of whether the automatic test is triggered or not can be flexibly and effectively realized by the test starting information.
After determining the test result corresponding to the change event, reporting and reminding corresponding responsible personnel are generally required for abnormal test results. However, some reasons for automated testing may not be the result of experimental changes, but may also be the result of problems with the test data, which may lead to frequent annoyance to the responsible personnel if reminded of each test failure for an experiment.
In this embodiment, therefore, the experiment corresponding to the change event may be referred to as a change experiment, and then the total number of test cases executed for the change experiment may be first determined, and the number of test cases whose test results are failed among the test cases executed for the change experiment may be determined, and then the test failure ratio corresponding to the change experiment may be determined, for example, based on the ratio of the latter and the former.
For example, if the test is performed according to 10 test cases for the experiment a, and the test results corresponding to 2 test cases are failures, it can be determined that the test failure rate corresponding to the experiment a is 20%.
And when the test failure proportion corresponding to the change experiment is greater than or equal to a preset threshold value, sending alarm information to the target equipment, wherein the alarm information is used for indicating that the test corresponding to the change experiment is abnormal.
In one possible implementation, the alert may be sent, for example, by a preset means, for example, by a telephone, when the alert is sent to the target device. That is, compared with the above-described notification information for reporting the test result, the warning information has a better prompting property, so that the related personnel can find and intervene in the corresponding abnormal problem in time.
It can be understood that by the mode, when the test failure condition corresponding to the change experiment reaches a certain proportion, the corresponding responsible personnel is reminded, so that the disturbance to the responsible personnel caused by frequent reminding abnormality is avoided.
In the prior art, besides the problem of lower test efficiency caused by the need of manually triggering the execution of the test flow, the problem of summarizing and optimizing the problem of adverse service quality development caused by the incapability of automatically recording experimental data and test results and consequently incapability of summarizing multi-dimensional quality data is also existed. Based on this, in the testing method for service system change provided by the application, the testing result can be further summarized and automatically analyzed, and the further detailed description is given below for the part of the content.
In this embodiment, after the test result is generated, for example, the generation time corresponding to the test result may be obtained, then the record message corresponding to the change event may be generated according to the generation time of the test result, the test result and the change message corresponding to the change event, and then the record message may be stored in the preset storage space
The recording message may be of the form: { change message corresponding to change event, test result, time of generation of test result }. A record message can be generated according to each change event, and the specific content in the record message can be selected and set according to actual requirements, which is not limited in this embodiment.
Then, the record information corresponding to each of the plurality of change events is stored in the preset storage space, and then, the test statistical information in the target period can be determined according to the record information stored in the preset storage space.
The target time period can be set according to actual requirements, and when screening is performed according to the target time period, the screening generation time can be the recorded message in the target time period, and then further screening is performed on the basis.
The test statistics may for example comprise the test frequency, that is to say how many times the test was triggered within a target period. In one possible implementation, for example, the test frequency of the global (that is to say, all traffic lines) in the target period may be counted, or the test frequency of the specified traffic line in the target period may be counted, or the test frequency of the specified test team in the target period may be counted. The specific attribution of the test frequency can be selected and set according to actual requirements.
The test statistics may further include a fail test duty ratio corresponding to the target service line, where the fail test duty ratio may be a ratio of a total number of test cases for which the test result is a fail in the target period and a number of test cases for which the test result is a fail in the target period.
Assuming that the total number of test cases with failed test results in the current target period is 40, and the test cases with failed test results corresponding to the target service line "service line a" is 14, it can be determined that the failed test duty ratio corresponding to the target service line "service line a" is 35%.
The test statistics information may further include a failure test duty ratio corresponding to the target application, a failure test duty ratio corresponding to the target experiment, and the like, where the meaning of the failure test duty ratio is similar, and is not described herein.
The test statistics information may further include a test list of test results failing, for example, the test list of test results failing in the target period may be listed, and the total number of test cases (cases) of the tests and the number of test cases of the test failing may be also listed.
And the test statistics information can also comprise the configuration change frequency of the experiments in the target time period, for example, the configuration change times of the target experiments in different time periods can be listed. Say 12 changes in the configuration of the target experiment at 10 months, 21 changes in the configuration of the target experiment at 11 months, etc.
And the test statistics may also include a ranking of the configuration change frequencies of the plurality of experiments within the target period, e.g., the configuration change frequency within the target period ranks experiment a first, experiment B second, and so on.
In the actual implementation process, the specific implementation mode of the test statistical information can be selected and set according to actual requirements, and all the statistical data obtained by analysis according to the record information stored in the preset storage space and required by corresponding personnel can be used as the test statistical information in the embodiment. The method has the advantages that the recorded information can be generated by recovering the experimental test result and the experimental changed basic data, and then the statistical multidimensional quality data can be customized according to the recorded information, so that even analysis and discovery of key problems can be carried out, reliable data basis can be provided for subsequent adjustment of service quality, and therefore, the record and statistics of the test data can be realized to the greatest extent, and the data utilization rate can be improved.
On the basis of the above description, the following description is made with reference to fig. 6, and fig. 6 is a schematic flow chart of a testing method for service system modification provided in the present application.
As shown in fig. 6, when a change event exists in the experiment for the service system, a message queue may be sent out, and then, for example, the configuration platform may monitor the message queue, so as to obtain a change message corresponding to the change event from the message queue.
And after the change message is acquired, the change message may be filtered first, and specific implementation of the filtering process may refer to the description of the foregoing embodiments. For the change message remaining after the filtering process is completed, relevant configuration data in the change message, such as a change service line identifier, a change application identifier, a change experiment identifier, and the like corresponding to the change event described above, can be obtained.
Then, the target test case can be determined according to the related configuration data in the change message, an automatic test flow is executed, a message notification can be sent to the corresponding target device after the automatic test is triggered, and a test result of the automatic test can be obtained through the callback interface.
The corresponding notification information may then be sent to the target device based on the test results, where the notification information may include, for example, a report of the test results described in the foregoing embodiment, or may further include warning information of test failure described in the foregoing embodiment.
And generating record information corresponding to the change event based on the test result, storing the record information, and further determining custom test statistical information based on the stored record information for analyzing the risk of possible configuration change.
In summary, in the method for testing service system change provided in the embodiment of the present application, on one hand, a series of automatic processes such as message queues may be set, so that corresponding test cases are automatically triggered at the first time when configuration change is completed, so that the manual triggering cost of manual access is avoided, and in addition, in case that corresponding conditions are met, alarm information is timely sent to the target device, so that loss caused by untimely problem discovery is reduced, and further, the test efficiency of experimental change is effectively improved. Meanwhile, by storing the record information corresponding to the change event, the follow-up custom data measurement statistics can be conveniently carried out, so that effective data reference is provided for the adjustment of the service system.
Fig. 7 is a schematic structural diagram of a testing device for service system modification according to an embodiment of the present application. As shown in fig. 7, the apparatus 70 includes: a processing module 701, an acquisition module 702, a testing module and a sending module 704.
The processing module 701 is configured to monitor a message queue, so as to obtain a change message corresponding to a change event of the service system from the message queue, where the message queue is configured to write the change message corresponding to the change event when the change event exists in the service system;
the obtaining module 702 is configured to obtain, according to the change message, a target test case corresponding to the change event from a plurality of preset test cases;
and the test module 703 is configured to perform a test operation on the changed service system according to the target test case, so as to obtain a test result corresponding to the change event.
In one possible design, the change message corresponding to the change event includes: a change business line identifier corresponding to the change event, a change application identifier corresponding to the change event and a change experiment identifier corresponding to the change event;
the business system comprises at least one business line, any business line comprises at least one application, at least one experiment is configured in any application, and the change event is configured change aiming at the existing experiment or is a new experiment.
In one possible design, the obtaining module 702 is specifically configured to:
if the changed service line identifier exists in the service line identifier set, determining whether a test case corresponding to the changed experiment identifier exists or not according to a preset mapping relation, wherein the preset mapping relation is used for indicating the corresponding relation among the preset test case, the experiment identifier and the application identifier;
if the target test case exists, determining the test case corresponding to the change experiment identifier in the preset mapping relation as the target test case; or alternatively, the process may be performed,
if the change event does not exist, acquiring a target test case corresponding to the change event from a plurality of preset test cases according to the change application identifier.
In one possible design, the obtaining module 702 is specifically configured to:
determining whether a test case corresponding to the changed application identifier exists or not according to the preset mapping relation;
if so, determining the test case corresponding to the change application identifier in the preset mapping relation as the target test case.
In one possible design, the transmitting module 704 is configured to:
transmitting notification information to the target device, wherein the notification information is used for indicating that the automated test of the experiment corresponding to the changed experiment identification starts to be executed
In one possible design, the transmitting module 704 is further configured to:
and if the test case corresponding to the change application identifier does not exist in the preset mapping relation, sending notification information to target equipment, wherein the notification information is used for indicating that the target test case corresponding to the change event does not exist.
In one possible design, the processing module 701 is further configured to:
after the change message corresponding to the change event of the service system is obtained, obtaining the message type of the change message;
and discarding the change message if the message type of the change message is not the preset type.
In one possible design, the processing module 701 is further configured to:
after the change message corresponding to the change event of the service system is acquired, acquiring test starting information, wherein the test starting information is used for indicating whether an automatic test corresponding to the change event is started or not;
and if the test starting information indicates that the automatic test corresponding to the change event is not started, discarding the change message.
In one possible design, the processing module 701 is further configured to:
after the test operation is executed on the changed service system according to the target test cases so as to obtain the test result corresponding to the change event, determining the test failure proportion corresponding to the change experiment according to the total number of the test cases executed for the change experiment and the number of the test cases with failed test results in the test cases executed for the change experiment, wherein the change experiment is the experiment corresponding to the change event;
And if the test failure proportion is greater than or equal to a preset threshold value, sending alarm information to the target equipment, wherein the alarm information is used for indicating the test abnormality corresponding to the change experiment.
In one possible design, the processing module 701 is further configured to:
after the test operation is executed on the changed service system according to the target test case so as to obtain a test result corresponding to the change event, acquiring a generation moment corresponding to the test result;
and generating a record message corresponding to the change event according to the generation time corresponding to the test result, the test result and the change message of the change event, and storing the record message in a preset storage space.
In one possible design, the processing module 701 is further configured to:
after the recorded messages are stored, determining test statistical information in a target period according to the recorded messages stored in the preset storage space;
the test statistics include at least one of: the test frequency, the failure test duty ratio corresponding to the target service line and the test result have a failed experiment list.
The device provided in this embodiment may be used to implement the technical solution of the foregoing method embodiment, and its implementation principle and technical effects are similar, and this embodiment will not be described herein again.
Fig. 8 is a schematic hardware structure of an electronic device provided in an embodiment of the present application, as shown in fig. 8, an electronic device 80 in the embodiment includes: a processor 801 and a memory 802; wherein the method comprises the steps of
A memory 802 for storing computer-executable instructions;
the processor 801 is configured to execute computer-executable instructions stored in the memory to implement the steps executed by the testing method for service system modification in the above embodiment. Reference may be made in particular to the relevant description of the embodiments of the method described above.
Alternatively, the memory 802 may be separate or integrated with the processor 801.
When the memory 802 is provided separately, the electronic device further comprises a bus 803 for connecting said memory 802 and the processor 801.
The embodiment of the application also provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and when a processor executes the computer execution instructions, the method for testing the business system change executed by the electronic equipment is realized.
In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, the division of the modules is merely a logical function division, and there may be additional divisions when actually implemented, for example, multiple modules may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or modules, which may be in electrical, mechanical, or other forms.
The integrated modules, which are implemented in the form of software functional modules, may be stored in a computer readable storage medium. The software functional module is stored in a storage medium, and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (english: processor) to perform some of the steps of the methods described in the embodiments of the present application.
It should be understood that the above processor may be a central processing unit (english: central Processing Unit, abbreviated as CPU), or may be other general purpose processors, digital signal processors (english: digital Signal Processor, abbreviated as DSP), application specific integrated circuits (english: application Specific Integrated Circuit, abbreviated as ASIC), or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present invention may be embodied directly in a hardware processor for execution, or in a combination of hardware and software modules in a processor for execution.
The memory may comprise a high-speed RAM memory, and may further comprise a non-volatile memory NVM, such as at least one magnetic disk memory, and may also be a U-disk, a removable hard disk, a read-only memory, a magnetic disk or optical disk, etc.
The bus may be an industry standard architecture (Industry Standard Architecture, ISA) bus, an external device interconnect (Peripheral Component, PCI) bus, or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, among others. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, the buses in the drawings of the present application are not limited to only one bus or one type of bus.
The storage medium may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the method embodiments described above may be performed by hardware associated with program instructions. The foregoing program may be stored in a computer readable storage medium. The program, when executed, performs steps including the method embodiments described above; and the aforementioned storage medium includes: various media that can store program code, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims (10)

1. A method for testing changes to a business system, comprising:
monitoring a message queue to acquire a change message corresponding to a change event of the service system from the message queue, wherein the message queue is used for writing the change message corresponding to the change event when the change event exists in the service system;
according to the change message, acquiring a target test case corresponding to the change event from a plurality of preset test cases;
and executing test operation on the changed service system according to the target test case so as to obtain a test result corresponding to the change event.
2. The method according to claim 1, wherein the change message includes: a change business line identifier corresponding to the change event, a change application identifier corresponding to the change event and a change experiment identifier corresponding to the change event;
The business system comprises at least one business line, any business line comprises at least one application, at least one experiment is configured in any application, and the change event is configured change aiming at the existing experiment or is a new experiment.
3. The method of claim 2, wherein the obtaining, according to the change message, the target test case corresponding to the change event from a plurality of preset test cases includes:
if the changed service line identifier exists in the service line identifier set, determining whether a test case corresponding to the changed experiment identifier exists or not according to a preset mapping relation, wherein the preset mapping relation is used for indicating the corresponding relation among the preset test case, the experiment identifier and the application identifier;
if the target test case exists, determining the test case corresponding to the change experiment identifier in the preset mapping relation as the target test case;
if the change event does not exist, acquiring a target test case corresponding to the change event from a plurality of preset test cases according to the change application identifier.
4. The method of claim 3, wherein the obtaining, according to the change application identifier, the target test case corresponding to the change event from a plurality of preset test cases includes:
Determining whether a test case corresponding to the changed application identifier exists or not according to the preset mapping relation;
if so, determining the test case corresponding to the change application identifier in the preset mapping relation as the target test case.
5. The method according to any one of claims 1-4, wherein after performing a test operation on the changed service system according to the target test case to obtain a test result corresponding to the change event, the method further includes:
determining a test failure proportion corresponding to a change experiment according to the total number of test cases executed for the change experiment and the number of test cases with failed test results in the test cases executed for the change experiment, wherein the change experiment is an experiment corresponding to the change event;
and if the test failure proportion is greater than or equal to a preset threshold value, sending alarm information to the target equipment, wherein the alarm information is used for indicating the test abnormality corresponding to the change experiment.
6. The method according to any one of claims 1-5, wherein after performing a test operation on the changed service system according to the target test case to obtain a test result corresponding to the change event, the method further includes:
Acquiring a generation moment corresponding to the test result;
and generating a record message corresponding to the change event according to the generation time corresponding to the test result, the test result and the change message of the change event, and storing the record message in a preset storage space.
7. The method of claim 6, wherein after said storing said recorded message, said method further comprises:
determining test statistical information in a target period according to the record information stored in the preset storage space;
the test statistics include at least one of: the test frequency, the failure test duty ratio corresponding to the target service line and the test result have a failed experiment list.
8. A test device for service system change, comprising:
the processing module is used for monitoring a message queue to acquire a change message corresponding to a change event of the service system from the message queue, wherein the message queue is used for writing the change message corresponding to the change event when the change event exists in the service system;
the acquisition module is used for acquiring a target test case corresponding to the change event from a plurality of preset test cases according to the change message;
And the test module is used for executing test operation on the changed service system according to the target test case so as to obtain a test result corresponding to the change event.
9. An electronic device, comprising:
a memory for storing a program;
a processor for executing the program stored by the memory, the processor being for performing the method of any one of claims 1 to 7 when the program is executed.
10. A computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1 to 7.
CN202310120093.0A 2023-02-09 2023-02-09 Service system change testing method and device Pending CN116302975A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310120093.0A CN116302975A (en) 2023-02-09 2023-02-09 Service system change testing method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310120093.0A CN116302975A (en) 2023-02-09 2023-02-09 Service system change testing method and device

Publications (1)

Publication Number Publication Date
CN116302975A true CN116302975A (en) 2023-06-23

Family

ID=86796968

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310120093.0A Pending CN116302975A (en) 2023-02-09 2023-02-09 Service system change testing method and device

Country Status (1)

Country Link
CN (1) CN116302975A (en)

Similar Documents

Publication Publication Date Title
CN112801666B (en) Monitoring management method, system, medium and equipment based on enterprise service bus
US9176803B2 (en) Collecting data from a system in response to an event based on an identification in a file of the data to collect
US8108724B2 (en) Field replaceable unit failure determination
US20080195614A1 (en) Proactive Analysis of Communication Network Problems
EP2645257A2 (en) System and method for visualisation of behaviour within computer infrastructure
CN111045881A (en) Slow disk detection method and system
US7933211B2 (en) Method and system for providing prioritized failure announcements
CN111881014A (en) System test method, device, storage medium and electronic equipment
CN102609350A (en) Server memory failure alarm method
CN112579400A (en) Equipment fault positioning method, device, equipment and storage medium
CN102045213B (en) Fault positioning method and device
CN109818808B (en) Fault diagnosis method and device and electronic equipment
CN113590405A (en) Hard disk error detection method and device, storage medium and electronic device
CN110633165B (en) Fault processing method, device, system server and computer readable storage medium
CN116302975A (en) Service system change testing method and device
CN112306871A (en) Data processing method, device, equipment and storage medium
CN114500249B (en) Root cause positioning method and device
CN113835961B (en) Alarm information monitoring method, device, server and storage medium
CN110968456A (en) Method and device for processing fault disk in distributed storage system
CN114978885A (en) Log management method and device, computer equipment and system
CN114138600A (en) Storage method, device, equipment and storage medium for firmware key information
CN113676356A (en) Alarm information processing method and device, electronic equipment and readable storage medium
CN112882892A (en) Data processing method and device, electronic equipment and storage medium
CN111694715A (en) Abnormity warning method, device, equipment and machine readable storage medium
CN111061609A (en) Log monitoring method and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination