CN116719710A - Data processing method, device, electronic equipment and computer readable medium - Google Patents
Data processing method, device, electronic equipment and computer readable medium Download PDFInfo
- Publication number
- CN116719710A CN116719710A CN202310711959.5A CN202310711959A CN116719710A CN 116719710 A CN116719710 A CN 116719710A CN 202310711959 A CN202310711959 A CN 202310711959A CN 116719710 A CN116719710 A CN 116719710A
- Authority
- CN
- China
- Prior art keywords
- fluctuation
- similarity
- component
- log information
- fluctuation type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title abstract description 23
- 238000000034 method Methods 0.000 claims abstract description 127
- 238000012360 testing method Methods 0.000 claims abstract description 127
- 230000008569 process Effects 0.000 claims abstract description 106
- 238000005516 engineering process Methods 0.000 claims abstract description 12
- 238000004806 packaging method and process Methods 0.000 claims abstract description 5
- 239000013598 vector Substances 0.000 claims description 75
- 238000012545 processing Methods 0.000 claims description 27
- 238000013507 mapping Methods 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 16
- 230000004044 response Effects 0.000 claims description 10
- 238000001514 detection method Methods 0.000 claims description 9
- 238000000605 extraction Methods 0.000 claims description 9
- 230000009191 jumping Effects 0.000 claims description 7
- 238000012430 stability testing Methods 0.000 abstract description 3
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 13
- 238000013112 stability test Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000012544 monitoring process Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000013145 classification model Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000012827 research and development Methods 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 206010000117 Abnormal behaviour Diseases 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000011990 functional testing Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000026676 system process Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a data processing method, a device, electronic equipment and a computer readable medium, which relate to the technical field of artificial intelligence, and one specific implementation mode comprises the following steps: invoking a dynamic proxy technology, packaging a public component class, detecting data fluctuation by a tangent plane mode, responding to the detected data fluctuation, acquiring current fluctuation time and a test component identifier, acquiring a corresponding component portrait based on the test component identifier, and determining historical fluctuation time according to the component portrait; determining a first boundary time point of the current fluctuation time and a second boundary time point of the historical fluctuation time, and generating a fluctuation time interval according to the first boundary time point and the second boundary time point; extracting performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval; and determining and outputting the fluctuation type according to the performance index, the log information and the process information. The manual handling effort can be reduced in stability testing.
Description
Technical Field
The present application relates to the field of artificial intelligence, and in particular, to a data processing method, apparatus, electronic device, and computer readable medium.
Background
In the stability test, the desired performance index is also smooth without fluctuations if there is no change in the parameters. In actual operation, due to factors such as environmental changes, problems of the product itself and the like, certain fluctuation exists in the test result. For these fluctuations, it is necessary for testers and developers to locate the problem one by one. Various performance indicators of related components and related platforms when the fluctuation occurs, and corresponding log information when the fluctuation occurs, are typically viewed. In this case, if the system components are relatively more, the system components need to be checked on different servers respectively; if the log quantity is relatively large, the log is deleted at regular time, so that the related log cannot be found, and the processing efficiency of the fluctuation data is low.
Disclosure of Invention
In view of the above, embodiments of the present application provide a data processing method, apparatus, electronic device, and computer readable medium, which can solve the problem of low processing efficiency of data that fluctuates in the existing stability test.
To achieve the above object, according to an aspect of an embodiment of the present application, there is provided a data processing method including:
invoking a dynamic proxy technology, packaging a public component class, detecting data fluctuation in a tangent plane mode, responding to the detected data fluctuation, acquiring current fluctuation time and a test component identifier, acquiring a corresponding component portrait based on the test component identifier, and further determining historical fluctuation time according to the component portrait;
determining a first boundary time point of the current fluctuation time and a second boundary time point of the historical fluctuation time, and further generating a fluctuation time interval according to the first boundary time point and the second boundary time point;
extracting performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval;
and determining and outputting the fluctuation type according to the performance index, the log information and the process information.
Optionally, extracting performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval includes:
acquiring a component address corresponding to the test component identifier;
and jumping to a component test database corresponding to the component address to extract performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval.
Optionally, determining the type of fluctuation includes:
and calling a fluctuation type database to match the performance index, the log information and the process information with the fluctuation types in the fluctuation type database, and determining the fluctuation types obtained by matching as the fluctuation types corresponding to the performance index, the log information and the process information.
Optionally, matching the performance index, the log information, and the process information with the fluctuation type in the fluctuation type database includes:
mapping the performance index into a performance index vector, mapping the log information into a log information vector, and mapping the process information into a process information vector based on a word embedding method;
mapping each fluctuation type in the fluctuation type database into each fluctuation type vector;
calculating first similarity of the performance index vector and each fluctuation type vector, calculating second similarity of the log information vector and each fluctuation type vector, and calculating third similarity of the process information vector and each fluctuation type vector;
and determining the fluctuation type matched with the performance index, the log information and the process information in the fluctuation type database based on the first similarity, the second similarity and the third similarity.
Optionally, determining the fluctuation type matching the performance index, the log information and the process information in the fluctuation type database based on the first similarity, the second similarity and the third similarity includes:
calculating the similarity sum of the first similarity, the second similarity and the third similarity corresponding to each fluctuation type in the fluctuation type database;
and determining the fluctuation type matched with the performance index, the log information and the process information in the fluctuation type database according to the similarity sum.
Optionally, determining the historical fluctuation time from the component representation includes:
obtaining each testing stage label corresponding to the component image;
summarizing the corresponding test stage labels when the data fluctuation occurs to obtain a label set;
and determining the historical fluctuation time according to the label set.
Optionally, determining the historical fluctuation time according to the tag set includes:
determining a timestamp corresponding to the tag set;
from the time stamp, a historical fluctuation time is determined.
In addition, the application also provides a data processing device, which comprises:
the acquisition unit is configured to call a dynamic proxy technology, package a public component class, detect data fluctuation in a tangent plane mode, acquire current fluctuation time and test component identification in response to the detection of the data fluctuation, acquire corresponding component portraits based on the test component identification, and further determine historical fluctuation time according to the component portraits;
A fluctuation time interval generation unit configured to determine a first boundary time point of the current fluctuation time and a second boundary time point of the historical fluctuation time, and further generate a fluctuation time interval according to the first boundary time point and the second boundary time point;
the extraction unit is configured to extract performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval;
and a fluctuation type determination unit configured to determine a fluctuation type from the performance index, the log information, and the process information and output.
Optionally, the extraction unit is further configured to:
acquiring a component address corresponding to the test component identifier;
and jumping to a component test database corresponding to the component address to extract performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval.
Optionally, the fluctuation type determination unit is further configured to:
and calling a fluctuation type database to match the performance index, the log information and the process information with the fluctuation types in the fluctuation type database, and determining the fluctuation types obtained by matching as the fluctuation types corresponding to the performance index, the log information and the process information.
Optionally, the fluctuation type determination unit is further configured to:
mapping the performance index into a performance index vector, mapping the log information into a log information vector, and mapping the process information into a process information vector based on a word embedding method;
mapping each fluctuation type in the fluctuation type database into each fluctuation type vector;
calculating first similarity of the performance index vector and each fluctuation type vector, calculating second similarity of the log information vector and each fluctuation type vector, and calculating third similarity of the process information vector and each fluctuation type vector;
and determining the fluctuation type matched with the performance index, the log information and the process information in the fluctuation type database based on the first similarity, the second similarity and the third similarity.
Optionally, the fluctuation type determination unit is further configured to:
calculating the similarity sum of the first similarity, the second similarity and the third similarity corresponding to each fluctuation type in the fluctuation type database;
and determining the fluctuation type matched with the performance index, the log information and the process information in the fluctuation type database according to the similarity sum.
Optionally, the acquisition unit is further configured to:
Obtaining each testing stage label corresponding to the component image;
summarizing the corresponding test stage labels when the data fluctuation occurs to obtain a label set;
and determining the historical fluctuation time according to the label set.
Optionally, the acquisition unit is further configured to:
determining a timestamp corresponding to the tag set;
from the time stamp, a historical fluctuation time is determined.
In addition, the application also provides data processing electronic equipment, which comprises: one or more processors; and a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the data processing method as described above.
In addition, the application also provides a computer readable medium, on which a computer program is stored, which when executed by a processor implements the data processing method as described above.
To achieve the above object, according to still another aspect of an embodiment of the present application, there is provided a computer program product.
A computer program product of an embodiment of the present application includes a computer program, which when executed by a processor implements a data processing method provided by the embodiment of the present application.
One embodiment of the above application has the following advantages or benefits: according to the application, a public component class is packaged by calling a dynamic proxy technology, data fluctuation detection is performed in a tangent plane mode, current fluctuation time and a test component identifier are obtained in response to the detected data fluctuation, a corresponding component portrait is obtained based on the test component identifier, and then historical fluctuation time is determined according to the component portrait; determining a first boundary time point of the current fluctuation time and a second boundary time point of the historical fluctuation time, and further generating a fluctuation time interval according to the first boundary time point and the second boundary time point; extracting performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval; and determining and outputting the fluctuation type according to the performance index, the log information and the process information. Therefore, the fluctuation data can be efficiently processed in the stability test, and the manual processing workload is reduced.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the application and are not to be construed as unduly limiting the application. Wherein:
FIG. 1 is a schematic diagram of the main flow of a data processing method according to one embodiment of the application;
FIG. 2 is a schematic diagram of the main flow of a data processing method according to one embodiment of the application;
FIG. 3 is a schematic diagram of the main flow of a data processing method according to one embodiment of the application;
FIG. 4 is a schematic diagram of the main units of a data processing apparatus according to an embodiment of the present application;
FIG. 5 is an exemplary system architecture diagram in which embodiments of the present application may be applied;
fig. 6 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the application.
Detailed Description
Exemplary embodiments of the present application will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present application are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness. In the technical scheme of the application, the aspects of acquisition, analysis, use, transmission, storage and the like of the related user personal information all meet the requirements of related laws and regulations, are used for legal and reasonable purposes, are not shared, leaked or sold outside the aspects of legal use and the like, and are subjected to supervision and management of a supervision department. Necessary measures should be taken for the personal information of the user to prevent illegal access to such personal information data, ensure that personnel having access to the personal information data comply with the regulations of the relevant laws and regulations, and ensure the personal information of the user. Once these user personal information data are no longer needed, the risk should be minimized by limiting or even prohibiting the data collection and/or deletion.
User privacy is protected by de-identifying data when used, including in some related applications, such as by removing a particular identifier, controlling the amount or specificity of stored data, controlling how data is stored, and/or other methods.
FIG. 1 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application, and as shown in FIG. 1, the data processing method includes:
step S101, calling a dynamic proxy technology, packaging a public component class, detecting data fluctuation in a tangent plane mode, responding to the detected data fluctuation, acquiring current fluctuation time and a test component identifier, acquiring a corresponding component portrait based on the test component identifier, and further determining historical fluctuation time according to the component portrait.
The execution body generates corresponding binary data based on the organization rule of the class byte code file according to the data fluctuation detection specification by the dynamic proxy technology in the running period, and then loads and converts the generated binary data into corresponding public component classes. The slice of the common component class generated based on the dynamic proxy may then be invoked for data fluctuation detection in real-time to determine whether data fluctuation is detected.
In this embodiment, the execution subject (for example, may be a server) of the data processing method may determine whether or not the data fluctuation is detected by means of a wired connection or a wireless connection. If the data fluctuation is detected, the current time of the data fluctuation is obtained, namely the current fluctuation time and the test component identification. Specifically, the current fluctuation time may be a time point or a time period, and the embodiment of the present application does not specifically limit the current fluctuation time. The test component identifier may be, for example, the number of the component being tested or the name of the component, and the embodiment of the present application does not specifically limit the test component identifier. After the execution body acquires the test component identifier, the portrait database can be called to acquire the component portrait corresponding to the test component identifier, and specifically, the corresponding component type can be determined according to the component identifier, and the component portrait corresponding to the component type can be acquired from the portrait database. The component portraits are used for representing the time points and the fluctuation maintaining duration of each component corresponding to the component type in each historical test time period. Specifically, a label can be generated according to a time point and a fluctuation maintaining time length of each component corresponding to the component type in each time period of the history, specifically, the label can be a key value pair of the time point of generating the fluctuation and the corresponding fluctuation maintaining time length, and a corresponding component portrait is generated according to the generated label.
According to each tag corresponding to the obtained component image, the time point and the fluctuation maintaining time of each component history corresponding to the component type corresponding to the component identifier can be known, so that the history fluctuation time corresponding to the component image can be determined.
Step S102, determining a first boundary time point of the current fluctuation time and a second boundary time point of the historical fluctuation time, and generating a fluctuation time interval according to the first boundary time point and the second boundary time point.
By way of example, the current fluctuation time may be a time point or a time period, and the embodiment of the present application does not specifically limit the current fluctuation time. Specifically, the execution body may merge the current fluctuation time with the historical fluctuation time to obtain the fluctuation time interval. The execution subject may sort the first boundary time point and the second boundary time point after obtaining the first boundary time point and the second boundary time point, for example, t1, t2, and the result of the sorting may be, for example, t1< t3< t2< t4. The corresponding fluctuation time interval may be (t 1, t 4).
As another implementation mode of the embodiment of the application, the execution main body can perform the de-duplication processing after the current fluctuation time and the historical fluctuation time are combined, and only one repeated fluctuation time is reserved, so that the data is ensured to be clean, and the data processing speed and accuracy are improved.
Step S103, extracting performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval.
The performance index may include, for example, an operation speed of the test component, a number of bits of the binary number processed in the same time, a capacity of the internal memory, a capacity of the external memory, a speed of the I/O, a capacity of the video memory, a bandwidth of the video memory, etc., which is not particularly limited in the embodiment of the present application. The log information may include event records of the test component running in a preset time, and each row of log records date, time, user, action and other operations, and the content of the log information is not specifically limited in the embodiment of the present application. The process information may include a parent process identifier of the current process, a process group identifier of the current process, a lock identifier of the current process (the lock identifier is used to determine whether the current process is locked), etc., which is not specifically limited in the embodiment of the present application.
Step S104, determining and outputting the fluctuation type according to the performance index, the log information and the process information.
Specifically, determining the type of fluctuation includes: and calling a fluctuation type database to match the performance index, the log information and the process information with the fluctuation types in the fluctuation type database, and determining the fluctuation types obtained by matching as the fluctuation types corresponding to the performance index, the log information and the process information.
The fluctuation type database may store therein respective fluctuation types, and respective descriptions of performance indexes, log information, and process information corresponding to the respective fluctuation types. The performance index, the log information and the process information corresponding to the component identifier are matched with each performance index, the log information and the process information in the fluctuation type database, for example, the performance index, the log information and the process information corresponding to the component identifier are matched through a similarity calculation method, so that the fluctuation type matched with the performance index, the log information and the process information corresponding to the component identifier is determined.
Specifically, matching the performance index, the log information, and the process information with the fluctuation type in the fluctuation type database includes: mapping the performance index into a performance index vector, mapping the log information into a log information vector, and mapping the process information into a process information vector based on a word embedding method; mapping each fluctuation type in the fluctuation type database into each fluctuation type vector; calculating first similarity of the performance index vector and each fluctuation type vector, calculating second similarity of the log information vector and each fluctuation type vector, and calculating third similarity of the process information vector and each fluctuation type vector; and determining the fluctuation type matched with the performance index, the log information and the process information in the fluctuation type database based on the first similarity, the second similarity and the third similarity.
The fluctuation type vector may be a vector corresponding to performance index, log information and process information of each fluctuation type in the fluctuation type database, and the execution body may calculate a similarity between the vector and the performance index vector, the log information vector and the process information vector corresponding to the test component identifier, and determine, based on each calculated similarity, the performance index, the log information and the fluctuation type corresponding to each test component corresponding to the test component identifier.
Specifically, the first similarity may be a cosine similarity of the test component identification corresponding performance index vector with the performance index vector in each of the surge type vectors. The second similarity may be a cosine similarity of the test component identification corresponding log information vector to the log information vector in each of the wave type vectors. The third similarity may be a cosine similarity of the test component identification corresponding process information vector to the process information vector in each of the surge type vectors.
Specifically, determining, based on the first similarity, the second similarity, and the third similarity, a type of fluctuation in the fluctuation type database that matches the performance index, the log information, and the process information, includes: calculating the similarity sum of the first similarity, the second similarity and the third similarity corresponding to each fluctuation type in the fluctuation type database; and determining the fluctuation type matched with the performance index, the log information and the process information in the fluctuation type database according to the similarity sum.
Further, as another implementation manner of the embodiment of the present application, for the same wave type in the wave type database, the execution body may multiply the corresponding first similarity, second similarity and third similarity with corresponding preset weights (for example, a first weight k1 corresponding to the first similarity s1, a second weight k2 corresponding to the second similarity s2, and a third weight k3 corresponding to the third similarity s 3) and then add the multiplied values to obtain a similarity sum c=s1+s1+s2+s2+s3, compare the similarity sums corresponding to the wave types in the wave type database, and determine the maximum similarity sum as the performance index, the log information and the wave type corresponding to the process information of each test component corresponding to the test component identifier, thereby improving accuracy of determining the wave type corresponding to each test component corresponding to the test component identifier and improving accuracy of processing the wave data.
According to the embodiment, a public component class is packaged by calling a dynamic proxy technology, data fluctuation detection is conducted in a tangent plane mode, current fluctuation time and test component identification are obtained in response to the detected data fluctuation, corresponding component portraits are obtained based on the test component identification, and then historical fluctuation time is determined according to the component portraits; determining a first boundary time point of the current fluctuation time and a second boundary time point of the historical fluctuation time, and further generating a fluctuation time interval according to the first boundary time point and the second boundary time point; extracting performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval; and determining and outputting the fluctuation type according to the performance index, the log information and the process information. Therefore, the fluctuation data can be efficiently processed in the stability test, and the manual processing workload is reduced.
FIG. 2 is a schematic flow chart of a data processing method according to an embodiment of the present application, and as shown in FIG. 2, the data processing method includes:
step S201, calling a dynamic proxy technology, packaging a public component class, detecting data fluctuation in a tangent plane mode, responding to the detected data fluctuation, acquiring current fluctuation time and a test component identifier, acquiring a corresponding component portrait based on the test component identifier, and further determining historical fluctuation time according to the component portrait.
Specifically, determining a historical fluctuation time from the component representation includes: obtaining each testing stage label corresponding to the component image; summarizing the corresponding test stage labels when the data fluctuation occurs to obtain a label set; and determining the historical fluctuation time according to the label set.
The component image corresponds to each test stage label, and may include a time identifier of each test stage, an amplitude of test data deviating from a standard line, a starting time point of data fluctuation, a duration of data fluctuation, and the like. Summarizing the test phase labels of the component images, which correspond to the test phase labels, for generating the data fluctuation, so as to obtain a label set, acquiring a starting time point of the data fluctuation and a time length of the data fluctuation in the label set, and further determining the historical fluctuation time based on the acquired starting time point of the historical data fluctuation and the time length of the data fluctuation. The historical fluctuation time can be a time point or a time period, and the embodiment of the application does not specifically limit the time point or the time period of the historical fluctuation time.
Specifically, determining the historical fluctuation time according to the tag set includes: determining a timestamp corresponding to the tag set; from the time stamp, a historical fluctuation time is determined.
The execution body may obtain a time stamp corresponding to each tag in the tag set, where the time stamp is used to characterize a start time point of the data fluctuation corresponding to the tag and a duration of occurrence of the data fluctuation. According to the data fluctuation starting time and the data fluctuation time corresponding to the time stamp, the time of the historical data fluctuation can be determined, and the time can be a time point or a time period. By way of example, the timestamp may be of the form: a2020-01-17:15:00 2min, meaning that the A component has A2 minute data fluctuation at 15 PM 2020-01-17. By introducing a time stamp, the processing and analysis of the fluctuation data can be made more intuitive, fast and accurate.
Step S202, determining a first boundary time point of the current fluctuation time and a second boundary time point of the historical fluctuation time, and generating a fluctuation time interval according to the first boundary time point and the second boundary time point.
And performing repeated processing after taking intersection of the time point or time period corresponding to the current fluctuation time and the time point or time period corresponding to the historical fluctuation time so as to obtain a fluctuation time interval.
Step S203, the component address corresponding to the test component identification is obtained.
The component address corresponding to the test component identifier may be an address link, and the performance index, log information and process information of the corresponding component in the test process may be stored in the page corresponding to the address link.
Step S204, jumping to a component test database corresponding to the component address to extract performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval.
And jumping to an address link corresponding to the component address, so that the software test database at the address link extracts the performance index, log information and process information of each test component corresponding to the test component identifier based on the determined fluctuation time interval.
Specifically, performance indexes, log information and process information of each test component corresponding to the fluctuation time interval can be extracted based on time stamps of each test data in the software test database.
Step S205, according to the performance index, the log information and the process information, determining the fluctuation type and outputting.
The execution body may input performance metrics, log information, and process information into the pre-trained classification model to output a corresponding type of fluctuation. The pre-trained classification model is used for representing the corresponding relation between the input fluctuation data and the fluctuation type.
Fig. 3 is a schematic view of an application scenario of a data processing method according to an embodiment of the present application. The data processing method of the embodiment of the application can be applied to a scene of processing the generated fluctuation data in the stability test. Stability test: stability testing is a non-functional test method that aims to test the efficiency and ability of a software application to run continuously over a long period of time. Stability testing is the testing of the stability of a system near maximum load, ensuring that the system is able to handle high traffic and data loads. Monitoring the validity of the system under test prior to release and increasing team confidence in the software error-free development process. Ensuring that the system has no memory leaks, unprecedented shutdown, or abnormal behavior outside of the development environment. Stability is an important indicator of the measurement of software in product testing, particularly in nonfunctional testing. Is a reference for evaluating whether the software product can stably run for a long time under a certain pressure. Performance index: the performance indexes in the software test comprise CPU, memory, disk, network, IO, process and other service conditions. The execution main body of the embodiment of the application can comprise a service configuration module, a system time synchronization module, a fluctuation information extraction module and a fluctuation information display module. In the service configuration module, resource information of the test environment, a fault trigger threshold value and a time period for collecting information when fluctuation occurs are collected. In the system time synchronization module, since the monitoring information of the resource is collected according to the time when the fluctuation occurs, the system time is ensured to be consistent. And in the fluctuation information extraction module, a fluctuation time interval is generated according to the time of fluctuation occurrence and the time interval in the service configuration module, and monitoring information, system process information and log information of resources in the test environment are extracted. And in the fluctuation information display module, the information and the data extracted in the last step are displayed, so that research and development and problem analysis and positioning by testers are facilitated. Thereby avoiding the direct frequent switching and searching of the research personnel at each resource. As shown in fig. 3, in the service configuration module, a response time fluctuation range, a TPS fluctuation range, and a time interval are to be configured: for example 5 seconds, component address: such as server addresses, log paths, etc. Before the test is started, the execution main body calls a system time synchronization module to synchronize the system time of the test resource, so that inconsistency of time is avoided. When the monitoring index in the system exceeds a threshold (such as a response time fluctuation range or a TPS fluctuation range) configured by the "service configuration module", a fluctuation time interval may be generated according to the time when the current fluctuation occurs and a time interval (i.e., a historical fluctuation time, such as 5 seconds, which may be determined according to the condition that the historical test component fluctuates in data) in the "service configuration module". The execution body may call the fluctuation information extraction module to extract performance indexes, log information, and process information of the respective components according to the fluctuation time interval. After the information is successfully extracted, the information can be displayed on the fluctuation information display module, so that research, development and test personnel can conveniently analyze and position the problem, related information can be automatically stored, namely, an execution main body can call the fluctuation information display module to display and store the fluctuation related information, and subsequent searching and analysis are convenient. The embodiment of the application can accurately position the fluctuation time, automatically collect the related information during fluctuation, automatically store the related information, facilitate the subsequent searching and analysis, and facilitate research and development and research and analysis of testers.
Fig. 4 is a schematic diagram of main units of a data processing apparatus according to an embodiment of the present application. As shown in fig. 4, the data processing apparatus 400 includes an acquisition unit 401, a fluctuation time interval generation unit 402, an extraction unit 403, and a fluctuation type determination unit 404.
The obtaining unit 401 is configured to invoke a dynamic proxy technology, encapsulate a common component class, perform data fluctuation detection by means of a tangent plane, obtain a current fluctuation time and a test component identifier in response to detecting data fluctuation, obtain a corresponding component portrait based on the test component identifier, and further determine a historical fluctuation time according to the component portrait.
The fluctuation time interval generation unit 402 is configured to determine a first boundary time point of the current fluctuation time and a second boundary time point of the historical fluctuation time, and further generate a fluctuation time interval according to the first boundary time point and the second boundary time point.
The extracting unit 403 is configured to extract performance indexes, log information, and process information of the respective test components corresponding to the test component identifications based on the fluctuation time intervals.
The fluctuation type determination unit 404 is configured to determine a fluctuation type from the performance index, log information, and process information and output.
In some embodiments, the extraction unit 403 is further configured to: acquiring a component address corresponding to the test component identifier; and jumping to a component test database corresponding to the component address to extract performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval.
In some embodiments, the fluctuation type determination unit 404 is further configured to: and calling a fluctuation type database to match the performance index, the log information and the process information with the fluctuation types in the fluctuation type database, and determining the fluctuation types obtained by matching as the fluctuation types corresponding to the performance index, the log information and the process information.
In some embodiments, the fluctuation type determination unit 404 is further configured to: mapping the performance index into a performance index vector, mapping the log information into a log information vector, and mapping the process information into a process information vector based on a word embedding method; mapping each fluctuation type in the fluctuation type database into each fluctuation type vector; calculating first similarity of the performance index vector and each fluctuation type vector, calculating second similarity of the log information vector and each fluctuation type vector, and calculating third similarity of the process information vector and each fluctuation type vector; and determining the fluctuation type matched with the performance index, the log information and the process information in the fluctuation type database based on the first similarity, the second similarity and the third similarity.
In some embodiments, the fluctuation type determination unit 404 is further configured to: calculating the similarity sum of the first similarity, the second similarity and the third similarity corresponding to each fluctuation type in the fluctuation type database; and determining the fluctuation type matched with the performance index, the log information and the process information in the fluctuation type database according to the similarity sum.
In some embodiments, the acquisition unit 401 is further configured to: obtaining each testing stage label corresponding to the component image; summarizing the corresponding test stage labels when the data fluctuation occurs to obtain a label set; and determining the historical fluctuation time according to the label set.
In some embodiments, the acquisition unit 401 is further configured to: determining a timestamp corresponding to the tag set; from the time stamp, a historical fluctuation time is determined.
The data processing method and the data processing apparatus of the present application have a corresponding relationship in terms of implementation, and therefore, the description is not repeated.
Fig. 5 illustrates an exemplary system architecture 500 in which a data processing method or data processing apparatus of an embodiment of the present application may be applied.
As shown in fig. 5, the system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 is used as a medium to provide communication links between the terminal devices 501, 502, 503 and the server 505. The network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 505 via the network 504 using the terminal devices 501, 502, 503 to receive or send messages or the like. Various communication client applications may be installed on the terminal devices 501, 502, 503, such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).
The terminal devices 501, 502, 503 may be a variety of electronic devices having a data processing screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 505 may be a server providing various services, such as a background management server (by way of example only) that provides support for data fluctuations detected by users using the terminal devices 501, 502, 503. The background management server can call a dynamic proxy technology, package a public component class, detect data fluctuation in a tangent plane mode, acquire current fluctuation time and test component identification in response to the detected data fluctuation, acquire corresponding component portraits based on the test component identification, and further determine historical fluctuation time according to the component portraits; determining a first boundary time point of the current fluctuation time and a second boundary time point of the historical fluctuation time, and further generating a fluctuation time interval according to the first boundary time point and the second boundary time point; extracting performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval; and determining and outputting the fluctuation type according to the performance index, the log information and the process information. Therefore, the fluctuation data can be efficiently processed in the stability test, and the manual processing workload is reduced.
It should be noted that, the data processing method provided by the embodiment of the present application is generally executed by the server 505, and accordingly, the data processing apparatus is generally disposed in the server 505.
It should be understood that the number of terminal devices, networks and servers in fig. 5 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 6, there is illustrated a schematic diagram of a computer system 600 suitable for use in implementing an embodiment of the present application. The terminal device shown in fig. 6 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiment of the present application.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data required for the operation of the computer system 600 are also stored. The CPU601, ROM602, and RAM603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, mouse, etc.; an output portion 607 including a Cathode Ray Tube (CRT), a liquid crystal credit authorization query processor (LCD), and the like, and a speaker, and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The drive 610 is also connected to the I/O interface 605 as needed. Removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on drive 610 so that a computer program read therefrom is installed as needed into storage section 608.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication portion 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the present application are performed when the computer program is executed by a Central Processing Unit (CPU) 601.
The computer readable medium shown in the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present application may be implemented in software or in hardware. The described units may also be provided in a processor, for example, described as: a processor includes an acquisition unit, a fluctuation time interval generation unit, an extraction unit, and a fluctuation type determination unit. Wherein the names of the units do not constitute a limitation of the units themselves in some cases.
As another aspect, the present application also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs, which when executed by the device, cause the device to invoke dynamic proxy techniques, encapsulate a common component class, perform data fluctuation detection by means of a tangential plane, obtain a current fluctuation time and a test component identifier in response to the detection of the data fluctuation, obtain a corresponding component representation based on the test component identifier, and further determine a historical fluctuation time according to the component representation; determining a first boundary time point of the current fluctuation time and a second boundary time point of the historical fluctuation time, and further generating a fluctuation time interval according to the first boundary time point and the second boundary time point; extracting performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval; and determining and outputting the fluctuation type according to the performance index, the log information and the process information.
The computer program product of the application comprises a computer program which, when being executed by a processor, implements the data processing method in the embodiments of the application.
According to the technical scheme provided by the embodiment of the application, the data of the fluctuation generation can be efficiently processed in the stability test, and the manual processing workload is reduced.
The above embodiments do not limit the scope of the present application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of the present application.
Claims (16)
1. A method of data processing, comprising:
packaging a public component class through a dynamic proxy technology, detecting data fluctuation in a tangent plane mode, acquiring current fluctuation time and a test component identifier in response to the detected data fluctuation, acquiring a corresponding component portrait based on the test component identifier, and further determining historical fluctuation time according to the component portrait;
determining a first boundary time point of the current fluctuation time and a second boundary time point of the historical fluctuation time, and further generating a fluctuation time interval according to the first boundary time point and the second boundary time point;
Extracting performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval;
and determining and outputting the fluctuation type according to the performance index, the log information and the process information.
2. The method of claim 1, wherein the extracting performance metrics, log information, and process information for each test component corresponding to the test component identification based on the fluctuation time interval comprises:
acquiring a component address corresponding to the test component identifier;
and jumping to a component test database corresponding to the component address so as to extract performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval.
3. The method of claim 1, wherein the determining the type of fluctuation comprises:
and calling a fluctuation type database to match the performance index, the log information and the process information with the fluctuation types in the fluctuation type database, and determining the fluctuation types obtained by matching as the fluctuation types corresponding to the performance index, the log information and the process information.
4. The method of claim 1, wherein said matching said performance metrics, said log information, and said process information with a surge type in said surge type database comprises:
mapping the performance index into a performance index vector, mapping the log information into a log information vector, and mapping the process information into a process information vector based on a word embedding method;
mapping each fluctuation type in the fluctuation type database into each fluctuation type vector;
calculating first similarity of the performance index vector and each fluctuation type vector, calculating second similarity of the log information vector and each fluctuation type vector, and calculating third similarity of the process information vector and each fluctuation type vector;
and determining a fluctuation type matched with the performance index, the log information and the process information in the fluctuation type database based on the first similarity, the second similarity and the third similarity.
5. The method of claim 4, wherein the determining a type of fluctuation in the fluctuation type database that matches the performance indicator, the log information, and the process information based on the first similarity, the second similarity, and the third similarity comprises:
Calculating a similarity sum of the first similarity, the second similarity and the third similarity corresponding to each fluctuation type in the fluctuation type database;
and determining the fluctuation type matched with the performance index, the log information and the process information in the fluctuation type database according to the similarity sum.
6. The method of claim 1, wherein said determining a historical volatility time from said component representation comprises:
obtaining each test stage label corresponding to the component image;
summarizing the corresponding test stage labels when the data fluctuation occurs to obtain a label set;
and determining the historical fluctuation time according to the label set.
7. The method of claim 6, wherein said determining a historical volatility time from said set of tags comprises:
determining a timestamp corresponding to the tag set;
and determining the historical fluctuation time according to the time stamp.
8. A data processing apparatus, comprising:
the acquisition unit is configured to call a dynamic proxy technology, package a public component class, detect data fluctuation in a tangent plane mode, acquire current fluctuation time and a test component identifier in response to the detection of the data fluctuation, acquire a corresponding component portrait based on the test component identifier, and further determine historical fluctuation time according to the component portrait;
A fluctuation time interval generation unit configured to determine a first boundary time point of the current fluctuation time and a second boundary time point of the historical fluctuation time, and further generate a fluctuation time interval according to the first boundary time point and the second boundary time point;
an extracting unit configured to extract performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval;
and a fluctuation type determination unit configured to determine a fluctuation type from the performance index, the log information, and the process information and output.
9. The apparatus of claim 8, wherein the extraction unit is further configured to:
acquiring a component address corresponding to the test component identifier;
and jumping to a component test database corresponding to the component address so as to extract performance indexes, log information and process information of each test component corresponding to the test component identification based on the fluctuation time interval.
10. The apparatus of claim 8, wherein the fluctuation type determination unit is further configured to:
And calling a fluctuation type database to match the performance index, the log information and the process information with the fluctuation types in the fluctuation type database, and determining the fluctuation types obtained by matching as the fluctuation types corresponding to the performance index, the log information and the process information.
11. The apparatus of claim 8, wherein the fluctuation type determination unit is further configured to:
mapping the performance index into a performance index vector, mapping the log information into a log information vector, and mapping the process information into a process information vector based on a word embedding method;
mapping each fluctuation type in the fluctuation type database into each fluctuation type vector;
calculating first similarity of the performance index vector and each fluctuation type vector, calculating second similarity of the log information vector and each fluctuation type vector, and calculating third similarity of the process information vector and each fluctuation type vector;
and determining a fluctuation type matched with the performance index, the log information and the process information in the fluctuation type database based on the first similarity, the second similarity and the third similarity.
12. The apparatus of claim 11, wherein the fluctuation type determination unit is further configured to:
calculating a similarity sum of the first similarity, the second similarity and the third similarity corresponding to each fluctuation type in the fluctuation type database;
and determining the fluctuation type matched with the performance index, the log information and the process information in the fluctuation type database according to the similarity sum.
13. The apparatus of claim 8, wherein the acquisition unit is further configured to:
obtaining each test stage label corresponding to the component image;
summarizing the corresponding test stage labels when the data fluctuation occurs to obtain a label set;
and determining the historical fluctuation time according to the label set.
14. A data processing electronic device, comprising:
one or more processors;
storage means for storing one or more programs,
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-7.
15. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-7.
16. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310711959.5A CN116719710A (en) | 2023-06-15 | 2023-06-15 | Data processing method, device, electronic equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310711959.5A CN116719710A (en) | 2023-06-15 | 2023-06-15 | Data processing method, device, electronic equipment and computer readable medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116719710A true CN116719710A (en) | 2023-09-08 |
Family
ID=87871221
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310711959.5A Pending CN116719710A (en) | 2023-06-15 | 2023-06-15 | Data processing method, device, electronic equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116719710A (en) |
-
2023
- 2023-06-15 CN CN202310711959.5A patent/CN116719710A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113489713B (en) | Network attack detection method, device, equipment and storage medium | |
| CN105653947A (en) | Method and device for assessing application data security risk | |
| CN113641544B (en) | Method, apparatus, device, medium and product for detecting application state | |
| CN108255700A (en) | Test result generation method and device | |
| CN110929110A (en) | Electronic document detection method, device, equipment and storage medium | |
| CN114153703A (en) | Micro-service exception positioning method and device, electronic equipment and program product | |
| CN113869789A (en) | Risk monitoring method and device, computer equipment and storage medium | |
| CN111756745B (en) | Alarm method, alarm device, terminal equipment and computer readable storage medium | |
| CN115705190A (en) | Method and device for determining dependence degree | |
| CN116719710A (en) | Data processing method, device, electronic equipment and computer readable medium | |
| CN113904828B (en) | Method, apparatus, device, medium and program product for detecting sensitive information of interface | |
| CN112860538A (en) | Method and device for performing interface regression test based on online log | |
| CN115906135A (en) | Tracing method and device for target data leakage path, electronic equipment and storage medium | |
| CN113835987B (en) | Chaotic experiment execution file generation method and device, electronic equipment and storage medium | |
| CN117195204B (en) | Abnormal data detection method, device, electronic equipment and computer readable medium | |
| CN114765599A (en) | Sub-domain name acquisition method and device | |
| CN112988560A (en) | Method and device for testing system robustness | |
| CN114710354B (en) | Abnormal event detection method and device, storage medium and electronic equipment | |
| CN109246297A (en) | A kind of method, apparatus and storage medium of determining mobile terminal factory reset time | |
| CN115190008B (en) | Fault processing method, fault processing device, electronic equipment and storage medium | |
| CN109409038A (en) | A kind of dynamic link library file cracks risk checking method and device | |
| CN117707653B (en) | Parameter monitoring method, device, electronic equipment and computer readable storage medium | |
| CN111625853B (en) | Snapshot processing method, device and equipment and readable storage medium | |
| CN112416982B (en) | Method and device for calculating real-time user characteristics | |
| CN117493127B (en) | Application program detection method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |