CN116149959B

CN116149959B - Data processing device, method, monitoring equipment and computer program product

Info

Publication number: CN116149959B
Application number: CN202310430267.3A
Authority: CN
Inventors: 张锐
Original assignee: Beijing Jidu Technology Co Ltd
Current assignee: Beijing Jidu Technology Co Ltd
Priority date: 2023-04-20
Filing date: 2023-04-20
Publication date: 2023-07-18
Anticipated expiration: 2043-04-20
Also published as: CN116149959A

Abstract

The present disclosure relates to the field of data compression technology, and in particular provides a data processing apparatus, a method, a monitoring device, and a computer program product, where the data processing apparatus includes: a processor and a memory, wherein the processor executes the following processes: acquiring monitoring data of a target service interface and dividing the monitoring data into a plurality of data sets; the different data sets comprise call information for carrying out service call on the target service interface in different interface call subintervals; the interface calling subperiods respectively corresponding to the plurality of data sets are a plurality of continuous subperiods determined based on the same target interface calling period; screening out the call information of at least part of the data sets in the plurality of data sets so as to screen out at least part of normal call information and obtain a target data set after screening out the call information; and taking the data group which is not subjected to screening treatment and the target data group after the screening treatment as compressed data of the monitoring data. The method and the device can ensure the continuity of compressed data and improve the compression rate.

Description

Data processing device, method, monitoring equipment and computer program product

Technical Field

The present disclosure relates to the field of data compression technology, and in particular, to a data processing apparatus, a method, a monitoring device, and a computer program product.

Background

When the performance monitoring report of the service interface is displayed, the problem that the monitoring data is overlarge in scale is generally encountered, and if the whole detection data is loaded on the report, the problems of difficult data loading, report rendering failure, web page breakdown and the like are generated. In order to solve the problem of overlarge monitoring data quantity, a mode capable of accurately describing monitoring conditions and compressing monitoring data with a high compression ratio becomes a current urgent problem to be solved.

Disclosure of Invention

Embodiments of the present disclosure provide at least a data processing apparatus, method, monitoring device, and computer program product

In a first aspect, an embodiment of the present disclosure provides a data processing apparatus, including: a processor, a memory storing machine-readable instructions executable by the processor, the processor configured to execute the machine-readable instructions stored in the memory, the machine-readable instructions when executed by the processor, the processor performing the following data processing procedures:

Acquiring monitoring data of a target service interface, and dividing the monitoring data into a plurality of data sets; the different data sets comprise call information for carrying out service call on the target service interface in different interface call subperiods; the interface calling subperiods respectively corresponding to the plurality of data sets are a plurality of continuous subperiods determined based on the same target interface calling period;

screening out the call information of at least part of the data sets so as to screen out at least part of normal call information, and obtaining a target data set after screening out;

and taking the data group which is not subjected to screening treatment and the target data group after the screening treatment in the plurality of data groups as compressed data of the monitoring data.

In this way, after the monitoring data of the target service interface are obtained, the monitoring data are grouped, and at least part of normal calling information in at least part of data groups is screened out, so that a screened-out target data group is obtained, the data group which is not screened out and the screened-out target data group are used as compressed data of the monitoring data, the monitoring data are grouped through interface calling subintervals, and the data screening process is carried out on the basis of the data groups, so that the continuity of the compressed data can be ensured; meanwhile, the amount of screened call information in the screening process is flexibly controlled according to the data compression requirement, so that the obtained compressed data can meet the display requirement.

In a possible implementation manner, the data processing apparatus further includes: a display module;

the processor is further configured to: and controlling the display module to display a monitoring data report based on the compressed data.

Therefore, based on the compressed data, the control display module displays the monitoring data report, and the compressed data is obtained by screening redundant data from the monitoring data, so that the display content which is continuous in time and has higher response speed can be provided.

In a possible implementation manner, the processor is configured to, when acquiring monitoring data of a target service interface and dividing the monitoring data into a plurality of data sets:

responding to a received monitoring data query request, and reading the monitoring data from a target storage space storing original monitoring data based on the target interface calling period carried in the monitoring data query request; the monitoring data comprises a plurality of pieces of calling information corresponding to the target interface calling period;

dividing the target interface call period into a plurality of interface call sub-periods based on a target duration;

and dividing the monitoring data into a plurality of data groups based on the service calling time carried in each piece of calling information and the starting time and the ending time respectively corresponding to each interface calling subperiod.

In a possible embodiment, the processor is specifically configured to:

comparing the number of call information in each data group with a target number threshold value for each data group;

determining the data set as a target data set corresponding to the data set in response to the number of call information in the data set being less than or equal to the target number threshold;

and responding to the number of the call information in the data group to be larger than the target number threshold, and taking screening out at least part of normal call information in the data group as a target, and screening out the call information in the data group to obtain a target data group corresponding to the data group.

Therefore, for the data group with less calling information, screening processing is not needed, so that the compressed data can be normally displayed, and the processing efficiency of the data is also ensured.

In one possible implementation, the call information corresponding to each service call includes: the execution duration of the service call;

the processor is specifically configured to, when screening any one of the data sets:

for each of the data sets, performing data screening processing of at least one iteration cycle, and in each iteration cycle, performing:

Determining a target floating interval based on execution time lengths respectively included by a plurality of pieces of calling information in a data set to be processed corresponding to a current iteration period;

determining a reference time length based on execution time lengths respectively included by a plurality of pieces of calling information in the data set to be processed, and determining target calling information to be screened out from the calling information of the data set to be processed based on the target floating interval and the reference time length;

screening out the target call information from the data set to be processed to obtain an intermediate data set corresponding to the current iteration period;

determining whether calling information in an intermediate data set corresponding to the current iteration period reaches a target compression condition;

in response to the target compression condition not being met, determining the intermediate data set corresponding to the current iteration period as a data set to be processed corresponding to a next iteration period, and entering the next iteration period;

and determining an intermediate data set corresponding to the current iteration cycle as the target data set in response to the target compression condition being reached.

In this way, the redundant data can be screened out by the data screening of at least one iteration period, so that the compressed data meeting the display requirement can be obtained.

In a possible implementation manner, the processor is configured to, when determining the target floating interval based on execution durations that are respectively included by multiple pieces of call information in the to-be-processed data set corresponding to the current iteration period:

sequencing a plurality of pieces of call information in the data set to be processed based on execution time length respectively included in each piece of call information in the data set to be processed, so as to obtain a sequencing queue of the plurality of pieces of call information in the data set to be processed;

determining a first sorting index and a second sorting index based on a target screening threshold and the number of calling information in the data set to be processed, and determining first calling information and second calling information from the sorting queue based on the first sorting index and the second sorting index;

and determining the target floating interval based on the execution duration included in the first call information and the execution duration included in the second call information.

In a possible embodiment, the processor is further configured to: determining a time factor corresponding to the data set to be processed based on the starting time and/or the ending time corresponding to the data set to be processed;

A target screening threshold is determined for the data set to be processed based on a time factor corresponding to the data set to be processed and an original screening threshold.

Thus, the time factor is utilized to control the data group which is longer than the current moment to correspond to a higher target screening threshold value; and the data group which is closer to the current moment corresponds to a lower target screening threshold value so as to preserve the calling information with larger value in the compressed data as much as possible.

In a possible implementation manner, the processor is configured to, when determining a reference duration based on execution durations respectively included in a plurality of pieces of call information in the data set to be processed, and determining target call information to be screened out from call information in the data set to be processed based on the target floating interval and the reference duration, determine the target call information to be screened out:

determining the execution time length of the call information corresponding to the first sequencing index of the sequencing queue as the reference time length;

determining current call information which is not traversed from the sequencing queue based on the front-back sequence of each piece of call information in the sequencing queue; the current calling information comprises calling information corresponding to other sequencing indexes except the first sequencing index and the last sequencing index;

Comparing the absolute value of the difference value between the execution time length and the reference time length contained in the current calling information with the target floating interval;

determining the current calling information as the target calling information in response to the difference value corresponding to the current calling information being smaller than or equal to the target floating interval, and returning to the step of determining the current calling information which is not traversed from the sequencing queue;

responding to the difference value corresponding to the current calling information being larger than the target floating interval, and intercepting an intermediate sub-queue from the sequencing queue based on the position of the current calling information in the sequencing queue; turning the intercepted intermediate sub-queue to obtain a turning queue, returning the turning queue as a new sequencing queue to the execution time length of calling information corresponding to the first sequencing index at the first end of the sequencing queue, and determining the execution time length as the reference time length;

and traversing call information corresponding to the maximum execution time length and the minimum execution time length respectively in the data set to be processed to obtain the target call information.

In this way, among the monitoring data, monitoring data belonging to abnormal data can be retained as much as possible.

In a second aspect, embodiments of the present disclosure provide a monitoring device comprising a data processing apparatus as described in the first aspect, or any one of the first aspects.

In a third aspect, an embodiment of the present disclosure further provides a data processing method, including:

In a fourth aspect, the presently disclosed embodiments also provide a computer program product comprising a computer program which, when executed, implements the data processing method according to the third aspect.

The foregoing objects, features and advantages of the disclosure will be more readily apparent from the following detailed description of the preferred embodiments taken in conjunction with the accompanying drawings.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for the embodiments are briefly described below, which are incorporated in and constitute a part of the specification, these drawings showing embodiments consistent with the present disclosure and together with the description serve to illustrate the technical solutions of the present disclosure. It is to be understood that the following drawings illustrate only certain embodiments of the present disclosure and are therefore not to be considered limiting of its scope, for the person of ordinary skill in the art may admit to other equally relevant drawings without inventive effort.

FIG. 1 illustrates a flow chart of a data processing method provided by some embodiments of the present disclosure;

FIG. 2 illustrates an example of a monitoring data report provided by some embodiments of the present disclosure;

FIG. 3 illustrates a flow chart of a particular method of determining a target data set provided by some embodiments of the present disclosure;

FIG. 4 illustrates a flow chart of a particular method of determining targeted call information provided by some embodiments of the present disclosure;

FIG. 5 illustrates a specific example of determining target call information provided by some embodiments of the present disclosure;

fig. 6 illustrates a schematic diagram of a data processing apparatus provided by some embodiments of the present disclosure.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are only some embodiments of the present disclosure, but not all embodiments. The components of the disclosed embodiments generally described and illustrated herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the disclosure, as claimed, but is merely representative of selected embodiments of the disclosure. All other embodiments, which can be made by those skilled in the art based on the embodiments of this disclosure without making any inventive effort, are intended to be within the scope of this disclosure.

It has been found that the service interface (service interface) is a shared boundary between one automation system and another automation system or terminal device, and that one system a or terminal device B can invoke a certain service provided by the system C through the service interface provided by the other system C. In order to ensure the service quality of the service interface, maintain the normal operation of the service interface, the called condition of the service interface needs to be monitored, abnormal data can be recorded through monitoring, and the basis for positioning and tracking problems is provided. Assuming that the average call frequency of a certain service interface is 30 ten thousand/second, the data volume of the monitoring data generated in one hour can reach about 10 hundred million lines, and if the monitoring data is to be displayed completely through a report, a series of problems such as difficult data loading, report rendering failure, web page failure and the like can be generated. This requires that the monitor data be compressed at a high compression ratio before the monitor data is presented in a report form, so that the monitor data can be loaded smoothly.

Currently, data compression is generally performed by using an interval sampling method. However, since the monitoring data generally contains some abnormal data; the detection sampling method is likely to delete abnormal monitoring data in the original monitoring data when the original monitoring data is sampled, so that the finally displayed performance monitoring report cannot truly reflect the performance abnormal condition of the service interface. Meanwhile, the interval sampling method is difficult to ensure the balance between the data continuity and the compression rate, and if the sampling interval is too large, the continuity of the monitoring data cannot be ensured; if the sampling interval is too small, the compression rate of the data cannot meet the display requirement.

Based on the above-mentioned research, the present disclosure provides a data processing apparatus and a data processing method, after obtaining monitoring data of a target service interface, grouping the monitoring data, and screening at least part of normal call information for call information in at least part of data groups to obtain a screened target data group, and using the data group which is not screened and the screened target data group as compressed data of the monitoring data, where the compressed method groups the monitoring data by calling a subinterval through the interface, and performs data screening on the basis of the data groups, so as to ensure continuity of the compressed data; meanwhile, the amount of screened call information in the screening process is flexibly controlled according to the data compression requirement, so that the obtained compressed data can meet the display requirement.

The present invention is directed to a method for manufacturing a semiconductor device, and a semiconductor device manufactured by the method.

It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.

For the sake of understanding the present embodiment, first, a detailed description will be given of a data processing method disclosed in an embodiment of the present disclosure, where an execution body of the data processing method provided in the embodiment of the present disclosure is generally a computer device having a certain computing capability, where the computer device includes, for example: the terminal device, or server or other processing device, may be a User Equipment (UE), mobile device, user terminal, cellular telephone, cordless telephone, personal digital assistant (Personal Digital Assistant, PDA), handheld device, computing device, vehicle mounted device, wearable device, etc. In some possible implementations, the data processing method may be implemented by way of a processor invoking computer readable instructions stored in a memory.

Referring to fig. 1, a data processing method provided in an embodiment of the present disclosure includes:

s101: acquiring monitoring data of a target service interface, and dividing the monitoring data into a plurality of data sets; the different data sets comprise call information for carrying out service call on the target service interface in different interface call subperiods; the interface calling subperiods respectively corresponding to the plurality of data sets are a plurality of continuous subperiods determined based on the same target interface calling period;

s102: screening out the call information of at least part of the data sets so as to screen out at least part of normal call information, and obtaining a target data set after screening out;

s103: and taking the data group which is not subjected to screening treatment and the target data group after the screening treatment in the plurality of data groups as compressed data of the monitoring data.

The following describes the steps S101 to S103 in detail.

For S101 described above: the target service interface may be, for example, an interface provided by any service system for calling a service provided by the service providing system; when monitoring the target service interface, the monitoring data can be obtained by, for example, realizing the monitoring of the flow or the embedding of the interface. The monitoring data comprises a plurality of pieces of calling information for calling the target service interface by the service calling party. Each piece of call information includes, for example: the user identity of the service requester, the execution duration of the service call, and the call time of the service call.

Here, the service caller includes, for example: other service systems, or terminal devices.

Flow monitoring includes, for example, obtaining corresponding monitoring data by monitoring flow through a target service interface; specifically, when a service calling party calls a service of a target service interface, a service calling request is sent to the target service interface; after receiving the service call request through the target service interface, the service system for providing the service executes the service logic corresponding to the provided service based on the related information carried in the service call request, generates a corresponding response result, and returns the response result to the service calling party. In the process, by monitoring the flow of the target service interface, information such as the user identity of the service requester sending the service call request, the time when the target service interface receives the service call request, the time when the target service interface sends a response result to the service requester and the like can be obtained, so that original monitoring data can be obtained based on the monitored information.

The embedded point of the interface is, for example, to implant a code for collecting monitoring data in the service call interface; and when the service calling party calls the service provided by the target service interface, the corresponding monitoring data are acquired by utilizing the code.

After the raw monitoring data is obtained as described above, the raw monitoring data may be stored in a target storage space, such as a data warehouse MongoDB, HBase. When a user needs to view the monitoring data of the target interface call period in a report form, the monitoring data of the target interface call period to be viewed can be obtained from the target storage space, and the data processing method or the data processing device provided by the embodiment of the disclosure is utilized to process the monitoring data of the target interface call period, so as to obtain compressed data.

Here, the target interface call period may be preset, for example, each time monitoring data within 1 hour from the current time is viewed; it may also be an input specified period, for example, when the current time is 3 months 15 days 20, it may be specified to view the monitoring data of a certain period before the current time, for example, the monitoring data from 3 months 15 days 12 to 3 months 14.

In another embodiment of the present disclosure, after the compressed data is obtained, a monitoring data report may also be generated based on the compressed data, and the monitoring data report may be displayed.

In the example shown in fig. 2, a specific example of a displayed monitoring data report is provided, in which the horizontal axis represents the call time, and the vertical axis represents the execution duration corresponding to the service call in milliseconds.

It can be seen that in the monitoring report shown in the figure, the trend line formed by the execution time length of each piece of call information in the compressed data can completely reflect the change condition of the execution time length of different service calls of the target service interface in a certain time period.

In a possible implementation manner, the embodiment of the present disclosure further provides a specific method for acquiring monitoring data of a target service interface and dividing the monitoring data into a plurality of data sets, including:

responding to a received monitoring data query request, and reading the monitoring data from a target storage space storing original monitoring data based on the target interface calling period carried in the monitoring data query request; the monitoring data comprises a plurality of pieces of calling information corresponding to the target interface calling period; dividing the target interface call period into a plurality of interface call sub-periods based on a target duration; and dividing the monitoring data into a plurality of data groups based on the service calling time carried in each piece of calling information and the starting time and the ending time respectively corresponding to each interface calling subperiod.

In implementations, the target duration may be determined, for example, according to actual needs, e.g., may be 100 milliseconds, 200 milliseconds, etc. of the facility; for example, the target duration may be inversely related to the query rate per second of the target service interface; that is, if the query rate per second of the target service interface is larger, more call information is generated within a fixed duration, and at this time, a smaller target duration may be set to control the amount of call information in any data set, so that when screening out the call information based on any data set, the temporal consistency of different data sets can be ensured as much as possible; otherwise, if the query rate per second of the target service interface is smaller, the generated call information is characterized to be smaller in a fixed duration, and at this time, a larger target duration can be set to ensure that the number of call information in any data set is enough, so that more redundant data can be screened out when redundant data (normal call information) can be screened out by using the data processing method provided by the embodiment of the disclosure, so that the data compression rate is ensured.

The monitoring data query request is, for example, input by a user via an input device connected to the data processing apparatus; in the query request, for example, time information of the query may be carried, where the time information of the query includes, for example: the start time and the end time of the target interface call period. After receiving a data query request input by a user through an input device, the data processing device can be used for receiving corresponding monitoring data from a target storage space based on the starting time and the ending time of a target interface calling period and the calling time included in each piece of calling information in original monitoring data.

Furthermore, the time information of the query includes, for example: the duration of the target interface call period, for example, taking the current time of the query as the termination time; thus, the starting time of a query can be determined by the duration of the query and the current time of the query, and then the corresponding monitoring data in the target storage space is selected based on the current time, the determined starting time of the query and the calling time included in each piece of calling information in the original monitoring data.

In addition, the starting time and the ending time corresponding to the interface calling sub-periods respectively can be determined based on the target interface calling period and the target duration corresponding to each interface calling period, and each interface calling sub-period corresponds to one data set; and dividing the monitoring data read from the target space into a plurality of data groups according to the starting time and the ending time which correspond to each interface calling sub-period respectively and the service calling time carried in each piece of calling information in the monitoring data.

Here, the number of call information included in different data sets is also different according to different call cases of the service caller to the target service interface.

In another embodiment of the present disclosure, the acquired monitoring data may further include: compressed data of a first target interface call period, and original monitoring data of a second target interface call period; wherein the first target interface invocation period and the second target interface invocation period are consecutive time periods.

For compressed data corresponding to the first target interface call period, for example, compressed data obtained after the original monitoring data corresponding to the first target interface call period is processed by adopting the data processing method provided by the embodiment of the disclosure at a certain moment of history.

For example, the time for the previous request to view the monitored data is: 10:00, and queried for 9:00-10:00 monitoring data. 9:00-10:00 are first target interface calling time periods in the process of requesting to view the monitoring data; in the data compression process, the original monitoring data A1 corresponding to 9:00-10:00 are processed to obtain compressed data M1;

the time for requesting to check the monitoring data is 12:00 of the same day; and inquiring monitoring data of 9:00-12:00 on the same day, wherein 9:00-10:00 is a first time period and 10:00-12:00 is a second target interface calling period in the process of requesting to view the monitoring data, and the acquired monitoring data comprises: compressed data M1 corresponding to the first target interface call period, and original monitoring data A2 corresponding to the second target interface call period. During processing, the monitoring data formed by the compressed data M1 and the original monitoring data A2 are processed to obtain the corresponding compressed data M2 in the current checking process.

In this way, not only can the amount of data that needs to be processed during each data processing be reduced, the pressure of data processing is reduced, but also the value of the monitored data is continuously reduced over time, and the expected scale of the data can be gradually reduced over time by the processing so as to further clear redundant data.

For S102 described above: after obtaining the plurality of data sets based on S101, screening processing may be performed on call information corresponding to at least some of the data sets, so as to obtain a target data set corresponding to the at least some data sets.

Wherein normal call information and abnormal call information are generally included in the monitoring data. The abnormal call information includes call information that fails to hit the business logic and/or call information that has long execution time, for example.

Illustratively, when a service requester calls a target service interface, the service requester needs to send a service call request to the target service interface; if the service call request does not carry relevant information required by executing the service logic, such as a user identity of a service requester, parameters required to be transmitted when executing the service logic, or a case of an interface call error, the service logic is not hit. For this case, the target service interface may feed back response information to the service requester, which may indicate that the service call failed. In this case, the corresponding execution duration of the service call is typically less than 1 millisecond. In addition, in the case where the target service interface or the service system is blocked, the execution time required for the service system to execute the service call may be prolonged, for example, up to 8 ms or more. And normal service call, the corresponding execution duration is usually 3-4 milliseconds.

Meanwhile, the proportion occupied by call information corresponding to normal service call in monitoring data can reach more than 99% under normal conditions. The value of the normal monitoring data is relatively small in the monitoring process of the target service interface, so that the value can be regarded as redundant data, and the data processing device provided by the embodiment of the disclosure can screen out at least part of normal calling information in the data set as a target and screen out the calling information in at least part of the data set.

When screening call information in any data set, the call information included in different data sets is different, and in some data sets, all the call information included may be normal call information; in order to ensure the temporal continuity of the resulting compressed data, only part of the call information in the data set needs to be screened out at this time. In some data sets, the call information includes few abnormal call information and most normal call information, and at this time, part of the normal call information in the data set may be screened out, and all of the normal call information in the data set may be screened out. In some data sets, the call information may include more abnormal call information, and under the requirement of a larger data compression rate, all normal call information in the data set may be screened out, and only the abnormal call information in the data sets is reserved.

Since the normal call information has no more monitoring value than the reference of the abnormal call information, the normal call information can be screened out as redundant data. Therefore, when screening out the call information in the data set, the embodiment of the disclosure achieves the purpose of data compression by screening out at least part of normal call information in the data set. Meanwhile, in order to ensure the continuity of the whole time sequence, partial normal call information needs to be reserved.

In addition, for any data set, if the number of call information in the data set is small, no matter whether the data screening process is performed on the data set, the influence on the compression rate of the compressed data is relatively small; in another embodiment of the present disclosure, when performing data screening processing on a data set, the method specifically includes:

The target number threshold may be set according to actual needs. For example, 9 bars may be provided. In this way, for the case that the number of call information in a certain data group a is less than 9, the data screening process is not performed on the data group a any more, but all call information in the data group a is directly used as call information in the corresponding target data group so as to form the target data group.

Aiming at the situation that the number of the call information in a certain data group B is greater than or equal to 9, at least part of the normal call information in the data group B is screened out, and data screening processing is carried out on the call information in the data group B to obtain a target data group corresponding to the data group B.

Referring to fig. 3, an embodiment of the present disclosure further provides a specific manner of screening call information in a data set, including:

for a data group to be subjected to the screening processing, the data screening processing of at least one iteration cycle is performed, and in each iteration cycle, the data screening processing is performed:

s301: and determining a target floating interval based on the execution duration respectively included by the pieces of calling information in the data set to be processed corresponding to the current iteration period.

Wherein, for the case that the current iteration period is the first iteration period, the data set to be processed includes: the data set; for the case that the current iteration period is not the first iteration period, the data set to be processed includes: and the intermediate data set is determined in the data screening process of the previous iteration period of the current iteration period.

In a specific implementation, for a first iteration period, the data set to be processed corresponding to the first iteration period is the data set into which the monitoring data is divided in S101. For the intermediate data set determined for the data screening processing procedure of the i-1 th iteration period for the data set to be processed corresponding to the i-th iteration period except the first iteration period, the specific determination procedure can be referred to the following steps S303 to S305, which are not described herein.

In addition, in an embodiment of the present disclosure, a specific manner of determining a target floating interval is provided, including:

determining the target floating interval based on the execution duration included in the first call information and the execution duration included in the second call information;

in a specific implementation, since the execution duration included in the normal call information is generally in the middle area of the execution duration included in the abnormal call information, and the difference between the execution duration included in the normal call information and the execution duration included in the abnormal call information is generally obvious, after the multiple call information are sorted to form the sorting queue, if the abnormal call information exists in the data set, the abnormal call information may be located at the first end or the second end of the sorting queue, and further, when the first sorting index and the second sorting index are determined according to the target screening threshold, for example, a part of call information may be respectively reserved at the first end and the second end of the sorting queue as call information that may be reserved, and based on the target screening threshold, the first sorting index and the second sorting index are respectively determined from a side near the first end and a side near the second end of the sorting queue, and the first call information and the second call information are respectively determined from the sorting queue according to the first sorting index and the second sorting index.

After the first call information and the second call information are determined, determining the absolute value of the difference between the execution duration corresponding to the first call information and the execution duration corresponding to the second call information as a target floating interval.

The target screening threshold may be predetermined, for example, expressed in percentage form. For example, 60%, 80%, etc., may be specifically set according to actual needs.

In one possible implementation, the target screening thresholds for different data sets may be the same.

In another possible implementation, as the value of the monitored data continuously decreases over time, in order to be able to further reduce the redundant data of low value, it is required that the data set longer from the current moment corresponds to a higher target screening threshold; and the data group which is closer to the current moment corresponds to a lower target screening threshold value so as to preserve the calling information with larger value in the compressed data as much as possible.

Further, in order to achieve the above object, a time factor corresponding to a data group may be determined based on a start time and/or an end time corresponding to the data group;

a target screening threshold is determined for the data set based on a time factor corresponding to the data set and an original screening threshold.

Wherein, when determining the time factor corresponding to the data group based on the start time and/or the end time corresponding to the data group, for example, the time difference between the start time or the end time corresponding to the data group and the current time may be determined; then, in a plurality of time difference intervals, determining a target time difference interval for the time difference; the time difference intervals are respectively corresponding to different time factors; and then determining the time factor corresponding to the target time difference interval as the time factor corresponding to the data set.

Then, a target screening threshold is determined for the data set based on the time factor corresponding to the data set and the original screening threshold.

The larger the time difference corresponding to a certain data set, the larger the target screening threshold corresponding to the data set is finally determined.

Exemplary: for a data set R, assuming that the data set includes 100 pieces of call information, namely a 1-a 100, and after the a 1-a 100 is sequenced based on execution time length respectively included in the a 1-a 100, sequencing indexes respectively corresponding to the call information in a sequencing queue are as follows: b1 to b100; wherein, each piece of calling information ai corresponds to a sequencing index bj in the sequencing queue. The sequencing queue can be sequenced according to the sequence from the big to the small of the execution time length, or sequenced according to the sequence from the small to the big of the execution time length; this example employs sorting in order of execution time length from small to large.

In this example, taking 60% as an example, 20% of call information is reserved at the first end and the second end of the sorting queue, and the determined first sorting index includes, for example: b21; the determined second ordering index includes, for example: b80.

then, according to the mapping relation between the sequencing index and the calling information, determining first calling information am and second calling information an from the data set, subtracting the execution duration tm corresponding to the first calling information am from the execution duration tn corresponding to the second calling information an, and determining the absolute value of the obtained difference value as a target floating interval T.

Other ways of determining the target float interval are possible, such as statistics where, for example, intervals of multiple execution durations may be determined; and counting the distribution condition of the execution time lengths corresponding to the pieces of calling information in the data set to be processed in the intervals of the execution time lengths, determining the main distribution interval of the execution time lengths corresponding to the normal calling information to be screened out according to the distribution condition and the target to be screened out, and then determining the target floating interval according to the distribution interval.

In addition, other manners of determining the target floating interval may be adopted, and the specific determination manner is not limited in the embodiments of the present disclosure.

S302: determining a reference time length based on execution time lengths respectively included by a plurality of pieces of calling information in the data set to be processed, and determining target calling information to be screened out from the calling information of the data set to be processed based on the target floating interval and the reference time length.

In a specific implementation, after the target floating interval is determined, for example, any one of the following modes a, B may be adopted to determine target call information to be screened out from multiple call information in the data set to be processed:

a: determining a maximum ordering index and a minimum ordering index from an ordering queue, determining the execution time length of calling information corresponding to the maximum ordering index as a first reference time length, and determining the execution time length of calling information corresponding to the minimum ordering index as a second reference time length.

And calculating the execution time length of calling information corresponding to other sequencing indexes except the maximum sequencing index and the minimum sequencing index, and respectively obtaining difference values with the first reference time length and the second reference time length.

And determining call information corresponding to any other sequencing index as target call information to be screened if the absolute value of the difference value between the corresponding execution time length and the first reference time length is smaller than the target floating interval or the absolute value of the difference value between the corresponding execution time length and the second reference time length is smaller than the target floating interval.

Illustratively, the hypothetical ordering queue includes: b ₁ ~b ₁₀₀ And sequencing the scheduling information according to the order of the execution time from small to small. Target floating interval is T, b ₁₀₀ The execution time length of the corresponding call information is determined as a first reference time length t _max And b ₁ The execution duration of the corresponding call information is determined as a second execution duration t _min Then for b ₂ ~b ₉₉ Respectively calculate b _i Execution duration t of corresponding call information _i And t _max 、t _min If the difference between the two values is satisfied:or alternatively, the process may be performed,determining call information corresponding to biInformation is invoked for the target.

B: as shown in fig. 4, the following procedure may be used to determine the target call information to be screened out, for example:

s401: and determining the execution time length of the calling information corresponding to the first sequencing index of the sequencing queue as the reference time length.

S402: determining current call information which is not traversed from the sequencing queue based on the front-back sequence of each piece of call information in the sequencing queue; the current calling information comprises calling information corresponding to other sequencing indexes except the first sequencing index and the last sequencing index;

s403: and comparing the absolute value of the difference value between the execution time length and the reference time length included in the current call information with the target floating interval.

S404: and in response to the absolute value of the difference value corresponding to the current call information being smaller than or equal to the target floating interval, determining the current call information as the target call information, returning to the step S402, determining the call information which is not traversed next as new current call information, and determining whether the new call information is the target call information.

S405: responding to the absolute value of the difference value corresponding to the current calling information being larger than the target floating interval, and intercepting an intermediate sub-queue from the sequencing queue based on the position of the current calling information in the sequencing queue;

s406: and turning the intercepted intermediate sub-queues to obtain turning queues, and returning the turning queues as new sequencing queues to the step S401.

And traversing the call information corresponding to the maximum execution time length and the minimum execution time length respectively in the data set to be processed to obtain target call information.

Exemplary: the process of the data screening process is described in detail with the following example shown in fig. 5:

for data set R, it is assumed that the data set includes 100 pieces of call information, respectively a ₁ ~a ₁₀₀ Based on a ₁ ~a ₁₀₀ Respectively comprising execution time length corresponding to a ₁ ~a ₁₀₀ After sorting, sorting indexes corresponding to each piece of calling information in the obtained sorting queue are as follows: b ₁ ~b ₁₀₀ The method comprises the steps of carrying out a first treatment on the surface of the Each piece of calling information corresponds to a sequencing index in the sequencing queue. The sequencing queue can be sequenced according to the sequence from the big to the small of the execution time length, or sequenced according to the sequence from the small to the big of the execution time length; this example employs sorting in order of execution time length from small to large.

(1): in this example, taking 60% as an example, 20% of call information is reserved at the first end and the second end of the sorting queue, and the determined first sorting index includes, for example: b ₂₁ The method comprises the steps of carrying out a first treatment on the surface of the The determined second ordering index includes, for example: b ₈₀ 。

(2): determining the index and the call information from the data group according to the mapping relation between the ordering index and the call information ₂₁ Corresponding first call information, and determining sum b ₈₀ Corresponding second call information, and then, executing time t corresponding to the first call information ₂₁ Execution time t corresponding to the second call information ₈₀ And subtracting, and determining the absolute value of the obtained difference value as a target floating interval T.

(3): the determined reference time length is the execution time length t in the call information corresponding to the sequencing index b1 ₁ . Then according to the slave rank index b ₂ ~b ₉₉ Traversing the call information.

For traversed-to sort index b _i If the execution time t of the corresponding call information _i The method meets the following conditions:then sort index b _i The corresponding call information is determined to be the target call information to be screened out.

If go through to b ₉₉ The corresponding execution time length of the call information satisfies the above conditions, and b is reserved in the data set ₁ And b ₁₀₀ And respectively corresponding call information serving as a target data set.

If for the ordering index b _i If the execution time t of the corresponding call information _i The method meets the following conditions:the traversal is stopped.

(4): in order index b _i To intercept position, from b ₁ ~b ₁₀₀ Middle intercepting middle sub-queue b _i ~b ₁₀₀ . Will b _i ~b ₁₀₀ Turning to obtain a turning queue b of the intermediate sub-queue ₁₀₀ ~b _i 。

At this time, the queue b will be flipped ₁₀₀ ~b _i As a new ordering queue, let b ₁₀₀ Execution duration t of corresponding call information ₁₀₀ As the reference time length, b ₉₉ Traversing a new ordering queue b for starting point ₁₀₀ ~b _i Is provided.

For the rank index b _j If the execution time t of the corresponding call information _j The method meets the following conditions: The AND ordering index b _j The corresponding call information is defined as target call information to be screened out.

If go through to b _i+1 The corresponding execution time length of the call information satisfies the above conditions, and b is reserved in the data set ₁₀₀ And b _i And calling information corresponding to the call information is used as data in the target data set, and the target data set comprises: b ₁ 、b _i And b ₁₀₀ And calling information corresponding to the calling information respectively.

If for the ordering index b _j If the execution time t of the corresponding call information _j The method meets the following conditions:the traversal is stopped.

(5): in order index b _j For the position of interception, from b ₁₀₀ ~b _i Middle intercepting middle sub-queue b _j ~b _i 。

Intermediate sub-queue b _j ~b _i Then overturn is carried out to obtain an overturned queue b after overturn _i ~b _j . Wherein i < j.

Then turn over the queue b _i ~b _j As a new ordering queue, let b _i Execution duration t of corresponding call information _i As the reference time length, b _i+1 Traversing the new rollover queue b as starting point _i ~b _j Is provided.

For the rank index b _k If the execution time t of the corresponding call information _k The method meets the following conditions:the AND ordering index b _k The corresponding call information is defined as target call information to be screened out.

If go through to b _j-1 The corresponding execution time length of the call information satisfies the above conditions, and b is reserved in the data set _i And b _j And calling information corresponding to the call information is used as data in the target data set, and the target data set comprises: b ₁ 、b _i 、b ₁₀₀ And b _j And calling information corresponding to the calling information respectively.

If for the ordering index b _k If the execution time t of the corresponding call information _k The method meets the following conditions:the traversal is stopped.

(6): in order index b _k For the position of interception, from b _i ~b _j Middle intercepting middle sub-queue b _k ~b _j ，

Intermediate sub-queue b _k ~b _j Then overturn is carried out to obtain an overturned queue b after overturn _j ~ b _k . Where j > k.

Then turn over the queue b _j ~ b _k As a new ordering queue, then b _j Execution duration t of corresponding call information _j The above-described comparison process is performed again for the reference time period.

Repeating the above process until b ₁ ~b ₁₀₀ B in the corresponding 100 pieces of call information ₂ ~b ₉₉ And traversing the call information corresponding to each to obtain the target call information.

S303: screening out the target call information from the data set to be processed to obtain an intermediate data set corresponding to the current iteration period;

s304: determining whether calling information in an intermediate data set corresponding to the current iteration period reaches a target compression condition; if not, jumping to S305; if yes, jump to S306;

s305: determining the intermediate data set corresponding to the current iteration period as a data set to be processed corresponding to the next iteration period, and entering the next iteration period;

S306: and determining the intermediate data set corresponding to the current iteration period as the target data set.

Specifically, here, the target call information determined in S302 is deleted from the data set, and only the non-target call information is retained, and at this time, the retained non-target call information constitutes an intermediate data set corresponding to the current iteration cycle.

Then, determining whether the target screening scale is reached according to the number of original call information in the data set and the number of non-target call information finally reserved; and if the target screening scale is reached, taking the finally reserved non-target calling information as a target data set.

If the target screening scale is not reached, a new data set is formed based on the reserved non-target calling information, and an iteration cycle is executed again for the new data set, namely, the screening process of the calling information is executed again until the target screening scale is reached, and the non-target calling information obtained by calling the information screening process in the last iteration cycle is determined to be the corresponding target data set.

After the target data group corresponding to each of the plurality of data groups is obtained, the plurality of target data groups constitute compressed data of the monitor data.

The monitoring data report may then be displayed based on the compressed data.

It will be appreciated by those skilled in the art that in the above-described method of the specific embodiments, the written order of steps is not meant to imply a strict order of execution but rather should be construed according to the function and possibly inherent logic of the steps.

Based on the same inventive concept, the embodiments of the present disclosure further provide a data processing device corresponding to the data processing method, and since the principle of solving the problem by the device in the embodiments of the present disclosure is similar to that of the data processing method in the embodiments of the present disclosure, the implementation of the device may refer to the implementation of the method, and the repetition is omitted.

As shown in fig. 6, a data processing apparatus provided in an embodiment of the present disclosure includes: a processor 10, a memory 20 storing machine readable instructions executable by the processor for executing machine readable instructions stored in the memory, the machine readable instructions when executed by the processor performing the following data processing procedures:

In a specific implementation, the storage 20 may include a memory and an external storage; the memory is also referred to as an internal memory, and is used for temporarily storing operation data in the processor 10 and data exchanged with an external memory such as a hard disk, and the processor 10 exchanges data with the external memory through the memory.

In a possible implementation manner, the data processing apparatus provided in the embodiment of the present disclosure further includes: and a display module 30.

The processor 10 is further configured to: based on the compressed data, the display module 30 is controlled to display a report of the monitoring data.

Here, the display module 30 includes, for example, at least one of: a display screen, a projection device, an augmented Reality (Augmented Reality, AR) module, a Virtual Reality (VR) module, and the like. The display module 30 can be controlled by the processing module 10, output a display interface, and display a report of the monitoring data generated based on the compressed data in the display interface.

In a possible implementation manner, the processor 10 is configured to, when acquiring the monitoring data of the target service interface and dividing the monitoring data into a plurality of data sets:

In a possible embodiment, the processor 10 is specifically configured to:

the processor 10 is specifically configured to perform, when the processor performs screening processing on any data set, data screening processing for at least one iteration cycle for the data set, and perform, in each iteration cycle:

In a possible implementation manner, the processor 10 is configured to, when determining the target floating interval based on the execution duration that is included in each of the pieces of call information in the to-be-processed data set corresponding to the current iteration period, determine:

In a possible embodiment, the processor 10 is further configured to: determining a time factor corresponding to the data set to be processed based on the starting time and/or the ending time corresponding to the data set to be processed;

In a possible implementation manner, the processor 10 is configured to, when determining a reference duration based on execution durations respectively included in a plurality of pieces of call information in the data set to be processed, and determining target call information to be screened out from call information in the data set to be processed based on the target floating interval and the reference duration, determine the target call information to be screened out:

The process flow of each module in the apparatus and the interaction flow between the modules may be described with reference to the related descriptions in the above method embodiments, which are not described in detail herein.

The embodiment of the disclosure also provides a monitoring device, which comprises the data processing device in any embodiment of the disclosure.

The disclosed embodiments also provide a computer program product comprising a computer program/instructions which, when executed, implement a data processing method as provided by the embodiments of the present disclosure. When the computer program/instructions are loaded and executed on a computer, the processes or functions described herein are performed in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, a network device, a user device, a core network device, an OAM, or other programmable apparatus. The computer program/instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium.

It will be clear to those skilled in the art that, for convenience and brevity of description, reference may be made to the corresponding process in the foregoing method embodiment for the specific working process of the apparatus described above, which is not described herein again. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, for example, the division of the units is merely a logical function division, and there may be other manners of division in actual implementation, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, device or unit indirect coupling or communication connection, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer readable storage medium executable by a processor. Based on such understanding, the technical solution of the present disclosure may be embodied in essence or a part contributing to the prior art or a part of the technical solution, or in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present disclosure. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

Finally, it should be noted that: the foregoing examples are merely specific embodiments of the present disclosure, and are not intended to limit the scope of the disclosure, but the present disclosure is not limited thereto, and those skilled in the art will appreciate that while the foregoing examples are described in detail, it is not limited to the disclosure: any person skilled in the art, within the technical scope of the disclosure of the present disclosure, may modify or easily conceive changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features thereof; such modifications, changes or substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the disclosure, and are intended to be included within the scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.

Claims

1. A data processing apparatus, comprising: a processor, a memory storing machine-readable instructions executable by the processor, the processor configured to execute the machine-readable instructions stored in the memory, the machine-readable instructions when executed by the processor, the processor performing the following data processing procedures:

Acquiring monitoring data of a target service interface, and dividing the monitoring data into a plurality of data sets; the different data sets comprise call information for carrying out service call on the target service interface in different interface call subperiods; the interface calling subperiods respectively corresponding to the plurality of data sets are a plurality of continuous subperiods determined based on the same target interface calling period; the target duration corresponding to the continuous subintervals is inversely related to the query rate per second corresponding to the target service interface;

screening out the call information of at least part of the data sets so as to screen out at least part of normal call information, and obtaining a target data set after screening out; wherein the number of normal call information in the target data set is inversely related to the number of abnormal call information in the target data set; the quantity of at least part of normal call information deleted from each of the at least part of data sets is related to a target screening threshold corresponding to each of the at least part of data sets; the data group longer from the current moment corresponds to a higher target screening threshold; the data group which is closer to the current moment corresponds to a lower target screening threshold value;

Taking the data group which is not subjected to screening treatment and the target data group after the screening treatment in the plurality of data groups as compressed data of the monitoring data;

the call information corresponding to each service call comprises: the execution duration of the service call; the processor is specifically configured to, when performing screening processing on the call information of each of the at least part of the data sets to screen at least part of the normal call information to obtain a target data set after the screening processing:

determining a target floating interval based on execution time lengths respectively included by a plurality of pieces of calling information in a data set to be processed corresponding to a current iteration period; the target floating interval is determined by utilizing a target screening threshold corresponding to the data set to be processed;

2. The data processing apparatus according to claim 1, wherein the processor, when acquiring the monitoring data for the target service interface and dividing the monitoring data into a plurality of data groups, is configured to:

3. The data processing apparatus according to claim 1, wherein the processor is specifically configured to:

4. The data processing apparatus according to claim 1, wherein the processor, when determining the target floating interval based on execution durations respectively included by pieces of call information in the data set to be processed corresponding to the current iteration period, is configured to:

5. The data processing apparatus of claim 4, wherein the processor is further configured to: determining a time factor corresponding to the data set to be processed based on the starting time and/or the ending time corresponding to the data set to be processed;

6. The apparatus according to claim 4 or 5, wherein the processor, when determining a reference time length based on execution time lengths respectively included in pieces of call information in the data set to be processed, and determining target call information to be screened out from call information in the data set to be processed based on the target floating zone and the reference time length, is configured to:

7. A monitoring device comprising a data processing apparatus as claimed in any one of claims 1 to 6.

8. A method of data processing, comprising: acquiring monitoring data of a target service interface, and dividing the monitoring data into a plurality of data sets; the different data sets comprise call information for carrying out service call on the target service interface in different interface call subperiods; the interface calling subperiods respectively corresponding to the plurality of data sets are a plurality of continuous subperiods determined based on the same target interface calling period; the target duration corresponding to the continuous subintervals is inversely related to the query rate per second corresponding to the target service interface;

the call information corresponding to each service call comprises: the execution duration of the service call;

screening out the call information of each at least part of the data sets so as to screen out at least part of normal call information, and obtaining a target data set after screening out, wherein the method specifically comprises the following steps of:

9. A computer storage medium comprising a computer program, characterized in that the data processing method according to claim 8 is implemented when the computer program is executed.