CN114938339A - Data processing method and related device - Google Patents
Data processing method and related device Download PDFInfo
- Publication number
- CN114938339A CN114938339A CN202210547255.4A CN202210547255A CN114938339A CN 114938339 A CN114938339 A CN 114938339A CN 202210547255 A CN202210547255 A CN 202210547255A CN 114938339 A CN114938339 A CN 114938339A
- Authority
- CN
- China
- Prior art keywords
- processed
- service
- service set
- historical
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 20
- 238000012545 processing Methods 0.000 claims abstract description 124
- 238000012163 sequencing technique Methods 0.000 claims abstract description 16
- 238000000034 method Methods 0.000 claims description 22
- 230000004044 response Effects 0.000 claims description 19
- 238000004364 calculation method Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 8
- 238000012360 testing method Methods 0.000 claims description 8
- 230000002159 abnormal effect Effects 0.000 claims description 5
- 230000000875 corresponding effect Effects 0.000 description 95
- 238000012549 training Methods 0.000 description 8
- 238000001514 detection method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000007689 inspection Methods 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000003648 Ljung–Box test Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/145—Network analysis or design involving simulating, designing, planning or modelling of a network
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/02—Banking, e.g. interest calculation or account maintenance
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/147—Network analysis or design for predicting network behaviour
Abstract
The embodiment of the application discloses a data processing method and a related device, when the processing duration corresponding to a service is predicted, a to-be-processed service set corresponding to a target time interval can be obtained first, the to-be-processed service set comprises a plurality of to-be-processed services, and each to-be-processed service has a corresponding service data volume. Sequencing the plurality of services to be processed according to the time sequence between the receiving moments corresponding to the plurality of services to be processed respectively to obtain a sequenced service set to be processed, so that the sequenced service set to be processed can carry time-related attributes; the processing device may determine the predicted processing time corresponding to the service set to be processed based on the sorted service set to be processed and the service data volume corresponding to each service to be processed, so that the processing time corresponding to the service set to be processed can be predicted more accurately by combining the time sequence and the service volume of the services.
Description
Technical Field
The present application relates to the field of business data processing technologies, and in particular, to a data processing method and a related apparatus.
Background
The business processing is an important part of the bank operation, and the monitoring of the business processing is one of effective means for guaranteeing the bank operation.
At present, the monitoring of the node processing service time consumption by a bank system mainly judges the operation rule of the service time consumption according to the existing time consumption record, manually sets a fixed threshold, and judges that the service processing has problems if the processing of batch service data exceeds the fixed threshold, so that the time consumption monitoring accuracy and effectiveness are poor, and potential safety hazards are easily brought.
Disclosure of Invention
In order to solve the technical problem, the application provides a data processing method, and a processing device can analyze the service processing time consumption by combining the service data volume, so as to obtain more accurate service prediction processing time.
The embodiment of the application discloses the following technical scheme:
in a first aspect, an embodiment of the present application discloses a data processing method, where the method includes:
acquiring a to-be-processed service set corresponding to a target time period, wherein the to-be-processed service set comprises a plurality of to-be-processed services, and each to-be-processed service has a corresponding service data volume;
sequencing the plurality of services to be processed according to the time sequence between the receiving moments respectively corresponding to the plurality of services to be processed to obtain a sequenced service set to be processed;
and determining the predicted processing time length corresponding to the service set to be processed based on the sorted service set to be processed and the service data volume corresponding to each service to be processed.
In a possible implementation manner, the determining, based on the sorted to-be-processed service sets and the service data amount corresponding to each to-be-processed service, a predicted processing duration corresponding to each to-be-processed service set includes:
and determining the predicted processing time length corresponding to the service set to be processed based on the sorted service set to be processed and the service data volume corresponding to each service to be processed through a batch time-consuming prediction model.
In one possible implementation, the batch time-consuming prediction model is obtained by:
acquiring a historical service set corresponding to a historical time period, wherein the historical service set has corresponding sample processing duration, the historical service set comprises a plurality of historical services processed in the historical time period, and the length of the historical time period is the same as that of the target time period;
sequencing the plurality of historical services according to the time sequence between the receiving moments respectively corresponding to the plurality of historical services to obtain a sequenced historical service set, wherein each historical service has a corresponding service data volume;
obtaining the time length to be processed according to the sorted historical service set and the service data volume corresponding to each historical service through an initial prediction model;
and adjusting the initial prediction model according to the difference between the sample processing duration and the to-be-processed duration to obtain the batch time-consuming prediction model.
In one possible implementation, the method further includes:
performing stationarity test on the sorted historical service set;
responding to the sequenced historical service set not meeting the stationarity checking requirement, and carrying out differential calculation on the service data volume corresponding to the historical service in the historical service set;
responding to the sorted historical service set after differential calculation to meet the stationarity testing requirement, and obtaining a first preprocessed historical service set;
the step of obtaining the time length to be processed according to the sorted historical service set and the service data amount corresponding to each historical service through the initial prediction model comprises the following steps:
and obtaining the undetermined processing time length according to the service data volume corresponding to each historical service in the first preprocessed historical service set and the historical service sequence in the first preprocessed historical service set through an initial prediction model.
In one possible implementation, the method further includes:
denoising the sorted historical service set;
the step of obtaining the time length to be processed according to the sorted historical service set and the service data amount corresponding to each historical service through the initial prediction model comprises the following steps:
and obtaining the time length to be processed according to the sorted historical service set subjected to denoising processing and the service data volume corresponding to each historical service through an initial prediction model.
In one possible implementation, the method further includes:
acquiring actual processing time corresponding to the service set to be processed;
and generating alarm information in response to the fact that the difference value between the actual processing time length and the predicted processing time length is larger than a preset threshold value, wherein the alarm information is used for indicating that the business processing of the business set to be processed is abnormal.
In a second aspect, an embodiment of the present application discloses a data processing apparatus, which includes a first obtaining unit, a sorting unit, and a determining unit:
the first acquisition unit is used for acquiring a to-be-processed service set corresponding to a target time period, wherein the to-be-processed service set comprises a plurality of to-be-processed services, and each to-be-processed service has a corresponding service data volume;
the sequencing unit is used for sequencing the plurality of services to be processed according to the time sequence between the receiving moments corresponding to the plurality of services to be processed respectively to obtain a sequenced service set to be processed;
and the determining unit is used for determining the predicted processing time length corresponding to the service set to be processed based on the sorted service set to be processed and the service data volume corresponding to each service to be processed.
In a possible implementation manner, the determining unit is specifically configured to:
and determining the predicted processing time length corresponding to the service set to be processed based on the sorted service set to be processed and the service data volume corresponding to each service to be processed through a batch time-consuming prediction model.
In one possible implementation, the batch time-consuming prediction model is obtained by:
acquiring a historical service set corresponding to a historical time period, wherein the historical service set has a corresponding sample processing duration, the historical service set comprises a plurality of historical services processed in the historical time period, and the length of the historical time period is the same as that of the target time period;
sequencing the plurality of historical services according to the time sequence between the receiving moments respectively corresponding to the plurality of historical services to obtain a sequenced historical service set, wherein each historical service has a corresponding service data volume;
obtaining the time length to be processed according to the sorted historical service set and the service data volume corresponding to each historical service through an initial prediction model;
and adjusting the initial prediction model according to the difference between the sample processing duration and the to-be-processed duration to obtain the batch time-consuming prediction model.
In one possible implementation, the apparatus further includes a checking unit, a first response unit, a second response unit, and a third response unit:
the checking unit is used for performing stationarity checking on the sorted historical service set;
the second response unit is used for responding that the sorted historical service set does not meet the stationarity checking requirement, and carrying out differential calculation on the service data volume corresponding to the historical service in the historical service set;
the third response unit is used for responding that the sorted historical service set after differential calculation meets the stationarity test requirement to obtain a first preprocessed historical service set;
the obtaining the length of time to be processed according to the sorted historical service set and the service data amount corresponding to each historical service through the initial prediction model comprises the following steps:
and obtaining the undetermined processing time length according to the service data volume corresponding to each historical service in the first preprocessed historical service set and the historical service sequence in the first preprocessed historical service set through an initial prediction model.
In one possible implementation, the apparatus further includes a denoising unit:
the denoising unit is used for denoising the sorted historical service set;
the step of obtaining the time length to be processed according to the sorted historical service set and the service data amount corresponding to each historical service through the initial prediction model comprises the following steps:
and obtaining the time length to be processed according to the sorted historical service set subjected to denoising processing and the service data volume corresponding to each historical service through an initial prediction model.
In a possible implementation manner, the apparatus further includes a second obtaining unit and a third responding unit:
the second obtaining unit is configured to obtain an actual processing duration corresponding to the service set to be processed;
and the third response unit is configured to generate alarm information in response to that a difference between the actual processing duration and the predicted processing duration is greater than a preset threshold, where the alarm information is used to indicate that the service processing of the service set to be processed is abnormal.
In a third aspect, an embodiment of the present application discloses a computer device, including a processor and a memory:
the memory is used for storing program codes and transmitting the program codes to the processor;
the processor is configured to perform the data processing method of any one of the first aspect according to instructions in the program code.
In a fourth aspect, an embodiment of the present application discloses a computer-readable storage medium, which is used for storing a computer program, where the computer program is used for executing the data processing method in any one of the first aspect.
In a fifth aspect, an embodiment of the present application discloses a computer program product including instructions, which when run on a computer, cause the computer to execute the data processing method of any one of the first aspects.
According to the technical scheme, when the processing time corresponding to the service is predicted, the service set to be processed corresponding to the target time period can be obtained first, the service set to be processed comprises a plurality of services to be processed, and each service to be processed has the corresponding service data volume. Sequencing the plurality of services to be processed according to the time sequence between the receiving moments corresponding to the plurality of services to be processed respectively to obtain a sequenced service set to be processed, so that the sequenced service set to be processed can carry time-related attributes; the processing device may determine the predicted processing time corresponding to the service set to be processed based on the sorted service set to be processed and the service data volume corresponding to each service to be processed, so that the processing time corresponding to the service set to be processed can be predicted more accurately by combining the time sequence and the service volume of the services.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present application;
fig. 2 is a schematic diagram of a data processing method according to an embodiment of the present application;
fig. 3 is a block diagram of a data processing apparatus according to an embodiment of the present disclosure.
Detailed Description
Embodiments of the present application are described below with reference to the accompanying drawings.
The method for manually setting the threshold value in the related technology has various disadvantages, firstly, the bank system has numerous batch nodes, more batch nodes are inevitably generated along with the production of a new bank business new system, and the manual setting of the threshold value is time-consuming and labor-consuming; secondly, under the condition that the network and system environment are not changed, the time consumption of the batch nodes is positively correlated with the service volume of batch processing, when the service promotion is performed, the service volume is increased, the time consumption of the batch nodes is prolonged, and the threshold value set manually is triggered, so that the unreasonable situation is caused, and the processing duration of the batch service is difficult to predict accurately.
In order to solve the technical problem, the application provides a data processing method, and a processing device can analyze the service processing time consumption by combining the service data volume, so as to obtain more accurate service prediction processing time.
It is understood that the method may be applied to a processing device, which is a processing device capable of data processing, for example, a terminal device or a server having a data processing function. The method can be independently executed through the terminal equipment or the server, can also be applied to a network scene of communication between the terminal equipment and the server, and can be executed through the cooperation of the terminal equipment and the server. The terminal device may be a computer, a mobile phone, or the like. The server may be understood as an application server or a Web server, and in actual deployment, the server may be an independent server or a cluster server.
Next, a data processing method provided by an embodiment of the present application will be described with reference to the drawings.
Referring to fig. 1, fig. 1 is a flowchart of a data processing method provided in an embodiment of the present application, where the method includes:
s101: and acquiring a to-be-processed service set corresponding to the target time period.
The target time interval may be any time interval, the set of services to be processed includes a plurality of services to be processed, and each service to be processed has a corresponding service data volume.
S102: and sequencing the plurality of services to be processed according to the time sequence between the receiving moments corresponding to the plurality of services to be processed respectively to obtain a sequenced service set to be processed.
The receiving time is the time when the service to be processed is received, and the processing device may sort the plurality of services to be processed based on the time when the service to be processed is received, so that the sorted plurality of services to be processed may carry information on a time sequence, thereby further improving the accuracy of predicting the processing time length.
S103: and determining the predicted processing time length corresponding to the service set to be processed based on the sorted service set to be processed and the service data volume corresponding to each service to be processed.
It can be understood that, in a general case, the larger the data volume of the pending transaction, the longer the processing time required by the bank node in processing the pending transaction. Therefore, the processing device can use the traffic data amount as a standard for measuring the traffic processing time. Based on the service data volume and the time sequence information obtained after sequencing, the processing equipment can accurately predict the processing time required by the bank node to process the service set to be processed.
According to the technical scheme, when the processing time corresponding to the service is predicted, the service set to be processed corresponding to the target time period can be obtained first, the service set to be processed comprises a plurality of services to be processed, and each service to be processed has the corresponding service data volume. Sequencing the plurality of services to be processed according to the time sequence between the receiving moments corresponding to the plurality of services to be processed respectively to obtain a sequenced service set to be processed, so that the sequenced service set to be processed can carry time-related attributes; the processing device may determine the predicted processing time corresponding to the service set to be processed based on the sorted service set to be processed and the service data volume corresponding to each service to be processed, so that the processing time corresponding to the service set to be processed can be predicted more accurately by combining the time sequence and the service volume of the services.
In a possible implementation manner, the processing device may determine, by using a batch time-consuming prediction model, a predicted processing duration corresponding to the service set to be processed based on the sorted service sets to be processed and the service data amount corresponding to each service to be processed, where the batch time-consuming prediction model may be obtained by:
the processing device may obtain a historical service set corresponding to a historical period, where the historical service set has a corresponding sample processing duration, and the sample processing duration is a duration required by the bank node to process the historical service in the historical service set in an actual situation. The historical service set comprises a plurality of historical services processed and completed in a historical period, and the length of the historical period is the same as that of the target period. The processing device may sort the plurality of historical services according to time sequences between receiving times corresponding to the plurality of historical services, respectively, to obtain a sorted historical service set, where each historical service has a corresponding service data volume. Then, the processing device may obtain a pending processing time length according to the sorted historical service set and the service data amount corresponding to each historical service through the initial prediction model, where the pending processing time length is a time length predicted by the initial prediction model according to the data. According to the difference between the sample processing time length and the time length to be processed, the accuracy of the initial prediction model in the time length to be processed can be reflected, so that the processing equipment can adjust the initial prediction model according to the difference, the initial prediction model can learn how to accurately determine the processing time length corresponding to the service to be processed, and the batch time-consuming prediction model can be obtained.
In one possible implementation, the processing device may perform various preprocessing operations on the trained data in order to obtain a more accurate batch time-consuming prediction model. In a possible implementation manner, the processing device may perform stationarity check on the sorted historical service set, where the stationarity check is used to detect whether a plurality of historical services have a large traffic difference, and the traffic data with a large difference is likely to affect model training, and in response to that the sorted historical service set does not satisfy the stationarity check requirement, the processing device may perform difference calculation on the traffic data amount corresponding to the historical services in the historical service set.
After the difference calculation, the processing device may perform stationarity detection on the historical service set again, and obtain a first preprocessed historical service set in response to the sorted historical service set after the difference calculation satisfying the stationarity detection requirement. The processing device may obtain the pending processing time length according to the service data amount corresponding to each historical service in the first preprocessed historical service set and the historical service sequence in the first preprocessed historical service set through the initial prediction model.
For example, the processing device may first construct a multivariate time series based on the raw data, sort the historical services according to the chronological order, and construct a multivariate time series dataset.
Then, carrying out stability inspection on the multivariate time sequence, and carrying out differential calculation if the multivariate time sequence is not stable until the stability detection is passed;
in which, the sample processing duration and the historical traffic data may be subjected to a stationarity ADF check, i.e., a unit root check. If the time sequence data does not meet the stationarity requirement, the data is subjected to differential calculation until the stationarity detection is passed, namely the p-value of the statistic is less than 0.05.
In addition to performing stationarity tests, in one possible implementation, the processing device may perform denoising on the data. The processing device can perform denoising processing on the sorted historical service set, and then obtains the time length to be processed according to the sorted historical service set subjected to denoising processing and the service data volume corresponding to each historical service through the initial prediction model.
For example, the processing device may perform a white noise test on the multivariate time sequence, such as a white noise Ljung-Box test on the sample processing duration and the historical traffic data, and if the p value of the statistic is less than 0.05 of the significance level, the sequence is considered to be a non-white noise sequence. The processing device may then divide the raw data into a training set and a test set in a 4:3 ratio, determine model parameters based on the training samples, including the steps of:
(1) combining the stationary time sequence difference order d value, carrying out BIC calculation with different autocorrelation coefficient p values and sliding average coefficient q values, taking the q value and the p value corresponding to the minimum BIC value, constructing an ARIMAX (p, d, q) model, and bringing a training data set into the model for training;
the method comprises the steps of adopting a BIC Bayesian Information Criterion (BIC), utilizing different autocorrelation coefficient p values and sliding average coefficient q values to carry out BIC calculation, taking the q value and the p value corresponding to the minimum BIC value, determining a difference order d value by combining whether difference is carried out on a stable time sequence or not, constructing an ARIMAX (p, d, q) model, substituting training set data into the model, and carrying out model training.
(2) And detecting the model effect by using the residual error.
And D-W inspection is carried out on the trained model, and if the D-W inspection value is close to 2, the model effect is ideal.
After a batch time-consuming prediction model is obtained through training and a predicted processing time corresponding to a service set to be processed is determined based on the model, a processing device can obtain an actual processing time corresponding to the service set to be processed, wherein the actual processing time refers to a time that the service in the service set to be processed passes through in an actual situation.
In response to that the difference between the actual processing time length and the predicted processing time length is greater than the preset threshold, the processing device may generate alarm information, where the alarm information is used to indicate that the service processing of the service set to be processed is abnormal, as shown in fig. 2.
Based on the data processing method provided in the foregoing embodiment, an embodiment of the present application further provides a data processing apparatus, referring to fig. 3, and fig. 3 is a block diagram of a structure of the data processing apparatus provided in the embodiment of the present application, where the apparatus includes a first obtaining unit 301, a sorting unit 302, and a determining unit 303:
the first obtaining unit 301 is configured to obtain a to-be-processed service set corresponding to a target time period, where the to-be-processed service set includes multiple to-be-processed services, and each to-be-processed service has a corresponding service data volume;
the sorting unit 302 is configured to sort the multiple services to be processed according to time sequences between receiving times corresponding to the multiple services to be processed, respectively, to obtain a sorted set of services to be processed;
the determining unit 303 is configured to determine a predicted processing time corresponding to the service set to be processed based on the sorted service set to be processed and the service data amount corresponding to each service to be processed.
In a possible implementation manner, the determining unit 303 is specifically configured to:
and determining the predicted processing time length corresponding to the service set to be processed based on the sorted service set to be processed and the service data volume corresponding to each service to be processed through a batch time-consuming prediction model.
In one possible implementation, the batch time-consuming prediction model is obtained by:
acquiring a historical service set corresponding to a historical time period, wherein the historical service set has corresponding sample processing duration, the historical service set comprises a plurality of historical services processed in the historical time period, and the length of the historical time period is the same as that of the target time period;
sequencing the plurality of historical services according to the time sequence between the receiving moments respectively corresponding to the plurality of historical services to obtain a sequenced historical service set, wherein each historical service has a corresponding service data volume;
obtaining the length of time to be processed according to the sorted historical service set and the service data amount corresponding to each historical service through an initial prediction model;
and adjusting the initial prediction model according to the difference between the sample processing duration and the to-be-processed duration to obtain the batch time-consuming prediction model.
In one possible implementation, the apparatus further includes a verification unit, a first response unit, a second response unit, and a third response unit:
the checking unit is used for performing stationarity checking on the sorted historical service set;
the second response unit is used for responding that the sorted historical service set does not meet the stationarity checking requirement, and carrying out differential calculation on the service data volume corresponding to the historical service in the historical service set;
the third response unit is used for responding that the sorted historical service set after differential calculation meets the stationarity test requirement to obtain a first preprocessed historical service set;
the obtaining the length of time to be processed according to the sorted historical service set and the service data amount corresponding to each historical service through the initial prediction model comprises the following steps:
and obtaining the undetermined processing time length according to the service data volume corresponding to each historical service in the first preprocessed historical service set and the historical service sequence in the first preprocessed historical service set through an initial prediction model.
In one possible implementation, the apparatus further includes a denoising unit:
the denoising unit is used for denoising the sorted historical service set;
the step of obtaining the time length to be processed according to the sorted historical service set and the service data amount corresponding to each historical service through the initial prediction model comprises the following steps:
and obtaining the time length to be processed according to the sorted historical service set subjected to denoising processing and the service data volume corresponding to each historical service through an initial prediction model.
In a possible implementation manner, the apparatus further includes a second obtaining unit and a third responding unit:
the second obtaining unit is configured to obtain an actual processing duration corresponding to the service set to be processed;
and the third response unit is configured to generate alarm information in response to that a difference between the actual processing duration and the predicted processing duration is greater than a preset threshold, where the alarm information is used to indicate that the service processing of the service set to be processed is abnormal.
The embodiment of the application also provides a computer device, and the processor included in the terminal device further has the following functions:
acquiring a to-be-processed service set corresponding to a target time period, wherein the to-be-processed service set comprises a plurality of to-be-processed services, and each to-be-processed service has a corresponding service data volume;
sequencing the plurality of services to be processed according to the time sequence between the receiving moments respectively corresponding to the plurality of services to be processed to obtain a sequenced service set to be processed;
and determining the predicted processing time length corresponding to the service set to be processed based on the sorted service set to be processed and the service data volume corresponding to each service to be processed.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. Wherein the name of an element does not in some cases constitute a limitation on the element itself.
In addition, an embodiment of the present application further provides a storage medium, where the storage medium is used to store a computer program, and the computer program is used to execute the data processing method provided in the foregoing embodiment.
The embodiment of the present application further provides a computer program product including instructions, which when run on a computer, causes the computer to execute the data processing method provided by the above embodiment.
Those of ordinary skill in the art will understand that: all or part of the steps for realizing the method embodiments can be completed by hardware related to program instructions, the program can be stored in a computer readable storage medium, and the program executes the steps comprising the method embodiments when executed; and the aforementioned storage medium may be at least one of the following media: various media that can store program codes, such as read-only memory (ROM), RAM, magnetic disk, or optical disk.
It should be noted that, in the present specification, all the embodiments are described in a progressive manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the apparatus and system embodiments, because they are substantially similar to the method embodiments, are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for related points. The above-described embodiments of the apparatus and system are merely illustrative, and the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
The above description is only one specific embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A method of data processing, the method comprising:
acquiring a to-be-processed service set corresponding to a target time period, wherein the to-be-processed service set comprises a plurality of to-be-processed services, and each to-be-processed service has a corresponding service data volume;
sequencing the plurality of services to be processed according to the time sequence between the receiving moments respectively corresponding to the plurality of services to be processed to obtain a sequenced service set to be processed;
and determining the predicted processing time length corresponding to the service set to be processed based on the sorted service set to be processed and the service data volume corresponding to each service to be processed.
2. The method of claim 1, wherein the determining a predicted processing duration corresponding to the to-be-processed service set based on the sorted to-be-processed service sets and a service data volume corresponding to each to-be-processed service comprises:
and determining the predicted processing time length corresponding to the service set to be processed based on the sorted service set to be processed and the service data volume corresponding to each service to be processed through a batch time-consuming prediction model.
3. The method of claim 2, wherein the batch-time consuming predictive model is derived by:
acquiring a historical service set corresponding to a historical time period, wherein the historical service set has corresponding sample processing duration, the historical service set comprises a plurality of historical services processed in the historical time period, and the length of the historical time period is the same as that of the target time period;
sequencing the plurality of historical services according to the time sequence between the receiving moments respectively corresponding to the plurality of historical services to obtain a sequenced historical service set, wherein each historical service has a corresponding service data volume;
obtaining the time length to be processed according to the sorted historical service set and the service data volume corresponding to each historical service through an initial prediction model;
and adjusting the initial prediction model according to the difference between the sample processing duration and the to-be-processed duration to obtain the batch time-consuming prediction model.
4. The method of claim 3, further comprising:
performing stationarity test on the sorted historical service set;
responding to the sequenced historical service set not meeting the stationarity checking requirement, and carrying out differential calculation on the service data volume corresponding to the historical service in the historical service set;
responding to the sorted historical service set after the difference calculation to meet the stationarity checking requirement, and obtaining a first preprocessed historical service set;
the step of obtaining the time length to be processed according to the sorted historical service set and the service data amount corresponding to each historical service through the initial prediction model comprises the following steps:
and obtaining the undetermined processing time length according to the service data volume corresponding to each historical service in the first preprocessed historical service set and the historical service sequence in the first preprocessed historical service set through an initial prediction model.
5. The method of claim 3, further comprising:
denoising the sorted historical service set;
the step of obtaining the time length to be processed according to the sorted historical service set and the service data amount corresponding to each historical service through the initial prediction model comprises the following steps:
and obtaining the time length to be processed according to the sorted historical service set subjected to denoising processing and the service data volume corresponding to each historical service through an initial prediction model.
6. The method of claim 1, further comprising:
acquiring actual processing time corresponding to the service set to be processed;
and generating alarm information in response to the fact that the difference value between the actual processing time length and the predicted processing time length is larger than a preset threshold value, wherein the alarm information is used for indicating that the service processing of the service set to be processed is abnormal.
7. A data processing apparatus, characterized in that the apparatus comprises a first acquisition unit, a sorting unit, and a determination unit:
the first obtaining unit is configured to obtain a to-be-processed service set corresponding to a target time period, where the to-be-processed service set includes multiple to-be-processed services, and each to-be-processed service has a corresponding service data volume;
the sequencing unit is used for sequencing the plurality of services to be processed according to the time sequence between the receiving moments corresponding to the plurality of services to be processed respectively to obtain a sequenced service set to be processed;
and the determining unit is used for determining the predicted processing time length corresponding to the service set to be processed based on the sorted service set to be processed and the service data volume corresponding to each service to be processed.
8. A computer device, the computer device comprising a processor and a memory:
the memory is used for storing program codes and transmitting the program codes to the processor;
the processor is configured to perform the data processing method of any one of claims 1-6 according to instructions in the program code.
9. A computer-readable storage medium for storing a computer program for executing the data processing method of any one of claims 1 to 6.
10. A computer program product comprising instructions which, when run on a computer, cause the computer to perform the data processing method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210547255.4A CN114938339A (en) | 2022-05-19 | 2022-05-19 | Data processing method and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210547255.4A CN114938339A (en) | 2022-05-19 | 2022-05-19 | Data processing method and related device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114938339A true CN114938339A (en) | 2022-08-23 |
Family
ID=82864452
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210547255.4A Pending CN114938339A (en) | 2022-05-19 | 2022-05-19 | Data processing method and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114938339A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115907710A (en) * | 2023-01-06 | 2023-04-04 | 浪潮通信信息系统有限公司 | Method and device for determining service processing time limit, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019193570A1 (en) * | 2018-04-07 | 2019-10-10 | Tata Consultancy Services Limited | Batch jobs execution time prediction using distinct service demand of threads and instantaneous cpu utilization |
| CN112926877A (en) * | 2021-03-26 | 2021-06-08 | 中国建设银行股份有限公司 | Health condition judgment method, device, equipment and storage medium for batch processing business |
| CN113918884A (en) * | 2020-07-09 | 2022-01-11 | 顺丰科技有限公司 | Traffic prediction model construction method and traffic prediction method |
| CN114062993A (en) * | 2021-10-20 | 2022-02-18 | 国网上海市电力公司 | CVT error state prediction method based on time convolution network |
| CN114066089A (en) * | 2021-11-25 | 2022-02-18 | 中国工商银行股份有限公司 | Batch job operation time-consuming interval determining method and device |
-
2022
- 2022-05-19 CN CN202210547255.4A patent/CN114938339A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019193570A1 (en) * | 2018-04-07 | 2019-10-10 | Tata Consultancy Services Limited | Batch jobs execution time prediction using distinct service demand of threads and instantaneous cpu utilization |
| CN113918884A (en) * | 2020-07-09 | 2022-01-11 | 顺丰科技有限公司 | Traffic prediction model construction method and traffic prediction method |
| CN112926877A (en) * | 2021-03-26 | 2021-06-08 | 中国建设银行股份有限公司 | Health condition judgment method, device, equipment and storage medium for batch processing business |
| CN114062993A (en) * | 2021-10-20 | 2022-02-18 | 国网上海市电力公司 | CVT error state prediction method based on time convolution network |
| CN114066089A (en) * | 2021-11-25 | 2022-02-18 | 中国工商银行股份有限公司 | Batch job operation time-consuming interval determining method and device |
Non-Patent Citations (1)
| Title |
|---|
| 侯永铭，姚远源，雷安琪，张磊: "基于时间序列分析的银行作业执行用时预测的应用研究", 《中国金融电脑》, pages 71 - 75 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115907710A (en) * | 2023-01-06 | 2023-04-04 | 浪潮通信信息系统有限公司 | Method and device for determining service processing time limit, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106951984B (en) | Dynamic analysis and prediction method and device for system health degree | |
| CN110839016B (en) | Abnormal flow monitoring method, device, equipment and storage medium | |
| US10248528B2 (en) | System monitoring method and apparatus | |
| US20190392252A1 (en) | Systems and methods for selecting a forecast model for analyzing time series data | |
| CN113518011B (en) | Abnormality detection method and apparatus, electronic device, and computer-readable storage medium | |
| CN112835769A (en) | Service data abnormity diagnosis method, device, equipment and storage medium | |
| CN110008247B (en) | Method, device and equipment for determining abnormal source and computer readable storage medium | |
| CN106649832B (en) | Estimation method and device based on missing data | |
| CN111796957B (en) | Transaction abnormal root cause analysis method and system based on application log | |
| CN111221706B (en) | CPU utilization rate prediction method, system, medium and equipment | |
| CN113010389A (en) | Training method, fault prediction method, related device and equipment | |
| CN111639798A (en) | Intelligent prediction model selection method and device | |
| CN102056200A (en) | Service processing flow monitoring method and system | |
| CN115204536A (en) | Building equipment fault prediction method, device, equipment and storage medium | |
| CN114938339A (en) | Data processing method and related device | |
| CN110795324A (en) | Data processing method and device | |
| CN115705413A (en) | Method and device for determining abnormal log | |
| CN110490132B (en) | Data processing method and device | |
| CN113123955A (en) | Plunger pump abnormality detection method and device, storage medium and electronic device | |
| CN116232851A (en) | Early warning method and device for network abnormality, electronic equipment and storage medium | |
| CN115293735A (en) | Unmanned factory industrial internet platform monitoring management method and system | |
| CN110098983B (en) | Abnormal flow detection method and device | |
| CN114331688A (en) | Method and device for detecting batch operation state of bank counter system business | |
| CN114239952A (en) | Method, device, equipment and medium for detecting abnormity of key performance index | |
| EP3457609B1 (en) | System and method for computing of anomalies based on frequency driven transformation and computing of new features based on point anomaly density |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |