CN117131421A - Verification method and device for data processing result and computer equipment - Google Patents
Verification method and device for data processing result and computer equipment Download PDFInfo
- Publication number
- CN117131421A CN117131421A CN202311164447.8A CN202311164447A CN117131421A CN 117131421 A CN117131421 A CN 117131421A CN 202311164447 A CN202311164447 A CN 202311164447A CN 117131421 A CN117131421 A CN 117131421A
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- risk level
- processing result
- sample data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 381
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000012795 verification Methods 0.000 title abstract description 28
- 230000002159 abnormal effect Effects 0.000 claims abstract description 27
- 230000008569 process Effects 0.000 claims abstract description 11
- 230000005856 abnormality Effects 0.000 claims description 50
- 238000012544 monitoring process Methods 0.000 claims description 25
- 238000004590 computer program Methods 0.000 claims description 20
- 238000012549 training Methods 0.000 claims description 17
- 238000012502 risk assessment Methods 0.000 claims description 11
- 238000010586 diagram Methods 0.000 description 10
- 238000001514 detection method Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000000670 limiting effect Effects 0.000 description 2
- 230000002829 reductive effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 238000009954 braiding Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000002860 competitive effect Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/2431—Multiple classes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The application relates to a data processing result verification method, a data processing result verification device and computer equipment. The method comprises the following steps: acquiring data and processing attributes of the data, wherein the processing attributes comprise at least one of the following: the method comprises the steps of a module where data is located, a processing object of the data, a processing state of the data, a reason for abnormal data, a reason for refusing to process the data, a type of accessory materials, a related external single number and a description of a title; inputting the data and the processing attribute of the data into a preset processing model, and outputting the prediction processing result of the data; and comparing and checking the original processing result of the data based on the prediction processing result. By adopting the method, the data processing result can be automatically predicted, and the error data processing result can be rapidly and accurately found out.
Description
Technical Field
The present application relates to the field of big data technologies, and in particular, to a method, an apparatus, a computer device, a storage medium, and a computer program product for verifying a data processing result.
Background
With the development of big data technology, the situation that big data is not updated timely occurs, the basic information of enterprises is asymmetric with the data content of a specific dimension on a data service platform, or the disclosed enterprise data is asymmetric with the data content of the specific dimension on the data service platform. And the data service platform receives the asymmetric feedback to form the objection data. The processing amount of the objection data is larger, and more objection data processing results are formed. However, the processing results of the objection data are irregular, and in the conventional technology, all inspection or spot check needs to be performed on the processing results of the objection data manually, so as to reduce rollback or abnormal increase of the processing results of the data.
However, the data volume of the processing result of the manually detected objection data is too large, the daily-growth data volume is too fast to increase, and the detection efficiency and the accuracy of the manually detected abnormal data processing result are low.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a method, an apparatus, a computer device, a computer-readable storage medium, and a computer program product for verifying a data processing result.
In a first aspect, the present application provides a method for verifying a data processing result. The method comprises the following steps:
Acquiring data and processing attributes of the data, wherein the processing attributes comprise at least one of the following: the method comprises the steps of a module where data is located, a processing object of the data, a processing state of the data, a reason for abnormal data, a reason for refusing to process the data, a type of accessory materials, a related external single number and a description of a title;
inputting the data and the processing attribute of the data into a preset processing model, and outputting the prediction processing result of the data;
and comparing and checking the original processing result of the data based on the prediction processing result.
In one embodiment, the training mode of the processing model includes:
acquiring sample data, processing attributes of the sample data and expected processing results;
analyzing the processing attribute of the sample data and determining the risk level of the sample data;
constructing an initial processing model, and training the initial processing model according to the sample data, the processing attribute of the sample data, the risk level and the expected processing result;
and determining the initial processing model as a processing model under the condition that the accuracy rate of the initial processing model is higher than a preset threshold value.
In one embodiment, the analyzing the processing attribute of the sample data to determine the risk level of the sample data includes:
acquiring processing attributes of the sample data;
comparing the processing attribute of the sample data with a preset risk assessment rule to determine the risk level of the sample data; the risk assessment rule comprises an association relation between a processing attribute and a risk level.
In one embodiment, the analyzing the processing attribute of the sample data to determine the risk level of the sample data includes:
obtaining dimension information of the sample data according to a module in which the sample data are located;
determining an initial risk level of the sample data according to a weight preset by dimension information of the sample data;
and updating the initial risk level according to the processing object, the processing state and the reason of the data abnormality of the sample data to obtain the risk level of the sample data.
In one embodiment, the inputting the data and the processing attribute of the data into a preset processing model, and outputting the prediction processing result of the data includes:
The processing model outputs the risk level of the data;
and monitoring the data with the risk level higher than the preset threshold value under the condition that the risk level of the data is higher than the preset threshold value.
In one embodiment, the monitoring the data with the risk level higher than the preset threshold includes:
setting expected processing results for data with risk levels higher than a preset threshold value;
acquiring an actual processing result of processing the data with the risk level higher than a preset threshold value;
and verifying the actual processing result based on the expected processing result.
In one embodiment, the method further comprises:
periodically monitoring the data with the risk level higher than a preset threshold value;
and feeding back the abnormality under the condition that the abnormality frequency of the abnormal data processing is greater than a preset threshold value.
In one embodiment, the method further comprises:
summarizing the prediction processing result of the data passing through the processing module;
analyzing the prediction processing result of the data, and determining the reason category of the data abnormality;
and determining the processing mode of the data abnormality based on the reason category of the data abnormality.
In a second aspect, the application further provides a device for verifying the data processing result. The device comprises:
the device comprises an acquisition module for acquiring data and processing attributes of the data, wherein the processing attributes comprise at least one of the following: the method comprises the steps of a module where data is located, a processing object of the data, a processing state of the data, a reason for abnormal data, a reason for refusing to process the data, a type of accessory materials, a related external single number and a description of a title;
the prediction module is used for inputting the data and the processing attribute of the data into a preset processing model and outputting a prediction processing result of the data;
and the comparison module is used for comparing and checking the original processing result of the data based on the prediction processing result.
In one embodiment, the prediction module includes:
the acquisition sub-module is used for acquiring sample data, processing attributes of the sample data and expected processing results;
the risk sub-module is used for analyzing the processing attribute of the sample data and determining the risk level of the sample data;
the training sub-module is used for constructing an initial processing model and training the initial processing model according to the sample data, the processing attribute of the sample data, the risk level and the expected processing result;
And the model submodule is used for determining the initial processing model as the processing model under the condition that the accuracy rate of the initial processing model is higher than a preset threshold value.
In one embodiment, the risk submodule includes:
an acquisition unit configured to acquire a processing attribute of the sample data;
the risk unit is used for comparing the processing attribute of the sample data with a preset risk evaluation rule and determining the risk level of the sample data; the risk assessment rule comprises an association relation between a processing attribute and a risk level.
In one embodiment, the risk submodule includes:
the dimension unit is used for obtaining dimension information of the sample data according to a module where the sample data are located;
the initial risk unit is used for determining an initial risk level of the sample data according to the weight preset by the dimension information of the sample data;
and the updating risk unit is used for updating the initial risk level according to the processing object, the processing state and the reason of the data abnormality of the sample data to obtain the risk level of the sample data.
In one embodiment, the prediction module includes:
The output sub-module is used for outputting the risk level of the data by the processing model;
and the monitoring sub-module is used for monitoring the data with the risk level higher than the preset threshold value under the condition that the risk level of the data is higher than the preset threshold value.
In one embodiment, the monitoring submodule includes:
an expected unit for setting an expected processing result for data having a risk level higher than a preset threshold value;
the processing unit is used for obtaining an actual processing result of processing the data with the risk level higher than a preset threshold value;
and the verification unit is used for verifying the actual processing result based on the expected processing result.
In one embodiment, the monitoring sub-module further comprises:
the detection unit is used for periodically monitoring the data with the risk level higher than a preset threshold value;
and the feedback unit is used for feeding back the abnormality when the abnormality frequency of the abnormal data processing is greater than a preset threshold value.
In one embodiment, the apparatus further comprises:
the summarizing module is used for summarizing the prediction processing result of the data passing through the processing module;
the reason category module is used for analyzing the prediction processing result of the data and determining the reason category of the data abnormality;
And the processing module is used for determining the processing mode of the data abnormality based on the reason category of the data abnormality.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing a form identification method according to any of the embodiments of the present disclosure when the processor executes the computer program.
In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements a form recognition method as set forth in any of the embodiments of the present disclosure.
In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements a form recognition method according to any one of the embodiments of the present disclosure.
According to the verification method, the verification device, the computer equipment, the storage medium and the computer program product of the data processing result, the risk level of the data is obtained by analyzing the data and the processing attribute of the data, the model is trained according to the data, the processing attribute of the data, the risk level and the expected processing result, the accuracy of the model is greatly improved, the high risk data predicted by the model is detected, the frequent repeated occurrence of the same data errors is avoided, the predicted data is analyzed, the cause type of the data abnormality is determined, the processing mode of the data abnormality is determined, the implementation of a plan is perfected, the processing flow of each department is standardized, the data content standard and the business rule are unified, and the processing efficiency of the objection data is simplified and improved.
Drawings
FIG. 1 is a first flow chart of a method for verifying data processing results in one embodiment;
FIG. 2 is a second flow chart of a method for verifying data processing results in one embodiment;
FIG. 3 is a third flow chart of a method for verifying data processing results in one embodiment;
FIG. 4 is a fourth flowchart of a method for verifying a data processing result according to one embodiment;
FIG. 5 is a fifth flowchart of a method for verifying a data processing result according to one embodiment;
FIG. 6 is a sixth flowchart of a method for verifying a data processing result according to one embodiment;
FIG. 7 is a seventh flowchart of a method for verifying a data processing result according to one embodiment;
FIG. 8 is a schematic diagram of an eighth flowchart of a method for verifying a data processing result in one embodiment;
FIG. 9 is a first schematic diagram of a verification device for data processing results in one embodiment;
FIG. 10 is a second schematic diagram of a verification device for data processing results in one embodiment;
FIG. 11 is a third schematic diagram of a verification device for data processing results in one embodiment;
FIG. 12 is a fourth schematic diagram of a verification device for data processing results in one embodiment;
FIG. 13 is a fifth schematic diagram of a verification device for data processing results in one embodiment;
FIG. 14 is a sixth schematic diagram of a verification device for data processing results in one embodiment;
FIG. 15 is a seventh schematic diagram of a verification device for data processing results in one embodiment;
FIG. 16 is an eighth schematic diagram of a verification device for data processing results in one embodiment;
fig. 17 is an internal structural view of a computer device in one embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
In one embodiment, as shown in fig. 1, a method for verifying a data processing result is provided, including the following steps:
step S1, acquiring data and processing attributes of the data, wherein the processing attributes comprise at least one of the following: the method comprises the steps of a module where data is located, a processing object of the data, a processing state of the data, a reason for abnormal data, a reason for refusing to process the data, a type of accessory materials, a related external single number and a description of a title;
step S3, inputting the data and the processing attribute of the data into a preset processing model, and outputting a prediction processing result of the data;
And S5, comparing and checking the original processing result of the data based on the prediction processing result.
Wherein, the original processing result may be an actual processing result of the data processing. Alternatively, the predicted processing result of the data may be a result of predicting abnormal data processing, where the predicted processing result may include: verification completion, rejection processing, masking/deleting, updating processing, etc.
In an exemplary embodiment, the acquiring data may include acquiring all data and processing attributes of the data. In another exemplary embodiment, the acquiring data may include covering all links with hierarchical extraction, and configuring the extraction proportion according to the data dimension weight, for example, the processing links include: customer service, collection and editing, content auditing, development and product processing, and the data acquisition can be as follows: collecting and braiding: content auditing: development: product processing was performed at a ratio of 100:200:100:50:75 to obtain data.
In one exemplary embodiment, the data may include data that is presented with anomalies from the disclosed data content. In another exemplary embodiment, the presented data content is data that differs from the actual information of the data object.
In an exemplary embodiment, the processing attributes of the data may include: some attributes in the processing of exception data, wherein the processing attributes may include: the module in which the data is located, the processing object of the data, the processing state of the data, the reason for data abnormality, the reason for data refusal processing, the type of the accessory material, the description of the associated external unit number and title, and the like.
In an exemplary embodiment, the verifying the original processing result of the data may include comparing the predicted processing result predicted by the processing model with the original processing result to see if the predicted processing result is the same. And if the data are different, checking and updating the original processing result of the data, putting the updated original processing result into a model for training, and continuously updating the model. In another exemplary embodiment, in a case where a predicted processing result of the processing model for the data is different from an original processing result, the data and a processing attribute of the data are put into the model for re-prediction, and whether the original processing result is correct and corrected is judged based on the predicted result.
According to the data processing result verification method, the data and the processing attributes of the data are obtained, the data and the processing attributes are subjected to model prediction to obtain the data prediction processing result, and the original processing result is subjected to comparison correction based on the prediction processing result, so that whether the original processing result is correct or not can be rapidly and accurately judged, the wrong original processing result is rapidly identified, and the data processing verification efficiency is improved.
In one embodiment, as depicted in fig. 2, step S3 includes:
step S31, acquiring sample data, processing attributes of the sample data and expected processing results;
step S33, analyzing the processing attribute of the sample data and determining the risk level of the sample data;
step S35, constructing an initial processing model, and training the initial processing model according to the sample data, the processing attribute of the sample data, the risk level and the expected processing result;
step S37, determining the initial processing model as the processing model in the case that the accuracy of the initial processing model is higher than a preset threshold.
In an exemplary embodiment, the risk level may include: a risk level of the data processing, wherein the risk level may include: high risk, medium risk, low risk, in another exemplary embodiment, the risk level may be divided into a plurality of levels, for example, risk levels divided into 1-10 levels.
In an exemplary embodiment, the expected processing results may be obtained by analyzing the data and the processing attributes of the data.
In an exemplary embodiment, training the initial processing model may include: inputting data and processing attributes of the data, analyzing and outputting a prediction processing result by a model, comparing the prediction processing result with an expected processing result, and counting the accuracy of the model, wherein when the accuracy of the model is higher than a preset threshold value, the model is determined to be a processing model. In another exemplary embodiment, training the initial process model may include: inputting data, processing attributes of the data and expected processing results of the data, predicting the predicted processing results of the data by the initial processing model, comparing the predicted processing results with the expected processing results, counting the accuracy of the model, outputting the predicted processing results and the accuracy, and determining the initial processing model as a processing model when the accuracy of the initial processing model is higher than a preset threshold.
In this embodiment, by constructing an initial processing model, analyzing processing attributes of sample data to obtain a risk level of the sample data, and training and updating the initial processing model based on the data, the data attributes, an expected processing result and the risk level, under the condition that accuracy of the initial processing model is ensured, determining the initial processing model as a processing model. The accuracy of the prediction of the processing model is greatly improved, whether the original processing result is correct or not can be rapidly and accurately judged, and the risk level of the data is timely judged, so that convenience is brought to follow-up monitoring of the high risk level data.
In one embodiment, as depicted in fig. 3, step S33 includes:
step S331, obtaining processing attributes of the sample data;
step S332, comparing the processing attribute of the sample data with a preset risk assessment rule to determine a risk level of the sample data; the risk assessment rule comprises an association relation between a processing attribute and a risk level.
In one exemplary embodiment, the risk assessment rule includes: the module where the data is located, the processing object of the data, the processing state of the data and the association relation between the reasons of the data abnormality and the risk level. For example: the module where the data is located is a financial risk of a company, the processing object of the data is a newly added employee, the processing state is incomplete, the reason of the data abnormality generates abnormality for a data source, and the data is high-risk-level data corresponding to the risk assessment rule. In another exemplary embodiment, the risk assessment rule includes: and acquiring the processing attributes of the data, analyzing the processing attributes one by one to determine whether the data belong to the data with high risk level, for example, analyzing the module where the data are located, judging whether the module with abnormal data belongs to the preset attribute of the data with high risk level, if so, determining that the data are the data with high risk level, and if not, analyzing the processing object of the data. Judging whether the processing object of the data is a preset attribute of the high risk level data, if so, determining that the data is the high risk level data, and if not, judging other processing attributes of the data one by one to determine whether the data is the high risk level data.
In this embodiment, by comparing the processing attribute of the sample data with the preset risk evaluation rule, the risk level of the sample data is determined, so that training of the processing model is facilitated, and convenience is provided for subsequent monitoring of high risk data.
In one embodiment, as depicted in fig. 4, step S33 includes:
step S341, obtaining dimension information of the sample data according to a module in which the sample data are located;
step S342, determining an initial risk level of the sample data according to a weight preset by dimension information of the sample data;
step S343, updating the initial risk level according to the processing object, the processing state and the cause of the data abnormality of the sample data, to obtain the risk level of the sample data.
In an exemplary embodiment, the determining the initial risk level of the sample data may include: and determining dimension information influenced by the abnormality of the sample data based on a module in which the sample data is positioned, and determining the initial risk level of the sample data based on the dimension information. For example, the abnormality of the sample data is that the basic information is in error, which directly causes the influence of other dimension data to the user of the data, belongs to important data, has extremely high accuracy requirement, and is high risk level data; legal litigation and management risks contained in some enterprises directly influence the image reputation of the enterprises, bring potential negative influence to the enterprises, and belong to high risk level data; the business information and the development information of the enterprises related to the basic dimension data have information guiding and reference values for the internal management and external investors of the enterprises, and the data are also important and belong to risk level data; the announcement, applet, etc. of the enterprise help to improve brand image, competitive advantage, etc. of the enterprise and produce important effects, belong to the low risk level data; the history information is basic information, risk information and the like of the recorded enterprises, has a reference function to the external public, is not main information, and can be used as low-risk level data.
In an exemplary embodiment, updating the initial risk level may include: and updating the initial risk level by carrying out overall analysis on the processing object, the processing state and the cause of the data abnormality, for example, judging whether the initial risk level of the data is high risk level data, if so, directly determining that the data is high risk level data, and if not, analyzing the processing object, the processing state and the cause of the data abnormality, giving the risk level one by one, and taking the maximum value as the risk level. For example, the processing object is not deeply understood on business logic for newly added staff, and may cause data processing errors, and is determined to be high risk level data; when the data are unprocessed data, the data are high risk level data; the problem is caused by data source errors, and the data is determined to be high risk level data; the data is generated because of error of calculation rules, so that the data can influence downstream data, and the data is determined to be risk level data; the data exception is that the latest message is not disclosed, but an object of the data provides an official information file, and if the data can be updated in time, the data is determined to be low risk level data.
In this embodiment, dimension information of the sample data is obtained by analyzing a module where the sample data is located, a risk level of the sample data is primarily determined according to the dimension information, and the risk level is corrected according to a processing object, a processing state and a cause of data abnormality of the sample data, so that accuracy of the risk level is higher, and convenience is provided for training of a processing model.
In one embodiment, as depicted in fig. 5, step S3 includes:
step S41, the processing model outputs the risk level of the data;
in step S42, in the case that the risk level of the data is higher than the preset threshold, the data with the risk level higher than the preset threshold is monitored.
In an exemplary embodiment, the monitoring the data that the risk level is higher than a preset threshold may include: monitoring data when the risk level of the data is high, in another exemplary embodiment, the monitoring data with the risk level higher than a preset threshold value may include: and lowering the preset threshold value to monitor all data.
In an exemplary embodiment, the monitoring may include: writing the expected processing result of the high-risk level data into a standard data file, and monitoring the high-risk level data by taking the data file as a data interface. Wherein the data file comprises: a database, a table, a key field name, a field content, an enterprise name, a function dimension name, an expected processing result and the like corresponding to the abnormal data.
In this embodiment, the risk level of the data is obtained through the output of the processing model, the high risk level data is determined based on the preset threshold value, and the high risk level data is monitored, so that the diversity of the data processing result verification is improved, and the accuracy of the high risk level data is ensured.
In one embodiment, as depicted in fig. 6, step S42 includes:
step S421, setting expected processing results for data with risk levels higher than a preset threshold;
step S422, obtaining an actual processing result of processing the data with the risk level higher than the preset threshold;
step S423, verifying the actual processing result based on the expected processing result.
In an exemplary embodiment, the expected processing result may be a result of the expected processing of the data, and may be obtained through analysis of a header description of the data, a processing state, a cause of data abnormality, a cause of refusal of processing, and an attachment type.
In an exemplary embodiment, the acquiring the actual processing result of the data with the risk level higher than the preset threshold value may automatically call a data interface through a script program, and automatically determine the actual processing result of the data.
In an exemplary embodiment, the verifying the actual processing result based on the expected processing result may include: judging whether the expected processing result is consistent with the actual processing result, if not, carrying out alarm reminding on abnormal data and processing the data again.
In this embodiment, by setting the expected processing result for the high risk level data, when the actual processing result is different from the expected processing result, the actual processing result is checked, so that the accuracy of the high risk level data is ensured, and the accuracy of data processing is further ensured.
In one embodiment, as illustrated in fig. 7, step S42 includes:
step S431, periodically monitoring the data with the risk level higher than a preset threshold value;
step S432, feeding back the abnormality when the abnormality frequency of the abnormal data processing is greater than a preset threshold value.
In an exemplary embodiment, the feeding back the anomaly in a case where the anomaly frequency of the anomaly data processing is greater than a preset threshold value may include: and regularly detecting abnormal frequencies of the data in the quantitative time, feeding back the abnormal frequencies of the data to a processing object when the abnormal frequencies are too high, performing flow checking on the data, and giving a solution, for example, feeding back information to the related processing object when the abnormal data is frequently abnormal and added with data records or the abnormal data is rolled back.
In this embodiment, by periodically detecting the high risk level data, statistics is performed on the processing of the abnormal frequency of the high risk level data, and the data abnormality is fed back, so that the frequent occurrence of the same data error is reduced, the risk caused by data rollback or abnormal new increase is greatly reduced, and the accuracy of data processing verification is further ensured.
In one embodiment, as illustrated in fig. 8, step S3 includes:
step S6, summarizing the prediction processing result of the data passing through the processing module;
s7, analyzing the prediction processing result of the data, and determining the reason category of the data abnormality;
and S8, determining a processing mode of the data abnormality based on the reason category of the data abnormality.
In an exemplary embodiment, the analyzing the prediction result of the data, and determining the cause category of the data anomaly may include: and comparing and analyzing the data attribute of the data with the prediction result, classifying the processing result, and determining the reason category of the data abnormality.
In an exemplary embodiment, the determining, based on the cause category of the data exception, a processing manner of the data exception may include: and determining solutions of the reason categories through analysis of the reason categories, and setting the solutions in advance. For example, the reason category of the data processing is a data source, which may cause a large amount of data to be abnormal, and the data source needs to be replaced to comprehensively modify the data abnormality caused by the data source; the reason for the data abnormality is that the data under the business rule may be the same abnormality due to incomplete business rule, and the business rule needs to be improved to adjust the data processing; the reason for the data abnormality is that the update of the data is not timely caused, and the business department is informed to increase the update frequency.
In this embodiment, by summarizing the prediction processing results of the data, analyzing the prediction processing results, determining the cause category of the abnormality, and determining the processing mode of the data abnormality according to the cause category, convenience is provided for setting a solution plan in advance for the processing of subsequent data, and the accuracy, timeliness, symmetry and consistency of the data are fundamentally improved.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a verification device for realizing the data processing result of the verification method of the data processing result. The implementation scheme of the solution provided by the device is similar to the implementation scheme described in the above method, so the specific limitation in the embodiments of the verification device for one or more data processing results provided below may refer to the limitation of the verification method for the data processing results hereinabove, and will not be repeated herein.
In one embodiment, as shown in fig. 9, there is provided a verification apparatus 10 of a data processing result, including: an acquisition module 100, a prediction module 300, and a comparison module 500, wherein:
the device comprises an acquisition module for acquiring data and processing attributes of the data, wherein the processing attributes comprise at least one of the following: the method comprises the steps of a module where data is located, a processing object of the data, a processing state of the data, a reason for abnormal data, a reason for refusing to process the data, a type of accessory materials, a related external single number and a description of a title;
the prediction module is used for inputting the data and the processing attribute of the data into a preset processing model and outputting a prediction processing result of the data;
And the comparison module is used for comparing and checking the original processing result of the data based on the prediction processing result.
In one embodiment, there is provided a model training apparatus 20, as shown in fig. 10, comprising: an acquisition sub-module 310, a risk sub-module 330, a training sub-module 350, and a model sub-module 370, wherein:
the acquisition sub-module is used for acquiring sample data, processing attributes of the sample data and expected processing results;
the risk sub-module is used for analyzing the processing attribute of the sample data and determining the risk level of the sample data;
the training sub-module is used for constructing an initial processing model and training the initial processing model according to the sample data, the processing attribute of the sample data, the risk level and the expected processing result;
and the model submodule is used for determining the initial processing model as the processing model under the condition that the accuracy rate of the initial processing model is higher than a preset threshold value.
In one embodiment, there is provided a risk level apparatus 30, as shown in fig. 11, comprising: an acquisition unit 331, a risk unit 332, wherein:
an acquisition unit configured to acquire a processing attribute of the sample data;
The risk unit is used for comparing the processing attribute of the sample data with a preset risk evaluation rule and determining the risk level of the sample data; the risk assessment rule comprises an association relation between a processing attribute and a risk level.
In one embodiment, there is provided a confirmed risk level apparatus 40, as shown in fig. 12, comprising: a dimension unit 341, an initial risk unit 342, an updated risk unit 343, wherein:
the dimension unit is used for obtaining dimension information of the sample data according to a module where the sample data are located;
the initial risk unit is used for determining an initial risk level of the sample data according to the weight preset by the dimension information of the sample data;
and the updating risk unit is used for updating the initial risk level according to the processing object, the processing state and the reason of the data abnormality of the sample data to obtain the risk level of the sample data.
In one embodiment, there is provided a monitoring device 50, as shown in fig. 13, comprising: an output sub-module 410, a monitoring sub-module 420, wherein:
the output sub-module is used for outputting the risk level of the data by the processing model;
And the monitoring sub-module is used for monitoring the data with the risk level higher than the preset threshold value under the condition that the risk level of the data is higher than the preset threshold value.
In one embodiment, there is provided a monitoring sub-device 60, as shown in fig. 14, comprising: an anticipation unit 421, a processing unit 422, a verification unit 423, wherein:
an expected unit for setting an expected processing result for data having a risk level higher than a preset threshold value;
the processing unit is used for obtaining an actual processing result of processing the data with the risk level higher than a preset threshold value;
and the verification unit is used for verifying the actual processing result based on the expected processing result.
In one embodiment, there is provided a monitoring sub-device 70, as shown in fig. 15, comprising: a detection unit 431, a feedback unit 432, wherein:
the detection unit is used for periodically monitoring the data with the risk level higher than a preset threshold value;
and the feedback unit is used for feeding back the abnormality when the abnormality frequency of the abnormal data processing is greater than a preset threshold value.
In one embodiment, there is provided a prediction apparatus 80, as shown in fig. 16, comprising: summary module 600, cause category module 700, processing module 800, wherein:
The summarizing module is used for summarizing the prediction processing result of the data passing through the processing module;
the reason category module is used for analyzing the prediction processing result of the data and determining the reason category of the data abnormality;
and the processing module is used for determining the processing mode of the data abnormality based on the reason category of the data abnormality.
The modules in the verification device for the data processing result can be all or partially realized by software, hardware and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 17. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used to store the process data. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a method of verifying the results of a data process.
It will be appreciated by those skilled in the art that the structure shown in FIG. 17 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.
Claims (12)
1. A method for verifying a data processing result, the method comprising:
acquiring data and processing attributes of the data, wherein the processing attributes comprise at least one of the following: the method comprises the steps of a module where data is located, a processing object of the data, a processing state of the data, a reason for abnormal data, a reason for refusing to process the data, a type of accessory materials, a related external single number and a description of a title;
Inputting the data and the processing attribute of the data into a preset processing model, and outputting the prediction processing result of the data;
and comparing and checking the original processing result of the data based on the prediction processing result.
2. The method of claim 1, wherein the training mode of the process model comprises:
acquiring sample data, processing attributes of the sample data and expected processing results;
analyzing the processing attribute of the sample data and determining the risk level of the sample data;
constructing an initial processing model, and training the initial processing model according to the sample data, the processing attribute of the sample data, the risk level and the expected processing result;
and determining the initial processing model as a processing model under the condition that the accuracy rate of the initial processing model is higher than a preset threshold value.
3. The method of claim 2, wherein analyzing the processing attributes of the sample data to determine a risk level of the sample data comprises:
acquiring processing attributes of the sample data;
comparing the processing attribute of the sample data with a preset risk assessment rule to determine the risk level of the sample data; the risk assessment rule comprises an association relation between a processing attribute and a risk level.
4. The method of claim 2, wherein analyzing the processing attributes of the sample data to determine a risk level of the sample data comprises:
obtaining dimension information of the sample data according to a module in which the sample data are located;
determining an initial risk level of the sample data according to a weight preset by dimension information of the sample data;
and updating the initial risk level according to the processing object, the processing state and the reason of the data abnormality of the sample data to obtain the risk level of the sample data.
5. The method according to claim 1, wherein inputting the data and the processing attribute of the data into a preset processing model, and outputting a predicted processing result of the data, comprises:
the processing model outputs the risk level of the data;
and monitoring the data with the risk level higher than the preset threshold value under the condition that the risk level of the data is higher than the preset threshold value.
6. The method of claim 5, wherein the monitoring the data for which the risk level is above a preset threshold comprises:
Setting expected processing results for data with risk levels higher than a preset threshold value;
acquiring an actual processing result of processing the data with the risk level higher than a preset threshold value;
and verifying the actual processing result based on the expected processing result.
7. The method of claim 6, wherein the method further comprises:
periodically monitoring the data with the risk level higher than a preset threshold value;
and feeding back the abnormality under the condition that the abnormality frequency of the abnormal data processing is greater than a preset threshold value.
8. The method according to claim 1, wherein the method further comprises:
summarizing the prediction processing result of the data passing through the processing module;
analyzing the prediction processing result of the data, and determining the reason category of the data abnormality;
and determining the processing mode of the data abnormality based on the reason category of the data abnormality.
9. A device for verifying a data processing result, the device comprising:
the acquisition module acquires data and processing attributes of the data, wherein the processing attributes comprise: the method comprises the steps of a module where data is located, a processing object of the data, a processing state of the data, a reason for abnormal data, a reason for refusing to process the data, a type of accessory materials, a related external single number and a description of a title;
The prediction module is used for inputting the data and the processing attribute of the data into a preset processing model and outputting a prediction processing result of the data;
and the comparison module is used for comparing and checking the original processing result of the data based on the prediction processing result.
10. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 8 when the computer program is executed.
11. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 8.
12. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311164447.8A CN117131421A (en) | 2023-09-11 | 2023-09-11 | Verification method and device for data processing result and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311164447.8A CN117131421A (en) | 2023-09-11 | 2023-09-11 | Verification method and device for data processing result and computer equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117131421A true CN117131421A (en) | 2023-11-28 |
Family
ID=88852667
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311164447.8A Pending CN117131421A (en) | 2023-09-11 | 2023-09-11 | Verification method and device for data processing result and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117131421A (en) |
-
2023
- 2023-09-11 CN CN202311164447.8A patent/CN117131421A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11249981B2 (en) | Data quality analysis | |
| US11829365B2 (en) | Systems and methods for data quality monitoring | |
| US8984360B2 (en) | Data quality analysis and management system | |
| US20070106577A1 (en) | Apparatus and method for facilitating trusted business intelligence | |
| US20200210401A1 (en) | Proactive automated data validation | |
| EP3686819A1 (en) | Cost analysis system and method for detecting anomalous cost signals | |
| CN116414815A (en) | Data quality detection method, device, computer equipment and storage medium | |
| CN117131421A (en) | Verification method and device for data processing result and computer equipment | |
| JP6142881B2 (en) | Failure notification system, failure notification method, failure notification device, failure notification program, and recording medium | |
| AU2013206466B2 (en) | Data quality analysis and management system | |
| TWI778634B (en) | Method for classifying faults, electronic equipment and storage medium | |
| US20230359597A1 (en) | System and method for reducing data inconsistency after purging client records, in financial institute (fi) databases, when exceeding retention period | |
| CN116610664A (en) | Data monitoring method, device, computer equipment, storage medium and product | |
| CN116414713A (en) | Job processing method, apparatus, computer device, and storage medium | |
| CN116756132A (en) | Big data quality management system and method | |
| CN117785939A (en) | Data analysis method and device based on rule engine and computer equipment | |
| CN117668007A (en) | Data quality detection method and device and electronic equipment | |
| CN116795705A (en) | Abnormal node determination method and device and computer equipment | |
| CN116756032A (en) | Data testing method, device, computer equipment and storage medium | |
| CN117290183A (en) | ETL-based cross-system exception monitoring processing method and device | |
| CN114049217A (en) | Method and device for supervising annuity data and electronic equipment | |
| CN117034036A (en) | Method and device for detecting timeliness of data processing and computer equipment | |
| CN115878597A (en) | Data quality monitoring method and device, computer equipment and storage medium | |
| CN117649239A (en) | Transaction information identification method and device based on generated language model | |
| CN115222171A (en) | Operation assessment method, system, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |