CN114880312B - Flexibly-set application system service data auditing method - Google Patents
Flexibly-set application system service data auditing method Download PDFInfo
- Publication number
- CN114880312B CN114880312B CN202210538649.3A CN202210538649A CN114880312B CN 114880312 B CN114880312 B CN 114880312B CN 202210538649 A CN202210538649 A CN 202210538649A CN 114880312 B CN114880312 B CN 114880312B
- Authority
- CN
- China
- Prior art keywords
- data
- audited
- service
- result
- auditing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computer Security & Cryptography (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a flexibly-set application system service data auditing method, which comprises the following steps: acquiring a data auditing request, analyzing the data auditing request, and acquiring a data auditing model identifier and service data information to be audited; matching the corresponding data auditing model according to the data auditing model identification, determining a matching result, and verifying the service data to be audited based on the matching result; and acquiring a verification result of the data to be audited, outputting and displaying the verification result based on preset display logic, and determining an output display result. The invention utilizes the advantage of flexible configuration, the service system can call different service data audit models according to the audit model identification, the audit mode is not solidified singly, the service data audit model can be increased or cancelled at any time, and the expansibility and the maintainability of the service system data audit are met.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a flexibly-set service data auditing method for an application system.
Background
At present, a plurality of methods for auditing service data of an application system exist, most auditing methods are still customized, developed and realized according to a single solidified service data auditing model, flexible and flexible setting cannot be realized, and different service data auditing models are called; in practical situations, due to the development of services, service data audit models are continuously increased, and only a single solidified audit model is considered to be difficult to meet the requirements; the invention discloses a method and a device for generating an audit model, which aims at performing customized development on each audit model, can ensure that the audit models do not interfere with each other and run independently, but has high labor cost, large subsequent operation and maintenance workload, low completion efficiency and high code redundancy of an application system, and causes low expansibility and maintainability of the system.
Disclosure of Invention
The invention provides a flexibly-set application system service data auditing method, which is used for solving the problems that the existing data auditing method cannot realize flexible setting and cannot call different service data auditing models.
A flexibly-set service data auditing method for an application system comprises the following steps:
acquiring a data auditing request, analyzing the data auditing request, and acquiring a data auditing model identifier and service data information to be audited;
matching the corresponding data auditing model according to the data auditing model identification, determining a matching result, and verifying the service data to be audited based on the matching result;
and acquiring a verification result of the data to be audited, outputting and displaying the verification result based on preset display logic, and determining an output display result.
As an embodiment of the present invention: the acquiring the data auditing request, analyzing the data auditing request, and acquiring the data auditing model identification and the service data information to be audited, comprises the following steps:
obtaining audited service data information, screening and classifying the audited service data information, and determining primary processing data;
establishing mapping connection between the data audit model in the primary processing data and the corresponding data audit model identification, and determining a mapping connection result; the data auditing model is used for auditing service data information to be audited;
and determining the service data information to be audited according to the data auditing request, and acquiring a data auditing model identifier corresponding to the service data information to be audited based on the mapping connection result.
As an embodiment of the invention: the step of matching the corresponding data auditing model according to the data auditing model identification, determining a matching result and verifying the service data to be audited based on the matching result comprises the following steps:
identifying the data audit model identification, determining an identification result, matching the corresponding data audit model based on the identification result, and acquiring a matching result;
when the matching result shows that the data audit model identification is successfully matched, checking the to-be-audited service data information according to the data audit model based on a preset service processing rule to determine a checking result;
when the verification result shows that the service data information to be audited meets the service processing rule, calculating and processing the service data information to be audited in a data auditing model;
and when the verification result shows that the service data information to be audited does not meet the service processing rule, marking the service data information to be audited and determining a marking result.
As an embodiment of the present invention: the acquiring a verification result of the data to be audited, outputting and displaying the verification result based on preset display logic, and determining an output display result includes:
when the verification result of the data to be audited is compared with a preset business processing rule, and when the comparison result shows that the business processing result of the data to be audited does not accord with the business processing rule, the business data auditing failure is judged, and the failure reason is fed back; wherein the failure reason at least comprises: the data value is too long to exceed the preset maximum value, the identification card number is only limited, the starting is overtime, the data format is inconsistent and the like;
and when the comparison result shows that the service processing result of the data to be audited accords with the service processing rule, judging that the service data is audited successfully, and outputting and displaying the audited service data successfully.
As an embodiment of the present invention: the acquiring the data auditing request, analyzing the data auditing request, and acquiring the data auditing model identification and the service data information to be audited, further comprises:
determining a data structure and a field relation in the service data information to be audited according to the service data information to be audited;
based on a preset data auditing system, carrying out dynamic modeling according to a data structure and a field relation in the auditing service data information, and determining a dynamic modeling result;
performing data analysis on the service data to be audited based on the dynamic modeling result, constructing a class diagram according to the data analysis result, and determining the content attribute in the data to be audited;
and performing data consistency detection on the data to be audited and determining a consistency detection result according to the content attribute of the data to be audited.
As an embodiment of the invention: the acquiring the data auditing request, analyzing the data auditing request, and acquiring the data auditing model identification and the service data information to be audited, further comprises:
performing data integration on the service data to be audited according to physical parameters to obtain an integration result;
when the integration result shows that the physical parameters in the service data to be audited are in a consistent state, extracting quintuple data in the service data to be audited;
comparing quintuple data in a service system with quintuple data in the service data to be audited to obtain a comparison result of the quintuple data, and judging that the quintuple data in the service data to be audited is valid when the comparison result shows that the quintuple data are consistent;
establishing an end-to-end resource tree aiming at the service data to be audited based on the comparison result of the quintuple data, judging whether the end-to-end resource tree is established successfully or not, and determining a judgment result; the resource tree is used for showing the service data to be audited in a resource topology mode;
and when the judgment result shows that the end-to-end resource tree construction is unsuccessful, performing abnormal reason analysis aiming at the end-to-end resource tree construction unsuccessfully to obtain an abnormal reason analysis result.
As an embodiment of the invention: when the judgment result shows that the end-to-end resource tree construction is unsuccessful, performing abnormal reason analysis on the unsuccessful end-to-end resource tree construction to obtain an abnormal reason analysis result, including:
performing abnormity judgment on the service data to be audited according to an auditing scene, and determining an abnormity judgment result; wherein the audit scenario comprises: whether the port occupied by the service data is abnormal or not, whether the data transmission link is complete or not and whether the logic parameters are consistent or not;
automatically outputting corresponding audit alarm according to the abnormal judgment result, and acquiring abnormal reasons of the fault service data according to the audit alarm;
distributing and distributing the fault service data to corresponding data repair ports based on the service system, automatically checking the repaired data according to the fault service data repair result of the data repair ports, analyzing whether the fault service data is recovered to be normal or not, and determining the analysis result.
As an embodiment of the invention: the acquiring the data auditing request, analyzing the data auditing request, and acquiring the data auditing model identification and the service data information to be audited, comprises the following steps:
inputting the service data information to be audited as an input item to a preset artificial intelligent cleaning model for data preprocessing, wherein the artificial intelligent cleaning model is used for acquiring heterogeneous data in the data and cleaning the heterogeneous data; the data preprocessing process comprises the following steps:
carrying out data standardization processing on the service data information to be audited, and determining first processing data; wherein the data normalization process comprises: the method comprises the following steps of performing qualitative standardization processing and quantitative standardization processing, wherein the qualitative standardization processing is used for integrating data of the same field in source data from different business systems, and the quantitative standardization processing is used for classifying and integrating the data according to preset type fields;
sorting according to a preset sorting method according to the first processing data, and determining second processing data;
performing data identification according to the second processing data and preset data characteristics, determining invalid data, and performing removal processing on the invalid data; wherein the invalid data comprises: repeating data and abnormal data.
As an embodiment of the present invention: the sorting processing is performed according to a preset sorting method according to the first processing data, and determining second processing data includes:
judging whether the first processing data has time sequence according to the first processing data, and determining a judgment result;
when the judgment result shows that the first processed data has the time sequence, sequencing according to the input time sequence of the data;
and when the judgment result shows that the first processing data has no time sequence, carrying out data vectorization processing on the first processing data, and sequencing the first processing data by adopting a weak connection graph in a graph theory algorithm.
As an embodiment of the present invention: when the judgment result shows that the end-to-end resource tree construction is unsuccessful, performing abnormal reason analysis on the unsuccessful end-to-end resource tree construction to obtain an abnormal reason analysis result, including:
acquiring the incidence relation between data and events in the service data information to be audited by adopting a data mining technology according to the service data information to be audited;
performing anomaly detection on the service data to be audited based on the incidence relation between the data and the events in the service data to be audited to obtain anomalous data information;
automatically labeling the abnormal data information, and sending the automatically labeled data information to a data restoration port; wherein the automatically labeled information comprises: an exception data type, an exception level, the exception level comprising: primary exception, intermediate exception, high exception.
The invention has the beneficial effects that: in the technical scheme, the flexible and flexible configuration is carried out on the data audit model, so that the audit efficiency of service data audit is improved, the redundancy of codes in an application system is reduced, in addition, the identification of the data audit model is identified, the corresponding audit model is favorably and rapidly selected for audit processing, finally, the audit result is output and displayed, the visualization of the audit data is favorably realized, the integrity and consistency of the data can be realized, and the quality and the safety of the audit data are improved.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and drawings.
The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:
FIG. 1 is a schematic structural flow chart illustrating a flexibly configurable auditing method for business data of an application system according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a framework flow for verifying data in a flexibly-configurable application system service data auditing method according to an embodiment of the present invention;
fig. 3 is a schematic flow chart of a flexibly configurable application system service data auditing method according to an embodiment of the present invention, the method being directed to data auditing requests.
Detailed Description
The preferred embodiments of the present invention will be described in conjunction with the accompanying drawings, and it will be understood that they are described herein for the purpose of illustration and explanation and not limitation.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions, and "plurality" means two or more unless specifically limited otherwise. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Example 1:
the embodiment of the invention provides a flexibly-set service data auditing method for an application system, which comprises the following steps of:
acquiring a data auditing request, analyzing the data auditing request, and acquiring a data auditing model identifier and service data information to be audited;
matching the corresponding data auditing model according to the data auditing model identification, determining a matching result, and verifying the service data to be audited based on the matching result;
acquiring a verification result of the data to be audited, outputting and displaying the verification result based on preset display logic, and determining an output display result;
in one practical scenario: with the continuous improvement of the types and the quantity of the business data, the corresponding business data auditing models are also continuously increased, in order to improve the efficiency of data auditing, each auditing model is subjected to customized development, the auditing models can be ensured not to interfere with each other and operate independently, but the labor cost is high, the subsequent operation and maintenance workload is large, the completion efficiency is low, the code redundancy of an application system is high, and the expansibility and the maintainability of the system are not high;
when the method is implemented, flexible configuration is adopted, the business system selects business data needing to be audited in the application system, the business data are input into the audit model and extracted aiming at the audit model, the system automatically carries out data classification setting by the way, extracts the business data and transmits the business data to the audit model for identification, in the process of identifying the audit model, identification is carried out on the transmitted data audit model identification, the appointed audit model is distributed and the audit processing is carried out, and the result is pushed to the result display part of the audit business data after the processing is finished; in the audit service data result display part, directly outputting results to service data meeting the service processing rules, and feeding back failure reasons such as data values which are too long and exceed a preset maximum value, unique identification card number limitation, startup timeout, inconsistent data formats and the like to the service data which do not meet the service processing rules;
the beneficial effects of the above technical scheme are: in the technical scheme, the flexible and flexible configuration is carried out on the data audit model, so that the audit efficiency of service data audit is improved, the redundancy of codes in an application system is reduced, in addition, the identification of the data audit model is identified, the corresponding audit model is favorably and rapidly selected for audit processing, finally, the audit result is output and displayed, the visualization of the audit data is favorably realized, the integrity and consistency of the data can be realized, and the quality and the safety of the audit data are improved.
Example 2:
in one embodiment, the obtaining the data audit request, analyzing the data audit request, and obtaining the data audit model identifier and the to-be-audited service data information includes:
obtaining audited service data information, screening and classifying the audited service data information, and determining primary processing data;
establishing mapping connection between the data audit model in the primary processing data and the corresponding data audit model identification, and determining a mapping connection result; the data auditing model is used for auditing service data information to be audited;
determining service data information to be audited according to the data auditing request, and acquiring a data auditing model identifier corresponding to the service data information to be audited based on the mapping connection result;
in one practical scenario: when the data auditing model is constructed, the data auditing model is mainly customized by experts in the field according to certain data and manual experience combined in the service data auditing process, so that in the prior art, the generation process of the auditing model depends on manual work, the automatic generation of the auditing model cannot be realized, the accuracy of the auditing model is reduced, and the quantification cannot be obtained;
when the method is implemented, firstly, screening and classification are carried out according to the audited service data, a data auditing model is automatically constructed according to screening and classification results, then, analysis is carried out according to the obtained data to be audited, the model identification of the data to be audited is determined, and a corresponding data auditing model is selected from a series of data auditing models according to the model identification;
the beneficial effects of the above technical scheme are: in the technical scheme, the screened and classified service data are screened and classified, so that the integrity and the uniformity of the data in the application system during data auditing are improved, and the pertinence is improved.
Example 3:
in one embodiment, the matching the corresponding data auditing model according to the data auditing model identification, determining a matching result, and checking the service data to be audited based on the matching result includes:
identifying the data audit model identification, determining an identification result, matching the corresponding data audit model based on the identification result, and acquiring a matching result;
when the matching result shows that the data audit model identification is successfully matched, checking the to-be-audited service data information according to the data audit model based on a preset service processing rule to determine a checking result;
when the verification result shows that the service data information to be audited meets the service processing rule, calculating and processing the service data information to be audited in a data auditing model;
when the verification result shows that the service data information to be audited does not meet the service processing rule, marking the service data information to be audited and determining a marking result;
in one practical scenario: in order to improve the pertinence and the versatility of the data auditing model, at least two candidate auditing rules are set, the auditing rules are respectively used for matching transaction sets and obtaining the support degree of data to be audited, and the like, when various auditing rules appear in an application system, the complexity in the data auditing process is easily improved, the selection of the corresponding auditing rules cannot be automatically managed, and the auditing efficiency and the accuracy of the data auditing model are further reduced;
when the method is implemented, the verified service data information is compared with the preset service processing rule, when the service processing rule is satisfied, the corresponding data auditing model is directly selected through the data auditing identification to carry out the data auditing process, and when the service processing rule is not satisfied, the reason analysis is carried out;
the beneficial effects of the above technical scheme are: according to the technical scheme, the data audit identification is automatically generated by analyzing and identifying the data to be audited, the corresponding data audit model is selected according to the audit identification, the redundancy of system codes is favorably reduced, the expansibility and the maintainability of an application system are favorably improved, in addition, only one rule, namely a business processing rule, is arranged in the data audit process, the calculation complexity in the data audit process is favorably reduced, and the accuracy and the efficiency of the data audit are further improved.
Example 4:
in an embodiment, as shown in fig. 2, the obtaining a verification result of the data to be audited, and performing output display on the verification result based on a preset display logic, and determining an output display result includes:
when the verification result of the data to be audited is compared with a preset business processing rule, and when the comparison result shows that the business processing result of the data to be audited does not accord with the business processing rule, the business data auditing failure is judged, and the failure reason is fed back; wherein the failure reason at least comprises: the data value is too long to exceed the preset maximum value, the identification number is only limited, the starting is overtime, the data format is inconsistent and the like;
when the comparison result shows that the service processing result of the data to be audited accords with the service processing rule, the successful auditing of the service data is judged, and the successfully audited service data is output and displayed;
in one practical scenario: in the process of auditing the data to be audited, the data which do not meet the requirements are often encountered, but the reasons for the non-meeting requirements are unknown for the lack of explanation of the data which do not meet the requirements, so that the efficiency of data auditing is easily reduced;
when the method is implemented, the data which do not meet the preset service processing rule are monitored and tracked in the whole process, and the reasons of non-satisfaction, such as the data value is too long and exceeds the preset maximum value, the unique limitation of the identity card number, the starting overtime, the inconsistent data format and the like, are analyzed, and the feedback reasons show that some data which do not meet the service processing rule are not completely because the data per se do not meet the requirement but are partially limited, so that the data are not met;
the beneficial effects of the above technical scheme are: according to the technical scheme, the reason feedback is carried out according to the data which does not accord with the business processing rule, so that the data auditing efficiency is improved in the secondary auditing process, the consistency of the data in an application system is further realized, the internal relation among the data is determined, and the data quality is improved.
Example 5:
in an embodiment, as shown in fig. 3, the acquiring the data audit request, analyzing the data audit request, and acquiring the data audit model identifier and the to-be-audited service data information further includes:
determining a data structure and a field relationship in the service data information to be audited according to the service data information to be audited;
based on a preset data auditing system, carrying out dynamic modeling according to a data structure and a field relation in the auditing service data information, and determining a dynamic modeling result;
performing data analysis on the service data to be audited based on the dynamic modeling result, constructing a class diagram according to the data analysis result, and determining the content attribute in the data to be audited;
according to the content attribute of the data to be audited, carrying out data consistency detection on the data to be audited and determining a consistency detection result;
the principle of the implementation of the invention is as follows: analyzing data input in an application system, performing consistency check on the data by establishing a relation between an internal structure and fields of the data, dynamically modeling the data according to a consistency check result, inputting, inquiring and visualizing the data, and realizing a high-quality data auditing process through a data threshold value, a field relation in a table and a field relation between tables;
the beneficial effects of the above technical scheme are: according to the technical scheme, the audit model identification is extracted aiming at the service data to be audited, and the corresponding data audit model is selected according to the model identification, so that the visualization of the data audit process is favorably realized, the integrity and consistency of the data are favorably improved, and the data quality and the data safety are improved.
Example 6:
in an embodiment, the obtaining the data audit request, analyzing the data audit request, and obtaining the data audit model identifier and the to-be-audited service data information further includes:
performing data integration on the service data to be audited according to physical parameters to obtain an integration result;
when the integration result shows that the physical parameters in the service data to be audited are in a consistent state, extracting quintuple data in the service data to be audited;
comparing quintuple data in a service system with quintuple data in the service data to be audited to obtain a comparison result of the quintuple data, and judging that the quintuple data in the service data to be audited is valid when the comparison result shows that the quintuple data are consistent;
establishing an end-to-end resource tree aiming at the service data to be audited based on the comparison result of the quintuple data, judging whether the end-to-end resource tree is established successfully or not, and determining a judgment result; the resource tree is used for showing the service data to be audited in a resource topology mode;
when the judgment result shows that the end-to-end resource tree construction is unsuccessful, carrying out abnormal reason analysis on the unsuccessful end-to-end resource tree construction to obtain an abnormal reason analysis result;
the principle of the implementation of the invention is as follows: when service data information in an application system is audited, acquiring, analyzing and integrating the service data information to be audited, constructing an end-to-end resource tree, auditing data of the service data to be audited on the basis, and outputting an alarm for representing problem data;
the beneficial effects of the above technical scheme are: according to the technical scheme, the method for judging whether the data is effective or not by comparing the quintuple data information is beneficial to improving the accuracy of data auditing, improving the speed of data auditing, promoting the intelligence of data management in an application system, improving the quality of the data, realizing the full-flow closed-loop management of intelligent auditing and automatic checking, and keeping the accuracy, the integrity and the consistency of the data in the application system.
Example 7:
in an embodiment, when the determination result shows that the end-to-end resource tree construction is unsuccessful, performing an abnormal reason analysis on the end-to-end resource tree construction, to obtain an abnormal reason analysis result, includes:
performing abnormity judgment on the service data to be audited according to an auditing scene, and determining an abnormity judgment result; wherein the audit scenario comprises: whether the port occupied by the service data is abnormal or not, whether the data transmission link is complete or not and whether the logic parameters are consistent or not;
automatically outputting corresponding audit alarm according to the abnormal judgment result, and acquiring abnormal reasons of the fault service data according to the audit alarm;
distributing and distributing the fault service data to corresponding data recovery ports based on the service system, automatically verifying the recovered data according to the fault service data recovery result of the data recovery ports, analyzing whether the fault service data is recovered to be normal or not, and determining the analysis result;
the principle of the implementation of the invention is as follows: in the technical scheme, the data to be audited is subjected to abnormal judgment aiming at the auditing scene, and due to various reasons causing the abnormality, such as whether a service data occupation port is abnormal or not, whether a data transmission link is complete or not, whether logic parameters are consistent or not and the like, the abnormal judgment is carried out aiming at the abnormality in the technical scheme, corresponding auditing alarms are sent out according to corresponding reasons, fault service data are distributed and distributed to corresponding data repair ports, data repair is carried out aiming at an abnormal report after the data repair ports receive the repairing alarms, the repaired data are input into a data auditing model again, and the repaired data still need to be automatically verified because the data after data repair is not necessarily valid, and the data are audited after the automatic verification result passes;
in a specific embodiment, if the cause of the abnormal data is to be determined, the abnormal data needs to be classified first, and the corresponding abnormal data detection and classification rules correspond to:
wherein i represents a serial number of data in a data set to be audited, error (i) represents an abnormal data function value corresponding to the ith serial number in a data value to be audited, q represents a total data value in the data set to be audited, and d i Representing the difference in the value of the ith data value in the data set to the mean, N i Representing the difference between the ith data value and the corresponding value in the audited data set;
and 2, step: after the abnormal data is detected, considering the situation that false alarm may occur, the false alarm rate of the abnormal data needs to be calculated:
wherein, rate represents the false alarm Rate calculation result for abnormal data, D i Indicating that the ith data is judged to be abnormal data by the step 1, and the data is calculated to be normal data, n 0 Indicating n in the data set to be audited 0 A normal sample value of (a);
and step 3: and (3) predicting abnormal data by constructing a linear combination of Gaussian models, wherein a prediction function is as follows:
wherein M represents the total number of parameters in the Gaussian model, j =1,2, \8230;, M, ω j Represents the weight coefficient corresponding to the jth parameter and satisfies
p j (x; mu, ∑ j) represents the mean value mu j The covariance matrix is a normal probability distribution function of sigma j, and pre represents a prediction function of abnormal data;
the beneficial effects of the above technical scheme are: according to the technical scheme, the abnormal judgment is carried out on the service data to be audited based on the audit scene, so that various abnormal reasons such as whether the service data occupy ports are abnormal or not, whether data transmission links are complete or not, whether logic parameters are consistent or not and the like can be obtained, the automatic monitoring on the data is realized, the data restoration efficiency and the data audit accuracy can be improved, the data quality in the whole application system is improved, and the prediction accuracy is improved by predicting abnormal data according to the Gaussian model.
Example 8:
in one embodiment, the obtaining the data auditing request, analyzing the data auditing request, and obtaining the data auditing model identifier and the to-be-audited service data information includes:
inputting the service data information to be audited as an input item to a preset artificial intelligent cleaning model for data preprocessing, wherein the artificial intelligent cleaning model is used for acquiring heterogeneous data in the data and cleaning the heterogeneous data; the data preprocessing process comprises the following steps:
carrying out data standardization processing on the service data information to be audited, and determining first processing data; wherein the data normalization process comprises: the method comprises the following steps of performing qualitative standardization processing and quantitative standardization processing, wherein the qualitative standardization processing is used for integrating data of the same field in source data from different business systems, and the quantitative standardization processing is used for classifying and integrating the data according to preset type fields;
sorting according to a preset sorting method according to the first processing data, and determining second processing data;
performing data identification according to the second processing data and preset data characteristics, determining invalid data, and performing removal processing on the invalid data; wherein the invalid data comprises: repeating data and abnormal data;
the principle of the implementation of the invention is as follows: the technical scheme includes that heterogeneous data in service data to be audited are preprocessed, full-process optimization of data acquisition, data mining, data conversion and data cleaning is performed, artificial intelligent cleaning and matching model construction of the large data are performed from the perspective of the large data and artificial intelligence, a model in the preprocessing process comprises a cleaning model and a matching model, the cleaning model is applied to cleaning of the large data, the matching model is applied to intelligent matching of the data, due to the existence of the heterogeneous data, standardization of the data is a premise for analyzing all data, meanwhile, the model is also a first module of the cleaning model, when the data are cleaned, a natural language understanding technology and a machine learning mechanism are adopted, the data are sorted after data standardization is completed, a basic sorting algorithm is adopted for sorting of data with time sequence, vectorization processing is performed on the data through the cleaning model for data without time sequence and other categories, the sorting is performed through the principle of a weak communication graph in a graph theory algorithm, invalid data are identified through the data subjected to data standardization and sorting, and are removed, and in addition, the effectiveness of the data is judged through adding a K neighbor algorithm;
in a specific embodiment, aiming at the existence of data islands and heterogeneous data, in order to improve the effectiveness and accuracy of data auditing, data needs to be preprocessed before data auditing is carried out, namely invalid data is removed, wherein the invalid data comprises repeated data and abnormal data,
step 1: aiming at the search of abnormal data, a K neighbor algorithm is adopted, K neighbor points are searched in an audited data set, so that whether the abnormal data exists in the data to be audited is judged, and the calculation formula of the K neighbor algorithm is as follows:
wherein j represents that the data in the data set to be audited which is being judged is valid data, N 0 Representing the set of K data points in the audited data set closest to the data points to be audited, K =1,3,5, \8230, i representing the sequence corresponding to the random data values in the audited data setNumber z i Representing the comparison result of the value of the serial number i corresponding to the audited data set and the corresponding data in the data set to be audited, wherein J () is a judgment function, when the comparison requirement is met, the result output is 1, and when the comparison requirement is not met, the result output is 0;
step 2: when invalid data in the data set to be audited are removed, the matching degree of the data to be audited and the audited data is intelligently monitored by adopting a clustering analysis method, and the matching precision of each iteration is improved through circular iteration, wherein the calculation formula of the matching precision is as follows:
wherein D is 1 ,D 2 ,…，D l Representing the divided group sets in the data to be audited, the data set corresponding to each group, i represents the quantity of the data in all the groups, i belongs to D l ，X(D l ) The method comprises the steps of representing a matching precision calculation function of data in different groups, wherein i' represents data corresponding to i in one group set in a data set to be audited;
the beneficial effects of the above technical scheme are: according to the technical scheme, the cleaning model and the matching model are constructed, automatic processing of data is facilitated, manual participation is not needed in the whole process, data processing is facilitated to accelerate data standardization by adopting natural language understanding, invalid data removing effect is better, intelligent standardized accuracy of the data is improved, in addition, effectiveness of the data is judged through a K adjacent algorithm, efficiency and accuracy of obtaining the invalid data are higher, relevance between the data can be improved after the data are matched, and further efficiency and accuracy of auditing are improved.
Example 9:
in an embodiment, the determining the second processing data by performing sorting processing according to the first processing data by a preset sorting method includes:
judging whether the first processing data has time sequence according to the first processing data, and determining a judgment result;
when the judgment result shows that the first processing data has the time sequence, sequencing according to the input time sequence of the data;
when the judgment result shows that the first processing data has no time sequence, carrying out data vectorization processing on the first processing data, and sequencing the first processing data by adopting a weak communication graph in a graph theory algorithm;
the principle of the implementation of the invention is as follows: after the data standardization is finished, sorting the data with time sequence by adopting a basic sorting algorithm, vectorizing the data without time sequence and other categories by a cleaning model, and sorting the data by adopting the principle of a weak communication graph in a graph theory algorithm;
the beneficial effects of the above technical scheme are: according to the technical scheme, when the data after standardized processing is sequenced, the data are preferentially processed according to the time sequence, so that the validity and the speed of the data in the data auditing process can be improved.
Example 10:
in an embodiment, when the determination result shows that the end-to-end resource tree construction is unsuccessful, performing an abnormal reason analysis on the end-to-end resource tree construction, to obtain an abnormal reason analysis result, includes:
acquiring the incidence relation between data and events in the service data information to be audited by adopting a data mining technology according to the service data information to be audited;
performing anomaly detection on the service data to be audited based on the incidence relation between the data and the events in the service data to be audited to obtain anomalous data information;
automatically labeling the abnormal data information, and sending the automatically labeled data information to a data restoration port; wherein the automatically labeled information comprises: an exception data type, an exception level, the exception level comprising: primary exception, intermediate exception, high exception;
the principle of the implementation of the invention is as follows: according to the technical scheme, a data mining technology is adopted to obtain the incidence relation between data and events in the service data information to be audited according to the service data information to be audited, abnormal detection is carried out on the service data to be audited, abnormal data information is obtained, automatic labeling is carried out on the abnormal data information, and the data information after automatic labeling is sent to a data restoration port;
the beneficial effects of the above technical scheme are: according to the technical scheme, the incidence relation between the data and the events in the service data to be audited is obtained, so that the data auditing efficiency and the auditing accuracy and effectiveness are improved, the auditing errors caused by human factors can be effectively reduced, and the auditing capacity of an application system is improved.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (8)
1. A flexibly-set service data auditing method for an application system is characterized by comprising the following steps:
acquiring a data auditing request, analyzing the data auditing request, and acquiring a data auditing model identifier and service data information to be audited;
matching the corresponding data auditing model according to the data auditing model identification, determining a matching result, and verifying the service data to be audited based on the matching result;
acquiring a verification result of the data to be audited, outputting and displaying the verification result based on preset display logic, and determining an output display result;
the acquiring the data auditing request, analyzing the data auditing request, and acquiring the data auditing model identification and the service data information to be audited, further comprises:
performing data integration on the service data to be audited according to physical parameters to obtain an integration result;
when the integration result shows that the physical parameters in the service data to be audited are in a consistent state, extracting quintuple data in the service data to be audited;
comparing quintuple data in a service system with quintuple data in the service data to be audited to obtain a comparison result of the quintuple data, and judging that the quintuple data in the service data to be audited is valid when the comparison result shows that the quintuple data are consistent;
establishing an end-to-end resource tree aiming at the service data to be audited based on the comparison result of the quintuple data, judging whether the end-to-end resource tree is established successfully or not, and determining a judgment result; wherein the content of the first and second substances,
the resource tree is used for showing the service data to be audited in a resource topology mode;
when the judgment result shows that the end-to-end resource tree construction is unsuccessful, carrying out abnormal reason analysis on the unsuccessful end-to-end resource tree construction to obtain an abnormal reason analysis result;
when the judgment result shows that the end-to-end resource tree construction is unsuccessful, performing abnormal reason analysis on the unsuccessful end-to-end resource tree construction to obtain an abnormal reason analysis result, including:
performing abnormity judgment on the service data to be audited according to an auditing scene, and determining an abnormity judgment result; wherein the content of the first and second substances,
the audit scene comprises the following steps: whether the port occupied by the service data is abnormal or not, whether the data transmission link is complete or not and whether the logic parameters are consistent or not;
automatically outputting a corresponding audit alarm according to the abnormal judgment result, and acquiring the abnormal reason of the fault service data according to the audit alarm;
distributing and distributing the fault service data to corresponding data repair ports based on the service system, automatically checking the repaired data according to the fault service data repair result of the data repair ports, analyzing whether the fault service data is recovered normally, and determining the analysis result;
if the abnormal reason of the abnormal data needs to be judged, firstly, the abnormal data needs to be classified, and the corresponding abnormal data detection and classification rules correspond to:
wherein i represents a serial number of data in a data set to be audited, error (i) represents an abnormal data function value corresponding to the ith serial number in a data value to be audited, q represents a total data value in the data set to be audited, and d i Representing the difference in the value of the ith data value in the data set to the mean, N i Representing the difference between the ith data value and the corresponding value in the audited data set;
and 2, step: after the abnormal data is detected, considering the situation that false alarm may occur, the false alarm rate of the abnormal data needs to be calculated:
wherein, rate represents the false alarm Rate calculation result for abnormal data, D i Indicating that the ith data is judged to be abnormal data by the step 1, and the data is calculated to be normal data, n 0 Indicating n in the data set to be audited 0 A normal sample value of (a);
and step 3: and (3) predicting abnormal data by constructing a linear combination of Gaussian models, wherein a prediction function is as follows:
where M represents the total number of parameters in the gaussian model, j =1,2 j Represents the weight coefficient corresponding to the jth parameter and satisfies
p j (x; mu, ∑ j) represents the mean value mu j The covariance matrix is a normal probability distribution function of Σ j, and pre represents a prediction function for abnormal data.
2. The method as claimed in claim 1, wherein the obtaining the data audit request, analyzing the data audit request, and obtaining the data audit model identifier and the service data information to be audited includes:
obtaining audited service data information, screening and classifying the audited service data information, and determining primary processing data;
establishing mapping connection between the data audit model in the primary processing data and the corresponding data audit model identification, and determining a mapping connection result; wherein, the first and the second end of the pipe are connected with each other,
the data auditing model is used for auditing the service data information to be audited;
and determining the service data information to be audited according to the data auditing request, and acquiring a data auditing model identifier corresponding to the service data information to be audited based on the mapping connection result.
3. The method as claimed in claim 1, wherein the step of matching the corresponding data auditing model according to the data auditing model identification, determining the matching result, and checking the service data to be audited based on the matching result comprises:
identifying the data audit model identification, determining an identification result, matching the corresponding data audit model based on the identification result, and acquiring a matching result;
when the matching result shows that the data audit model identification is successfully matched, checking the to-be-audited service data information according to the data audit model based on a preset service processing rule to determine a checking result;
when the verification result shows that the service data information to be audited meets the service processing rule, calculating and processing the service data information to be audited in a data auditing model;
and when the verification result shows that the service data information to be audited does not meet the service processing rule, marking the service data information to be audited and determining a marking result.
4. The method as claimed in claim 1, wherein the step of obtaining the verification result of the data to be audited, performing output display on the verification result based on a preset display logic, and determining an output display result includes:
when the verification result of the data to be audited is compared with a preset business processing rule, and when the comparison result shows that the business processing result of the data to be audited does not accord with the business processing rule, the business data auditing failure is judged, and the failure reason is fed back; wherein the content of the first and second substances,
the failure reasons at least include: the data value is too long to exceed the preset maximum value, the identification card number is only limited, the starting is overtime, the data format is inconsistent and the like;
and when the comparison result shows that the service processing result of the data to be audited accords with the service processing rule, judging that the service data audit is successful, and outputting and displaying the successfully audited service data.
5. The method as claimed in claim 1, wherein the method for auditing service data of flexibly configurable application systems, where the method for auditing service data by acquiring data audit requests, analyzes the data audit requests, and acquires data audit model identifiers and service data information to be audited, further comprises:
determining a data structure and a field relation in the service data information to be audited according to the service data information to be audited;
based on a preset data auditing system, carrying out dynamic modeling according to a data structure and a field relation in the auditing service data information, and determining a dynamic modeling result;
performing data analysis on the service data to be audited based on the dynamic modeling result, constructing a class diagram according to the data analysis result, and determining the content attribute in the data to be audited;
and performing data consistency detection on the data to be audited and determining a consistency detection result according to the content attribute of the data to be audited.
6. The method as claimed in claim 1, wherein the obtaining the data audit request, analyzing the data audit request, and obtaining the data audit model identifier and the service data information to be audited includes:
inputting the service data information to be audited as an input item into a preset artificial intelligence cleaning model for data preprocessing, wherein,
the artificial intelligent cleaning model is used for acquiring heterogeneous data in the data and cleaning the heterogeneous data;
the data preprocessing process comprises the following steps:
performing data standardization processing on the service data information to be audited to determine first processing data; wherein the data normalization process comprises: the method comprises the following steps of performing qualitative standardization processing and quantitative standardization processing, wherein the qualitative standardization processing is used for integrating the same field data in source data from different business systems, and the quantitative standardization processing is used for classifying and integrating the data according to preset type fields;
sorting according to a preset sorting method according to the first processing data to determine second processing data;
performing data identification according to the second processing data and preset data characteristics, determining invalid data, and performing removal processing on the invalid data; wherein the invalid data comprises: repeating data and abnormal data.
7. The method as claimed in claim 6, wherein the step of determining the second processed data by performing the sorting process according to the first processed data according to the predetermined sorting method comprises:
judging whether the first processing data has a time sequence according to the first processing data, and determining a judgment result;
when the judgment result shows that the first processed data has the time sequence, sequencing according to the input time sequence of the data;
and when the judgment result shows that the first processing data has no time sequence, carrying out data vectorization processing on the first processing data, and sequencing the first processing data by adopting a weak communication graph in a graph theory algorithm.
8. The method as claimed in claim 1, wherein when the determination result indicates that the peer-to-peer resource tree construction is unsuccessful, performing an abnormal cause analysis on the unsuccessful peer-to-peer resource tree construction to obtain an abnormal cause analysis result, the method comprises:
acquiring the incidence relation between data and events in the service data information to be audited by adopting a data mining technology according to the service data information to be audited;
performing anomaly detection on the service data to be audited based on the incidence relation between the data and the events in the service data to be audited to obtain anomalous data information;
automatically labeling the abnormal data information, and sending the automatically labeled data information to a data recovery port; wherein the content of the first and second substances,
the automatically labeled information includes: exception data type, exception level;
the exception level includes: primary exception, intermediate exception, high exception.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210538649.3A CN114880312B (en) | 2022-05-17 | 2022-05-17 | Flexibly-set application system service data auditing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210538649.3A CN114880312B (en) | 2022-05-17 | 2022-05-17 | Flexibly-set application system service data auditing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114880312A CN114880312A (en) | 2022-08-09 |
| CN114880312B true CN114880312B (en) | 2023-02-28 |
Family
ID=82675057
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210538649.3A Active CN114880312B (en) | 2022-05-17 | 2022-05-17 | Flexibly-set application system service data auditing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114880312B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115982665B (en) * | 2023-03-21 | 2023-07-14 | 北京东华博泰科技有限公司 | Quality anomaly auditing method and system for water turbine measurement data |
| CN117235062A (en) * | 2023-11-10 | 2023-12-15 | 广州汇通国信科技有限公司 | Service system data modeling method based on data center |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106971002A (en) * | 2017-04-18 | 2017-07-21 | 北京思特奇信息技术股份有限公司 | A kind of data auditing method and system |
| CN110109900A (en) * | 2018-01-26 | 2019-08-09 | 北京京东尚科信息技术有限公司 | Data auditing method, system, electronic equipment and readable storage medium storing program for executing |
| CN112214532A (en) * | 2020-10-13 | 2021-01-12 | 北京思特奇信息技术股份有限公司 | Service data auditing method and related device |
| CN112669134A (en) * | 2020-12-31 | 2021-04-16 | 山东浪潮通软信息科技有限公司 | Method, equipment and medium for realizing auditing intellectualization through auditing rule machine learning |
| CN114168682A (en) * | 2021-12-06 | 2022-03-11 | 中国铁塔股份有限公司 | Data auditing method, device, equipment and storage medium |
| CN114281807A (en) * | 2021-11-19 | 2022-04-05 | 中国联合网络通信集团有限公司 | Data quality auditing method, device, equipment and storage medium |
| CN114418714A (en) * | 2021-11-26 | 2022-04-29 | 浪潮通信信息系统有限公司 | 5G base station operation and maintenance management system and method |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190228353A1 (en) * | 2018-01-19 | 2019-07-25 | EMC IP Holding Company LLC | Competition-based tool for anomaly detection of business process time series in it environments |
| KR102291869B1 (en) * | 2019-12-31 | 2021-08-19 | 아주대학교산학협력단 | Method and apparatus for anomaly detection of traffic pattern |
-
2022
- 2022-05-17 CN CN202210538649.3A patent/CN114880312B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106971002A (en) * | 2017-04-18 | 2017-07-21 | 北京思特奇信息技术股份有限公司 | A kind of data auditing method and system |
| CN110109900A (en) * | 2018-01-26 | 2019-08-09 | 北京京东尚科信息技术有限公司 | Data auditing method, system, electronic equipment and readable storage medium storing program for executing |
| CN112214532A (en) * | 2020-10-13 | 2021-01-12 | 北京思特奇信息技术股份有限公司 | Service data auditing method and related device |
| CN112669134A (en) * | 2020-12-31 | 2021-04-16 | 山东浪潮通软信息科技有限公司 | Method, equipment and medium for realizing auditing intellectualization through auditing rule machine learning |
| CN114281807A (en) * | 2021-11-19 | 2022-04-05 | 中国联合网络通信集团有限公司 | Data quality auditing method, device, equipment and storage medium |
| CN114418714A (en) * | 2021-11-26 | 2022-04-29 | 浪潮通信信息系统有限公司 | 5G base station operation and maintenance management system and method |
| CN114168682A (en) * | 2021-12-06 | 2022-03-11 | 中国铁塔股份有限公司 | Data auditing method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114880312A (en) | 2022-08-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111209131B (en) | Method and system for determining faults of heterogeneous system based on machine learning | |
| CN114880312B (en) | Flexibly-set application system service data auditing method | |
| WO2021184630A1 (en) | Method for locating pollutant discharge object on basis of knowledge graph, and related device | |
| CN113282461B (en) | Alarm identification method and device for transmission network | |
| CN110704231A (en) | Fault processing method and device | |
| CN113935497A (en) | Intelligent operation and maintenance fault processing method, device and equipment and storage medium thereof | |
| CN106708738B (en) | Software test defect prediction method and system | |
| CN114385391A (en) | NFV virtualization device operation data analysis method and device | |
| CN111782460A (en) | Large-scale log data anomaly detection method and device and storage medium | |
| CN114519524A (en) | Enterprise risk early warning method and device based on knowledge graph and storage medium | |
| CN115809183A (en) | Method for discovering and disposing information-creating terminal fault based on knowledge graph | |
| CN111984442A (en) | Method and device for detecting abnormality of computer cluster system, and storage medium | |
| CN115204536A (en) | Building equipment fault prediction method, device, equipment and storage medium | |
| CN111913824A (en) | Method for determining data link fault reason and related equipment | |
| CN113609393B (en) | Digital platform based on data service and data management | |
| CN113448808B (en) | Method, system and storage medium for predicting single task time in batch processing task | |
| CN114416573A (en) | Defect analysis method, device, equipment and medium for application program | |
| CN116932523B (en) | Platform for integrating and supervising third party environment detection mechanism | |
| CN111062827B (en) | Engineering supervision method based on artificial intelligence mode | |
| CN105138612A (en) | Method and system for analyzing and locating reasons of data consistency difference | |
| CN113723637B (en) | Economic repair level analysis method and system for ship repair system | |
| CN115203014A (en) | Ecological service abnormity restoration system and restoration method based on deep learning | |
| CN111913872A (en) | Software static inspection warning sequencing optimization method based on defect prediction | |
| CN113051307A (en) | Alarm signal analysis method, equipment, storage medium and device | |
| CN109474445B (en) | Distributed system root fault positioning method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |