CN117271553A - Method for generating and operating supervision report data quality rule - Google Patents
Method for generating and operating supervision report data quality rule Download PDFInfo
- Publication number
- CN117271553A CN117271553A CN202311165758.6A CN202311165758A CN117271553A CN 117271553 A CN117271553 A CN 117271553A CN 202311165758 A CN202311165758 A CN 202311165758A CN 117271553 A CN117271553 A CN 117271553A
- Authority
- CN
- China
- Prior art keywords
- rule
- data
- supervision
- generating
- result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012549 training Methods 0.000 claims abstract description 20
- 238000013528 artificial neural network Methods 0.000 claims abstract description 18
- 238000013145 classification model Methods 0.000 claims abstract description 16
- 238000003062 neural network model Methods 0.000 claims abstract description 16
- 238000007781 pre-processing Methods 0.000 claims abstract description 15
- 230000011218 segmentation Effects 0.000 claims description 17
- 238000013507 mapping Methods 0.000 claims description 14
- 239000013598 vector Substances 0.000 claims description 14
- 238000012795 verification Methods 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 10
- 230000014509 gene expression Effects 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 7
- 238000010586 diagram Methods 0.000 description 7
- 238000009960 carding Methods 0.000 description 6
- 210000002569 neuron Anatomy 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000005284 excitation Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000003058 natural language processing Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000001537 neural effect Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000002411 adverse Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000002787 reinforcement Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Computational Biology (AREA)
- Mathematical Physics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a method for generating and operating a supervision report data quality rule, which comprises the following steps: s1, acquiring supervision rule sample data and preprocessing the sample data; s2, constructing a neural network model, and training the neural network by using the preprocessed sample data to obtain a rule classification model; s3, preprocessing the rule data to be processed, inputting a rule classification model, and outputting a classification result; s4, based on the classification result, associating and acquiring corresponding SQL template data from a preset database and generating rule script information; s5, repeatedly executing the steps S3-S4 until rule script information corresponding to all the rule data to be processed is generated; and S6, calculating all the generated rule script information in batches, and transmitting the operation result to the supervision reporting system. Compared with the prior art, the method and the device can efficiently and accurately automatically generate and operate the data quality rule, thereby improving the data reporting quality and ensuring the service stability.
Description
Technical Field
The invention relates to the technical field of computer data processing, in particular to a method for generating and operating a supervision report data quality rule.
Background
With the rapid development of the information age, the business types and the quantity of each financial enterprise are more, and the data quality requirements of each supervision department on the supervision and reporting of the financial enterprises are gradually improved. At present, most technicians of financial enterprises bear verification and configuration work of supervision and report data quality rules, in order to ensure the data quality reported by the enterprises, part of the enterprises specially arrange double persons to write quality rule codes manually to perform cross verification of the quality rules, and the link needs to manually understand supervision rule semantics in depth in advance and then write the codes according to personal understanding for verification; some enterprises adopt regular expression matching to carry out single matching on supervision rules or carry out automatic analysis by using reinforcement learning in similar business scenes so as to ensure the quality of data submitted by supervision.
However, the existing manual and regular expression matching methods have the following disadvantages:
(1) The time consumption for manual carding is long: the supervision rule has large data volume, and the manual carding rule has the problems of long carding time consumption, slow progress, easy error and the like.
(2) And the manual carding cost is high: the supervision rules are various, the complexity is high, and the labor cost for technicians to comb different rules and complete the writing of SQL (Structured Query Language ) codes is high; and the technical capability of different personnel is uneven, and the quality of the result of manual carding cannot be ensured accurately.
(3) Poor adaptability: at present, some enterprises are classified in similar business scenes by adopting a regular matching technology, but regular expressions are multiple in variety and complex in grammar, the possibility of regular multi-layer nesting is high, and good automatic classification cannot be realized for new supervision rules.
It can be said that it is difficult to efficiently and accurately automatically generate and run data quality rules in the prior art, which results in adverse effects on the quality of data delivery and service stability.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provide a method for generating and operating the supervision report data quality rule, which can efficiently and accurately automatically generate and operate the data quality rule, thereby improving the data report quality and ensuring the service stability.
The aim of the invention can be achieved by the following technical scheme: a method for generating and operating a supervision report data quality rule comprises the following steps:
s1, acquiring supervision rule sample data and preprocessing the sample data;
s2, constructing a neural network model, and training the neural network by using the preprocessed sample data to obtain a rule classification model;
s3, preprocessing the rule data to be processed, inputting a rule classification model, and outputting a classification result;
s4, based on the classification result, associating and acquiring corresponding SQL template data from a preset database and generating rule script information;
s5, repeatedly executing the steps S3-S4 until rule script information corresponding to all the rule data to be processed is generated;
and S6, calculating all the generated rule script information in batches, and transmitting the operation result to the supervision reporting system.
Further, the specific process of step S1 is as follows: acquiring supervision rule sample data, and performing illegal character preprocessing on the supervision rule sample data; and then carrying out word segmentation on the sample data, and simultaneously carrying out stop word processing on the word segmentation result.
Further, the step S1 specifically comprises the steps of performing illegal character preprocessing on supervision rule sample data by using a regular expression;
word segmentation is carried out by using a word segmentation method based on dictionary and word bank matching or using a barker word segmentation tool.
Further, the neural network model constructed in the step S2 includes an input layer, a hidden layer and an output layer, wherein the input of the input layer is a feature vector obtained by converting the preprocessed sample data, and the number of nodes of the input layer is greater than or equal to the dimension of the feature vector; each node in the output layer represents a classification result.
Further, the specific process of training the neural network in step S2 is as follows:
initializing neural network parameters;
converting the preprocessed sample data into feature vectors, transmitting the feature vectors to an input layer, and transmitting corresponding known classification labels, namely a given target, to an ideal output unit;
acquiring node output of an output layer and a hidden layer;
calculating the deviation and calculation error of the target value and the actual output so as to update the weight;
judging whether the current neural network model meets the preset precision requirement, if so, stopping training to obtain a rule classification model, and otherwise, returning to continue training.
Further, the database preset in step S4 stores SQL template mapping information corresponding to different supervision rules, specifically, key-value mapping.
Further, the step S4 specifically includes the following steps:
s41, taking the classification result as a key, and connecting database association mapping file data through a database connection pool to obtain corresponding SQL template data;
s42, automatically filling the SQL template data with dynamic parameters to generate complete SQL verification rule script information.
Further, in step S6, all the generated rule script information is submitted to the big data cluster for calculation, so as to obtain the result status information of each rule operation, and when all the result status information is "pass", the current corresponding data information is transmitted to the supervision reporting system.
Further, the big data cluster comprises a distributed computing unit and a distributed storage unit, wherein the distributed computing unit is used for performing distributed computation on the received rule script information; the distributed storage unit is used for storing metadata information.
Further, the step S6 specifically includes the following steps:
s61, storing each generated rule script information into a queue of a thread pool as a single task;
s62, submitting tasks in the queue to a big data cluster in batches through interface call for distributed calculation, and obtaining result data of each rule operation through an interface callback mode after the big data cluster is calculated;
and S63, comparing the result data with preset expected data, wherein if the result data is consistent with the preset expected data, the result data is passed, if the result data is inconsistent with the preset expected data, the result data is failed, and if all the rule running states are passed, the corresponding table data information is uploaded to the supervision reporting system.
Compared with the prior art, the invention has the following advantages:
1. according to the invention, the neural network model is built, the neural network is trained by utilizing the preprocessed supervision rule sample data, so that a rule classification model is obtained, automatic identification and classification can be carried out on supervision rules, rule script information can be automatically generated by combining a mode of acquiring SQL template data in a correlation mode, and then the generated rule script information is automatically operated in a batch calculation mode. The neural network language classification model is supported, and the supervised autonomous learning is carried out through the sample library, so that the manual understanding of massive rule semantics and the participation in the complicated supervision rule code writing are not required, the learning of complex regular expression grammar is also not required, and the labor cost is greatly reduced; and the method can well automatically identify and classify aiming at the new supervision rule, and generate an expected rule script, so that the accuracy of quality rule codes can be greatly improved, and the quality of data reporting and the service stability are ensured.
2. In the invention, the similar SQL sentences are considered, and only the table names, field names or filtering conditions related in the SQL sentences have slight differences, so that the similar SQL sentences are extracted into templates, namely, the supervision rules are classified, then the key value pair mapping is carried out with the corresponding SQL templates, and then the mapping result is stored in a database, so that the one-to-one mapping of the supervision rules and the SQL templates is realized.
3. According to the method, all the generated rule script information is calculated in batches, and in a distributed mode of the big data cluster, manual copying and pasting code execution are not needed, so that index calculation can be quickly carried out on massive report data, the real-time performance of report is improved, and the quality of data report is ensured.
Drawings
FIG. 1 is a schematic flow chart of the method of the present invention;
FIG. 2 is a schematic diagram of an application process of an embodiment;
FIG. 3 is a diagram showing the word segmentation result in the embodiment;
FIG. 4 is a schematic diagram of a neural network model constructed in an embodiment;
FIG. 5 is a schematic diagram of different rule classification labels according to an embodiment;
FIG. 6 is a schematic diagram of a training process for a neural network model;
FIG. 7 is a schematic diagram of a process for generating regular script information;
fig. 8 is a schematic diagram of a system architecture constructed in an embodiment.
Detailed Description
The invention will now be described in detail with reference to the drawings and specific examples.
Examples
As shown in fig. 1, a method for generating and running a quality rule of supervision report data includes the following steps:
s1, acquiring supervision rule sample data and preprocessing the sample data;
s2, constructing a neural network model, and training the neural network by using the preprocessed sample data to obtain a rule classification model;
s3, preprocessing the rule data to be processed, inputting a rule classification model, and outputting a classification result;
s4, based on the classification result, associating and acquiring corresponding SQL template data from a preset database and generating rule script information;
s5, repeatedly executing the steps S3-S4 until rule script information corresponding to all the rule data to be processed is generated;
and S6, calculating all the generated rule script information in batches, and transmitting the operation result to the supervision reporting system.
By applying the technical scheme, as shown in fig. 2, the embodiment mainly includes:
step 1, sample data preprocessing
Performing illegal character preprocessing on a large number of supervision rule sample data by using a regular expression to prevent special characters from generating special samples so as to interfere with the training effect of the model; after the sample data is subjected to regular pretreatment, performing word segmentation on the sample data again, and stopping words and the like on the segmented result; the training data of the model consists of training samples and classification labels.
The key step of the subsequent training of the neural network is to ensure the correctness of the sample, and a good neural network can be trained by using correct sample data, so that the sample data must be preprocessed before training to obtain a good learning sample; according to the scheme, the regular expression is used for carrying out illegal character processing on the text data, so that the phenomenon that special characters generate special samples to interfere with a model training effect is prevented, and illegal characters such as "#", etc. are processed in a preprocessing stage; secondly, based on dictionary and word library matching word segmentation methods, carrying out word segmentation processing on sample data (text data can be directly segmented by using a barking word segmentation tool); the supervision rules may be classified into null value verification, enumeration value verification, length verification, and the like, and the word segmentation result in this embodiment is shown in fig. 3.
In order to improve efficiency, for common words that provide little useful information, such as "what appears in sentences", "has been" and the like, the present embodiment uses subsampling technique to improve training speed and accuracy of word vectors, and a common processing method is to give a threshold parameter t of word frequency, where word w is to be discarded with prob probability, and f (w) is the frequency of w, where the formula is:
step 2, neural network model input
According to the method, a deep learning method is adopted to automatically classify the text, so that all words after word segmentation are required to be changed into feature vectors to serve as input of a neural network model, the number of input neural nodes is adjusted according to the dimension of the feature vectors, and n input layer neurons are set in order to ensure that each sample has corresponding neurons.
Training sample data of the neural network language classification model consists of a sample word stock and classification labels, so that all words in a document are changed into word vectors by using FastTest to be used as input of the neural network model, the design of the number of input layer nodes is required to be set according to the input quantity of an actual problem, the number of the input neural nodes is adjusted according to the dimension of a feature vector, and n input layer neurons are set for ensuring that each sample has corresponding neurons; in order to accelerate the learning process, feature vectors are typically normalized to between 0 and 1 first when they are passed into the input layer.
Step 3, neural network model output
The input layer and the output layer of the neural network are respectively one layer, and the hidden layer can be provided with a plurality of layers; each node in the last layer represents an output node, and each node represents a class, and then the n nodes represent n classes of models, for example, when there are 3 output nodes, there are 3 judgment segments of different classes, and there are 100 output nodes and 100 judgment segments of different classes. The neural network language model constructed in this embodiment is shown in fig. 4, where each node of the neural network output layer represents an output node, each node represents a class, and then the n nodes represent models of n classes. The corresponding class label is shown in fig. 5, assuming that there are fixed value check, null value check, format check, enumerated value check, length check, etc. In addition, the excitation function is arranged at the last layer of the neural network and is used as an output node of the prediction classification, each node represents one classification, and the excitation function of any output node is the same.
The excitation function is specifically:
the training process of the whole neural network model is shown in fig. 6, and a rule classification model is obtained by a self-supervision learning mode.
Step 4, one-to-one mapping of supervision rules and SQL templates
According to the past test experience, the scheme discovers that the SQL sentences corresponding to most supervision rules are similar, and only the table names, field names or filtering conditions related to the SQL sentences have small differences, so that the similar SQL sentences are designed to be extracted into templates, namely the supervision rules are classified, key-value mapping is carried out on the supervision rules and the corresponding SQL templates, and the mapping result is stored in a MySQL database as dynamic configuration, so that manual modification and program identification and calling are facilitated.
Step 5, intelligent processing of the existing supervision rules
And calling the trained neural network language classification model, performing natural language processing on the existing rule data to be processed, automatically returning processed classification result data after the model processing, taking the type calculated by the model as a key, and removing database association mapping file data to obtain corresponding SQL template data.
As shown in fig. 7, the automation program reads the text data of the to-be-processed supervision rule in advance, and then invokes the rule classification model to perform natural language processing on the existing to-be-processed rule data to obtain processed classification result data, for example: the rule data to be processed is 'transaction date is not allowed to be null', after natural language processing, a null value check type is output, the type calculated by the model is used as a key, mySQL database association mapping file data are connected through a database connection pool, corresponding SQL template data are obtained, dynamic parameters such as table names/fields and the like are automatically filled in the SQL template data, and a complete SQL verification rule script code is generated.
Step 6, big data cluster distributed computation and visual display of results
And submitting the generated verification rules to a big data cluster for calculation, and obtaining result state information of each rule operation after calculation, wherein in practical application, the result state information can be displayed on a front page, a user can further intuitively know whether the rule passes or fails according to the state information of the page, and if all rule operation states pass, the table data information corresponding to the rule is uploaded to a supervision and reporting system, so that reporting of supervision data can be completed.
Based on the above scheme, the embodiment builds a system architecture including a local system, a distributed computing platform and a reporting system, as shown in fig. 8, where the local system is configured to execute a process of automatically generating rule script codes, and the distributed computing platform is configured to perform distributed computation on all received rule script codes, and then transmit data information that all rule running state information passes through to the reporting system in a manner of timing scheduling.
When distributed computation is performed, each generated SQL verification rule is stored into a queue of a thread pool as an independent task, tasks in the queue are submitted to a big data cluster in batches through interface calls to perform distributed computation, result data of each rule operation is obtained through an interface callback mode after the big data cluster is computed, the result data is compared with preset expected data, the result data is passed if the comparison is inconsistent, the comparison is failed, the result is displayed on a front web page as state information, a user can further intuitively know whether the rule is passed or not according to the state information of the page, when all rule operation states are passed, the table data information corresponding to the rule is uploaded to a supervision and report system, and as the operation result is displayed in batches on a front page, the user can rapidly and accurately locate whether a certain rule normally passes or not, and complexity of problem investigation is reduced.
In summary, the technical scheme aims to solve the problem of inaccurate quality of reporting of the supervision data, automatically identifies the quality requirement of the reporting data through AI semantic analysis of the supervision quality rule, and then classifies the quality rule. The automatic testing tool can customize the expandable SQL detection templates according to different classifications, automatically generate a quality rule verification script by combining metadata information of the enterprise data warehouse model and the big data platform, schedule computing resources and storage resources of the big data cluster to execute rules and visually display, so that instantaneity of reporting is improved, and quality of data reporting is ensured.
According to the scheme, deep learning is combined with an automatic program, automatic analysis of supervision quality rules and automatic generation of supervision rule codes can be achieved, the generated rules can be directly applied to the supervision and report field, the learning threshold of data report personnel is low, the supervision rule semantics are not required to be understood, and rule combing can be learned in a short time.
According to the scheme, the supervision rule codes are automatically operated in a distributed computing mode, manual copying and pasting code execution are not needed, and index computation can be quickly carried out on mass report data.
The scheme adopts a mode of customizing the classification templates and automatically generating rules, is particularly suitable for scenes with large supervision rules and high rule complexity, and can effectively solve the problems of long time consumption, easy error and the like of the traditional manual carding.
Claims (10)
1. The method for generating and operating the supervision report data quality rule is characterized by comprising the following steps:
s1, acquiring supervision rule sample data and preprocessing the sample data;
s2, constructing a neural network model, and training the neural network by using the preprocessed sample data to obtain a rule classification model;
s3, preprocessing the rule data to be processed, inputting a rule classification model, and outputting a classification result;
s4, based on the classification result, associating and acquiring corresponding SQL template data from a preset database and generating rule script information;
s5, repeatedly executing the steps S3-S4 until rule script information corresponding to all the rule data to be processed is generated;
and S6, calculating all the generated rule script information in batches, and transmitting the operation result to the supervision reporting system.
2. The method for generating and running the supervision report data quality rule according to claim 1, wherein the specific process of step S1 is as follows: acquiring supervision rule sample data, and performing illegal character preprocessing on the supervision rule sample data; and then carrying out word segmentation on the sample data, and simultaneously carrying out stop word processing on the word segmentation result.
3. The method for generating and running the quality rule of the supervision report data according to claim 2, wherein the step S1 is specifically to use a regular expression to perform illegal character preprocessing on the supervision rule sample data;
word segmentation is carried out by using a word segmentation method based on dictionary and word bank matching or using a barker word segmentation tool.
4. The method for generating and operating the quality rule of the supervision report data according to claim 1, wherein the neural network model constructed in the step S2 includes an input layer, a hidden layer and an output layer, the input of the input layer is a feature vector obtained by converting the preprocessed sample data, and the number of nodes of the input layer is greater than or equal to the dimension of the feature vector; each node in the output layer represents a classification result.
5. The method for generating and running the supervision report data quality rule according to claim 4, wherein the specific process of training the neural network in step S2 is as follows:
initializing neural network parameters;
converting the preprocessed sample data into feature vectors, transmitting the feature vectors to an input layer, and transmitting corresponding known classification labels, namely a given target, to an ideal output unit;
acquiring node output of an output layer and a hidden layer;
calculating the deviation and calculation error of the target value and the actual output so as to update the weight;
judging whether the current neural network model meets the preset precision requirement, if so, stopping training to obtain a rule classification model, and otherwise, returning to continue training.
6. The method for generating and running the quality rule of the supervision report data according to claim 1, wherein the database preset in step S4 stores SQL template mapping information corresponding to different supervision rules, specifically key-value mapping.
7. The method for generating and running the supervision report data quality rule according to claim 6, wherein the step S4 specifically includes the following steps:
s41, taking the classification result as a key, and connecting database association mapping file data through a database connection pool to obtain corresponding SQL template data;
s42, automatically filling the SQL template data with dynamic parameters to generate complete SQL verification rule script information.
8. The method for generating and running the quality rule of the supervision and reporting data according to claim 1, wherein the step S6 is specifically to submit all the generated rule script information to a big data cluster for calculation, so as to obtain the result state information of each rule operation, and when all the result state information is "pass", the current corresponding data information is transmitted to the supervision and reporting system.
9. The method for generating and running the supervision report data quality rule according to claim 8, wherein the big data cluster comprises a distributed computing unit and a distributed storage unit, and the distributed computing unit is used for performing distributed computation on the received rule script information; the distributed storage unit is used for storing metadata information.
10. The method for generating and running the supervision report data quality rule according to claim 8, wherein the step S6 specifically includes the following steps:
s61, storing each generated rule script information into a queue of a thread pool as a single task;
s62, submitting tasks in the queue to a big data cluster in batches through interface call for distributed calculation, and obtaining result data of each rule operation through an interface callback mode after the big data cluster is calculated;
and S63, comparing the result data with preset expected data, wherein if the result data is consistent with the preset expected data, the result data is passed, if the result data is inconsistent with the preset expected data, the result data is failed, and if all the rule running states are passed, the corresponding table data information is uploaded to the supervision reporting system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311165758.6A CN117271553A (en) | 2023-09-08 | 2023-09-08 | Method for generating and operating supervision report data quality rule |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311165758.6A CN117271553A (en) | 2023-09-08 | 2023-09-08 | Method for generating and operating supervision report data quality rule |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117271553A true CN117271553A (en) | 2023-12-22 |
Family
ID=89215169
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311165758.6A Pending CN117271553A (en) | 2023-09-08 | 2023-09-08 | Method for generating and operating supervision report data quality rule |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117271553A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109408526A (en) * | 2018-10-12 | 2019-03-01 | 平安科技（深圳）有限公司 | SQL statement generation method, device, computer equipment and storage medium |
| CN109542929A (en) * | 2018-11-28 | 2019-03-29 | 山东工商学院 | Voice inquiry method, device and electronic equipment |
| CN114416987A (en) * | 2022-01-14 | 2022-04-29 | 平安普惠企业管理有限公司 | Text classification method, device and equipment based on artificial intelligence and storage medium |
| US20220147814A1 (en) * | 2020-11-09 | 2022-05-12 | Moore & Gasperecz Global Inc. | Task specific processing of regulatory content |
-
2023
- 2023-09-08 CN CN202311165758.6A patent/CN117271553A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109408526A (en) * | 2018-10-12 | 2019-03-01 | 平安科技（深圳）有限公司 | SQL statement generation method, device, computer equipment and storage medium |
| CN109542929A (en) * | 2018-11-28 | 2019-03-29 | 山东工商学院 | Voice inquiry method, device and electronic equipment |
| US20220147814A1 (en) * | 2020-11-09 | 2022-05-12 | Moore & Gasperecz Global Inc. | Task specific processing of regulatory content |
| CN114416987A (en) * | 2022-01-14 | 2022-04-29 | 平安普惠企业管理有限公司 | Text classification method, device and equipment based on artificial intelligence and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11676067B2 (en) | System and method for creating data to train a conversational bot | |
| US11948058B2 (en) | Utilizing recurrent neural networks to recognize and extract open intent from text inputs | |
| CN109471793B (en) | Webpage automatic test defect positioning method based on deep learning | |
| CN110580308B (en) | Information auditing method and device, electronic equipment and storage medium | |
| US12026471B2 (en) | Automated generation of chatbot | |
| US11016740B2 (en) | Systems and methods for virtual programming by artificial intelligence | |
| CN114547072A (en) | Method, system, equipment and storage medium for converting natural language query into SQL | |
| CN112163424A (en) | Data labeling method, device, equipment and medium | |
| CN113157859B (en) | Event detection method based on upper concept information | |
| CN112036168B (en) | Event main body recognition model optimization method, device, equipment and readable storage medium | |
| CN116775183A (en) | Task generation method, system, equipment and storage medium based on large language model | |
| CN113807973B (en) | Text error correction method, apparatus, electronic device and computer readable storage medium | |
| CN115964273A (en) | Spacecraft test script automatic generation method based on deep learning | |
| US11921764B2 (en) | Utilizing artificial intelligence models to manage and extract knowledge for an application or a system | |
| CN116304748A (en) | Text similarity calculation method, system, equipment and medium | |
| CN112685374B (en) | Log classification method and device and electronic equipment | |
| CN115510188A (en) | Text keyword association method, device, equipment and storage medium | |
| EP4302244A1 (en) | Explainable artificial intelligence in computing environment | |
| CN116361147A (en) | Method for positioning root cause of test case, device, equipment, medium and product thereof | |
| CN114722198A (en) | Method, system and related device for determining product classification code | |
| CN118132305A (en) | Log analysis method, device, equipment, medium and program product | |
| CN117592470A (en) | Low-cost gazette data extraction method driven by large language model | |
| CN112035668A (en) | Event subject recognition model optimization method, device and equipment and readable storage medium | |
| CN117271553A (en) | Method for generating and operating supervision report data quality rule | |
| CN114328936A (en) | Method and device for establishing classification model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |