CN117131031A

CN117131031A - Service data quality inspection method and equipment

Info

Publication number: CN117131031A
Application number: CN202311087847.3A
Authority: CN
Inventors: 孔明礼; 张文卓
Original assignee: Beijing Uyu Government Software Co ltd
Current assignee: Beijing Uyu Government Software Co ltd
Priority date: 2023-08-25
Filing date: 2023-08-25
Publication date: 2023-11-28

Abstract

The application relates to the technical field of data processing, in particular to a service data quality inspection method and equipment, wherein the method comprises the following steps: receiving a service data checking instruction; creating auditing rules according to a preset rule template and a business data checking instruction; optimizing the auditing rules according to preset configuration data; counting the current business sub-module, and combining the auditing rules after optimization processing according to the current business sub-module to generate an auditing scheme; executing an auditing scheme, and generating and displaying auditing reports and problem data. According to the technical scheme, the issued business data inspection instruction is intelligently identified, and the inspection rule is generated according to the business data inspection instruction, so that scanning inspection of business data is efficiently completed, the problem data is effectively screened, the quality report is generated, the existing repeated and complicated manual inspection work is omitted, and the work efficiency is improved.

Description

Service data quality inspection method and equipment

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a method and apparatus for checking quality of service data.

Background

The quality of service data inspection is a highly technical and specialized heavy task involving multiple departments. The business data of the budget management integrated system is summarized to a preposed database of a business part through a data exchange platform, and the data is reported in a T+1 mode.

The budget management integrated system is provided with a plurality of service modules, and is in face of lack of analysis of problem data of hundreds of service tables, and cannot automatically generate an error correction scheme, so that the service data is required to be checked and checked one by one manually, and a great deal of time and energy are consumed.

Disclosure of Invention

The application provides a business data quality inspection method and equipment, which aims to overcome the problem that a great deal of time and energy are consumed by checking and checking business data one by one manually in the related technology at least to a certain extent.

The scheme of the application is as follows:

according to a first aspect of an embodiment of the present application, there is provided a service data quality inspection method, including:

receiving a service data checking instruction;

creating auditing rules according to a preset rule template and the business data checking instruction;

optimizing the auditing rules according to preset configuration data;

counting the current business sub-module, and combining the auditing rules after optimization processing according to the current business sub-module to generate an auditing scheme;

executing the auditing scheme, and generating and displaying auditing reports and problem data.

Preferably, the rule template includes: table-level rule templates and data-level rule templates.

Preferably, the table level rule template includes: whole table checking, field type checking and table number checking;

the whole table check comprises checking whether a table field accords with a standard table;

the field type check comprises checking whether the field type and the precision of the table are in accordance with a standard table;

the table number checking includes checking whether the number of data lines in the table meets a preset number of lines requirement and is within a preset number of line interval.

Preferably, the data-level rule template includes:

an integrated rule template and a universal rule template;

the integrated rule template comprises: code normalization test, code set normalization test, non-null value normalization test and check relation normalization test;

the code normalization test comprises: according to the element information and the code set information associated with the table field, checking whether the length of the table field meets the coding specification according to the coding rule of the associated code set;

the code set normalization check includes: according to the element information and the code set information associated with the table field, checking whether the content of the table field is in the value range of the code set;

the non-null normalization check includes: according to metadata information of the table, checking whether the content of the table field is empty or not;

the checking relationship normalization check comprises the following steps: checking the association relation between the tables and the association relation in the tables by writing database language;

the universal rule template comprises: mailbox validity check, phone number validity check, IP address validity check, uniqueness check, length check and range check;

the mailbox validity check includes: checking whether the content of the table field accords with a mailbox format;

the phone number validity check includes: checking whether the contents of the table field conform to the telephone number format;

the validity check of the IP address comprises the following steps: checking whether the content of the table field accords with the IP address format;

the uniqueness check includes: checking whether the content of the table field is unique;

the length check includes: checking whether the length of the table field meets the preset length requirement or not and whether the length of the table field is in the range of a preset length interval or not;

the range check includes: whether the content value of the table field meets the requirement of the preset content value or not is checked, and whether the content value of the table field is in the range of the preset content value interval or not is checked.

Preferably, creating an audit rule according to a preset rule template and the service data checking instruction includes:

analyzing the service data checking instruction to obtain dynamic parameters in the service data checking instruction;

and substituting the dynamic parameters in the business data checking instruction and the corresponding information in the rule template by combining the system context information to create the auditing rule.

Preferably, the preset configuration data includes:

a canonical standard, a corresponding code set under the canonical standard, and an element associated with the code set.

Preferably, the method further comprises:

and executing the auditing scheme at regular time according to the preset scheduling time.

Preferably, generating and presenting audit reports and issue data includes:

generating and displaying audit reports and problem data in different dimensions;

the dimensions include at least: the audit report and the audit scheme, the data table, the catalog, the question field and the rule type of the question data.

Preferably, the method further comprises:

and inputting the audit report and the problem data into a pre-trained error correction model to generate an error correction scheme.

According to a second aspect of an embodiment of the present application, there is provided a service data quality inspection apparatus including:

a processor and a memory;

the processor is connected with the memory through a communication bus:

the processor is used for calling and executing the program stored in the memory;

the memory is configured to store a program at least for executing a service data quality check method according to any one of the above.

The technical scheme provided by the application can comprise the following beneficial effects: the service data quality inspection method in the application comprises the following steps: receiving a service data checking instruction; creating auditing rules according to a preset rule template and a business data checking instruction; optimizing the auditing rules according to preset configuration data; counting the current business sub-module, and combining the auditing rules after optimization processing according to the current business sub-module to generate an auditing scheme; executing an auditing scheme, and generating and displaying auditing reports and problem data. According to the technical scheme, the issued business data inspection instruction is intelligently identified, and the inspection rule is generated according to the business data inspection instruction, so that scanning inspection of business data is efficiently completed, the problem data is effectively screened, the quality report is generated, the existing repeated and complicated manual inspection work is omitted, and the work efficiency is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application as claimed.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.

Fig. 1 is a schematic flow chart of a method for checking quality of service data according to an embodiment of the present application;

fig. 2 is a schematic structural diagram of a service data quality inspection device according to an embodiment of the present application.

Reference numerals: a processor-21; and a memory 22.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the application. Rather, they are merely examples of apparatus and methods consistent with aspects of the application as detailed in the accompanying claims.

Fig. 1 is a flow chart of a method for checking quality of service data according to an embodiment of the present application, referring to fig. 1, a method for checking quality of service data includes:

s11: receiving a service data checking instruction;

s12: creating auditing rules according to a preset rule template and a business data checking instruction;

s13: optimizing the auditing rules according to preset configuration data;

s14: counting the current business sub-module, and combining the auditing rules after optimization processing according to the current business sub-module to generate an auditing scheme;

s15: executing an auditing scheme, and generating and displaying auditing reports and problem data.

It should be noted that, the service data inspection instruction generally refers to a service data inspection instruction issued by an upper-level department, and may be in an SQL (Structured Query Language ) format,

in this embodiment, templates of inspection rules are thinned according to inspection types.

The rule template includes: table-level rule templates and data-level rule templates.

Specifically, the table-level rule template includes: whole table checking, field type checking and table number checking;

the whole table check comprises checking whether the table field accords with the standard table;

The data-level rule template includes:

an integrated rule template and a universal rule template;

the code normalization test comprises the following steps: according to the element information and the code set information associated with the table field, checking whether the length of the table field meets the coding specification according to the coding rule of the associated code set;

the universal rule template includes: mailbox validity check, phone number validity check, IP address validity check, uniqueness check, length check and range check;

The detailed description is as follows:

in the data level rule template, the code normalization test is to check whether the length of the content meets the code normalization according to the coding rule of the code set according to the element information of the table field and the code set information. Such as: whether the length of the unit code corresponds to 3, 6, 12, etc.

The length check is to check whether the content length of the table field is greater than a certain value, less than a certain value, within an interval.

The range check is to check whether the content of the table field is greater than a certain value, less than a certain value, equal to a certain value, in the interval range.

In the table level rule template, the whole table check is to check whether the field of the table meets the standard table, such as: multi-field, few fields, etc.

The field type is the field type of the check list, and whether the precision accords with the standard list, such as: whether the varchar field, length is correct.

The number of table rows is a range of data in the check table, which is larger than a certain value, smaller than a certain value, equal to a certain value.

It should be noted that, creating an audit rule according to a preset rule template and a service data inspection instruction includes:

It should be noted that, the preset configuration data includes:

It can be understood that the technical solution in this embodiment maintains auditing rules in two ways. Firstly, in order to meet the requirements of the superior departments, the custom auditing rules are carried out according to business data checking instructions issued by the superior departments, and secondly, unreasonable parts in the created auditing rules are optimized based on preset configuration data.

In this embodiment, the specification standard is configured by uploading the specification file in advance, then the corresponding code set is configured under the specification standard, and then the configuration element is associated with the code set.

When a service data checking instruction issued by an upper-level department is received, the service data checking instruction is dynamically analyzed, dynamic parameters in the service data checking instruction are obtained, and then the dynamic parameters in the service data checking instruction are replaced with corresponding information in a rule template by combining system context information, so that customized auditing rules are automatically generated according to the requirements of the upper-level department.

It should be noted that, in the technical solution of this embodiment, the current service sub-module is counted, the configured auditing rule of each service sub-module is determined, then the auditing range is configured, and the auditing rules after optimization processing are combined to generate the auditing solution.

It should be noted that the method further includes:

Preferably, the scheduling time can be dynamically configured in the auditing scheme, and the auditing scheme is executed at regular time. The auditing scheme can automatically complete the preset task, so that the time for manually operating is saved, the efficiency is greatly improved, and the time is saved.

It can be understood that the scheduling time in the embodiment is only required to be set once, and the tasks are automatically executed for multiple times, so that the workload of a system maintainer can be greatly reduced, and the method is simple to operate, convenient and easy to implement. And the task execution performed at the timing is more accurate.

It should be noted that, generating and displaying audit report and problem data includes:

It can be understood that, in the technical solution in this embodiment, for the inspection result, the audit report and the problem data are displayed from the dimensions of the audit scheme, the data table, the problem field, the rule type, the catalog and the like. The multidimensional display can enable a manager to know audit reports and problem data more clearly and intuitively.

It should be noted that the method further includes:

It should be noted that, the error correction model in this embodiment is a pre-trained model, and the training data is a history audit report and problem data, and a history error correction scheme corresponding to the history audit report and problem data.

It should be noted that, the error correction model in this embodiment performs updating in real time after generating a new error correction scheme or entering new training data.

It can be understood that the service data quality inspection method in this embodiment includes: receiving a service data checking instruction; creating auditing rules according to a preset rule template and a business data checking instruction; optimizing the auditing rules according to preset configuration data; counting the current business sub-module, and combining the auditing rules after optimization processing according to the current business sub-module to generate an auditing scheme; executing an auditing scheme, and generating and displaying auditing reports and problem data. According to the technical scheme, the issued business data inspection instruction is intelligently identified, and the inspection rule is generated according to the business data inspection instruction, so that scanning inspection of business data is efficiently completed, the problem data is effectively screened, the quality report is generated, the existing repeated and complicated manual inspection work is omitted, and the work efficiency is improved.

It can be understood that, according to the technical scheme in this embodiment, the behavior of the service which is not standard can be found in time by checking the service data. An error correction scheme is also generated for the problem data, and a decision means is provided for the user.

Example two

Fig. 2 is a schematic structural diagram of a service data quality inspection apparatus according to an embodiment of the present application, and referring to fig. 2, a service data quality inspection apparatus includes:

a processor 21 and a memory 22;

the processor 21 is connected to the memory 22 via a communication bus:

wherein the processor 21 is used for calling and executing the program stored in the memory 22;

a memory 22 for storing a program for executing at least one service data quality check method in the above embodiment.

It is to be understood that the same or similar parts in the above embodiments may be referred to each other, and that in some embodiments, the same or similar parts in other embodiments may be referred to.

It should be noted that in the description of the present application, the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Furthermore, in the description of the present application, unless otherwise indicated, the meaning of "plurality" means at least two.

Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and further implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.

It is to be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.

Those of ordinary skill in the art will appreciate that all or a portion of the steps carried out in the method of the above-described embodiments may be implemented by a program to instruct related hardware, where the program may be stored in a computer readable storage medium, and where the program, when executed, includes one or a combination of the steps of the method embodiments.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing module, or each unit may exist alone physically, or two or more units may be integrated in one module. The integrated modules may be implemented in hardware or in software functional modules. The integrated modules may also be stored in a computer readable storage medium if implemented in the form of software functional modules and sold or used as a stand-alone product.

The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, or the like.

In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

While embodiments of the present application have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the application, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the application.

Claims

1. A method for quality of service data inspection, comprising:

receiving a service data checking instruction;

optimizing the auditing rules according to preset configuration data;

2. The method of claim 1, wherein the rule template comprises: table-level rule templates and data-level rule templates.

3. The method of claim 2, wherein the table-level rule template comprises: whole table checking, field type checking and table number checking;

4. The method of claim 2, wherein the data-level rule template comprises:

an integrated rule template and a universal rule template;

5. The method of claim 1, wherein creating auditing rules based on a pre-set rule template and the business data inspection instructions comprises:

6. The method of claim 2, wherein the preset configuration data comprises:

7. The method according to claim 1, wherein the method further comprises:

8. The method of claim 1, wherein generating and presenting audit report and issue data comprises:

9. The method according to claim 1, wherein the method further comprises:

10. A service data quality inspection apparatus, comprising:

a processor and a memory;

the processor is connected with the memory through a communication bus:

the memory for storing a program for performing at least one service data quality check method according to any one of claims 1 to 9.