CN112783882A - Big data quality inspection method, system, storage medium and equipment - Google Patents
Big data quality inspection method, system, storage medium and equipment Download PDFInfo
- Publication number
- CN112783882A CN112783882A CN202110085577.7A CN202110085577A CN112783882A CN 112783882 A CN112783882 A CN 112783882A CN 202110085577 A CN202110085577 A CN 202110085577A CN 112783882 A CN112783882 A CN 112783882A
- Authority
- CN
- China
- Prior art keywords
- quality inspection
- index data
- data
- inspection index
- rule
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007689 inspection Methods 0.000 title claims abstract description 371
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000005070 sampling Methods 0.000 claims description 31
- 238000012216 screening Methods 0.000 claims description 26
- 238000004590 computer program Methods 0.000 claims description 6
- 238000003908 quality control method Methods 0.000 claims 3
- 238000004364 calculation method Methods 0.000 abstract description 17
- 230000009286 beneficial effect Effects 0.000 description 6
- 238000013215 result calculation Methods 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 239000002699 waste material Substances 0.000 description 2
- 230000007812 deficiency Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24564—Applying rules; Deductive queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06395—Quality analysis or management
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Entrepreneurship & Innovation (AREA)
- Economics (AREA)
- Strategic Management (AREA)
- Development Economics (AREA)
- Educational Administration (AREA)
- Databases & Information Systems (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Operations Research (AREA)
- Marketing (AREA)
- Game Theory and Decision Science (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- General Factory Administration (AREA)
Abstract
The invention relates to a big data quality inspection method, a system, a storage medium and equipment, wherein the method comprises the steps of reading service data information in a database, calculating quality inspection index data according to the service data information and a predefined rule, and constructing a quality inspection index data pool; determining a corresponding quality inspection rule according to the quality inspection target task, and calculating a quality inspection result according to the quality inspection rule and the quality inspection index data; and directly comparing the quality inspection result with a corresponding preset quality inspection threshold value, and generating early warning information when the quality inspection result and the preset quality inspection threshold value trigger an early warning rule. According to the quality inspection method and the quality inspection system, the quality inspection index data are calculated in advance according to the service data information and the predefined rule, and the quality inspection index data pool is constructed, so that the quality inspection result can be calculated by directly adopting the data in the quality inspection index data pool, the workload of repeated calculation is greatly reduced, the data in the quality inspection index data pool can be shared, the expansibility is strong, the flexibility and the convenience are realized, the quality inspection efficiency is improved, and the quality inspection result can be directly returned, so that the convenience and the intuition are realized.
Description
Technical Field
The invention relates to the technical field of big data, in particular to a big data quality inspection method, a big data quality inspection system, a big data quality inspection storage medium and big data quality inspection equipment.
Background
In the prior art, a quality inspection method for big data generally includes that a quality inspection request is sent by a structure client, then a corresponding data quality inspection scheme is obtained according to an inspection task identifier carried by the quality inspection request, then a parameter value of an inspection parameter carried by the quality inspection request is utilized to analyze the data quality inspection scheme to generate a data quality inspection instruction executable by a big data platform, then the data quality inspection instruction is sent to the big data platform, and finally a data quality inspection result returned after the big data platform executes the data quality inspection instruction is received. The quality inspection method of big data mainly has the following problems and pain points: 1. data quality inspection schemes need to be configured in advance, and a plurality of quality inspection schemes can have the condition that the same field of the same table needs to be checked, so that the inspection results of the schemes cannot be shared, repeated calculation exists, resource waste is caused, and if the schemes need to be modified, the schemes are troublesome and are not convenient to expand; 2. the background needs SQL sentences generated through quality inspection, the same problems exist, the same object to be detected appears in different sentences, the detection granularity is too coarse, and resources cannot be shared; 3. the quality inspection result cannot be returned in real time, and the larger the data volume is, the slower the time for returning the result is; there is also a problem of resource waste.
Disclosure of Invention
The present invention provides a method, a system, a storage medium and a device for quality inspection of big data, aiming at the above-mentioned deficiencies of the prior art.
The technical scheme for solving the technical problems is as follows: a big data quality inspection method comprises the following steps:
s1: reading service data information in a database, calculating quality inspection index data according to the service data information and a predefined rule, and constructing a quality inspection index data pool;
s2: determining a corresponding quality inspection rule according to a quality inspection target task, and calculating a quality inspection result according to the quality inspection rule and quality inspection index data in a quality inspection index data pool;
s3: and directly comparing the quality inspection result with a corresponding preset quality inspection threshold value, and generating early warning information when the quality inspection result and the preset quality inspection threshold value trigger an early warning rule.
According to the big data quality inspection method, the quality inspection index data is calculated in advance according to the service data information and the predefined rule, and the quality inspection index data pool is constructed, so that the quality inspection result can be calculated by directly adopting the quality inspection index data in the quality inspection index data pool for a specific quality inspection task subsequently, the method is plug and play, the process is simple, the repeated calculation workload is greatly reduced, the quality inspection work difficulty is obviously reduced, the data in the quality inspection index data pool can be shared, the expansibility is strong, flexibility and convenience are realized, the quality inspection efficiency is improved, the performance is not influenced along with the increase of the data amount, and the quality inspection result can be directly returned, so that the method is convenient and visual.
On the basis of the technical scheme, the invention can be further improved as follows:
further: before the step S2, the method further includes the following steps:
and screening the quality inspection index data in the quality inspection index data pool according to a quality inspection target task, and marking the screened quality inspection index data.
The beneficial effects of the further scheme are as follows: by screening and marking the quality inspection index data in the quality inspection index data pool, the corresponding required quality inspection index data can be conveniently screened out according to the mark aiming at a specific quality inspection task, so that the quality inspection result is quickly calculated, the quality inspection efficiency is improved, and the system performance is favorably optimized.
Further: the specific method for screening the quality inspection index data in the quality inspection index data pool according to the quality inspection target task comprises the following steps:
and determining the type of target service data according to the quality inspection target task, and screening quality inspection index data of which the data type is matched with the type of the target quality inspection index data from the quality inspection index data pool according to the type of the target quality inspection index data.
The beneficial effects of the further scheme are as follows: by determining the data type of the quality inspection target task, quality inspection index data with the data type matched with the target quality inspection index data type can be accurately screened from the quality inspection index data pool, and the quality inspection index data are quality inspection index data required by the current quality inspection target task, so that the quality inspection result calculation of the current quality inspection target task can be conveniently completed.
Further: in step S1, the step of calculating quality inspection index data according to the service data information and the predefined rule specifically includes the following steps:
s11: judging the data volume of the service data information, and determining a sampling proportion according to the data volume of the service data information;
s12: sampling the service data information according to the sampling proportion, and calculating quality inspection index data according to the service data information obtained by sampling and a predefined rule.
The beneficial effects of the further scheme are as follows: by sampling in a corresponding proportion according to the data volume of the service data information, the calculation amount of quality inspection index data can be reduced, the calculation efficiency is improved, and the system performance is ensured.
The invention also provides a big data quality inspection system, which comprises a quality inspection index data pool module, a quality inspection rule engine module and an early warning module;
the quality inspection index data pool module is used for reading the service data information in the database, calculating quality inspection index data according to the service data information and a predefined rule, and constructing a quality inspection index data pool;
the quality inspection rule engine module is used for determining a corresponding quality inspection rule according to a quality inspection target task and calculating a quality inspection result according to the quality inspection rule and quality inspection index data in the quality inspection index data pool;
and the early warning module is used for directly comparing the quality inspection result with a corresponding preset quality inspection threshold value and generating early warning information when the quality inspection result and the preset quality inspection threshold value trigger an early warning rule.
According to the big data quality inspection system, the quality inspection index data is calculated in advance according to the service data information and the predefined rule, and the quality inspection index data pool is constructed, so that the quality inspection result can be calculated by directly adopting the quality inspection index data in the quality inspection index data pool for a specific quality inspection task subsequently, the system is plug and play, the process is simple, the repeated calculation workload is greatly reduced, the quality inspection work difficulty is obviously reduced, the data in the quality inspection index data pool can be shared, the expansibility is strong, the system is flexible and convenient, the quality inspection efficiency is improved, the performance is not influenced along with the increase of the data amount, and the quality inspection result can be directly returned, so that the system is convenient and visual.
On the basis of the technical scheme, the invention can be further improved as follows:
further: the big data quality inspection system also comprises a screening and marking module which is used for screening the quality inspection index data in the quality inspection index data pool according to the quality inspection target task and marking the screened quality inspection index data.
The beneficial effects of the further scheme are as follows: by screening and marking the quality inspection index data in the quality inspection index data pool, the corresponding required quality inspection index data can be conveniently screened out according to the mark aiming at a specific quality inspection task, so that the quality inspection result is quickly calculated, the quality inspection efficiency is improved, and the system performance is favorably optimized.
Further: the screening and marking module screens the quality inspection index data in the quality inspection index data pool according to the quality inspection target task, and the screening and marking module specifically realizes the following steps:
and determining the type of target service data according to the quality inspection target task, and screening quality inspection index data of which the data type is matched with the type of the target quality inspection index data from the quality inspection index data pool according to the type of the target quality inspection index data.
The beneficial effects of the further scheme are as follows: by determining the data type of the quality inspection target task, quality inspection index data with the data type matched with the target quality inspection index data type can be accurately screened from the quality inspection index data pool, and the quality inspection index data are quality inspection index data required by the current quality inspection target task, so that the quality inspection result calculation of the current quality inspection target task can be conveniently completed.
Further: the quality inspection index data pool module calculates quality inspection index data according to the service data information and the predefined rule, and the specific implementation is as follows:
judging the data volume of the service data information, and determining a sampling proportion according to the data volume of the service data information;
sampling the service data information according to the sampling proportion, and calculating quality inspection index data according to the service data information obtained by sampling and a predefined rule.
The beneficial effects of the further scheme are as follows: by sampling in a corresponding proportion according to the data volume of the service data information, the calculation amount of quality inspection index data can be reduced, the calculation efficiency is improved, and the system performance is ensured.
The invention also provides a computer readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the big data quality inspection method.
The invention also provides big data quality inspection equipment which comprises the storage medium and a processor, wherein the processor realizes the steps of the big data quality inspection method when executing the computer program on the storage medium.
Drawings
FIG. 1 is a schematic flow chart illustrating a big data quality inspection method according to an embodiment of the present invention;
fig. 2 is a block diagram of a big data quality inspection system according to an embodiment of the present invention.
Detailed Description
The principles and features of this invention are described below in conjunction with the following drawings, which are set forth by way of illustration only and are not intended to limit the scope of the invention.
As shown in fig. 1, a big data quality inspection method includes the following steps:
s1: reading service data information in a database, calculating quality inspection index data according to the service data information and a predefined rule, and constructing a quality inspection index data pool;
s2: determining a corresponding quality inspection rule according to a quality inspection target task, and calculating a quality inspection result according to the quality inspection rule and quality inspection index data in a quality inspection index data pool;
s3: and directly comparing the quality inspection result with a corresponding preset quality inspection threshold value, and generating early warning information when the quality inspection result and the preset quality inspection threshold value trigger an early warning rule.
According to the big data quality inspection method, the quality inspection index data is calculated in advance according to the service data information and the predefined rule, and the quality inspection index data pool is constructed, so that the quality inspection result can be calculated by directly adopting the quality inspection index data in the quality inspection index data pool for a specific quality inspection task subsequently, the method is plug and play, the process is simple, the repeated calculation workload is greatly reduced, the quality inspection work difficulty is obviously reduced, the data in the quality inspection index data pool can be shared, the expansibility is strong, flexibility and convenience are realized, the quality inspection efficiency is improved, the performance is not influenced along with the increase of the data amount, and the quality inspection result can be directly returned, so that the method is convenient and visual.
Optionally, in one or more embodiments of the present invention, before the step S2, the method further includes the following steps:
and screening the quality inspection index data in the quality inspection index data pool according to a quality inspection target task, and marking the screened quality inspection index data.
By screening and marking the quality inspection index data in the quality inspection index data pool, the corresponding required quality inspection index data can be conveniently screened out according to the mark aiming at a specific quality inspection task, so that the quality inspection result is quickly calculated, the quality inspection efficiency is improved, and the system performance is favorably optimized.
Specifically, in one or more embodiments of the present invention, the specific method for screening the quality inspection index data in the quality inspection index data pool according to the quality inspection target task is as follows:
and determining the type of target service data according to the quality inspection target task, and screening quality inspection index data of which the data type is matched with the type of the target quality inspection index data from the quality inspection index data pool according to the type of the target quality inspection index data.
By determining the data type of the quality inspection target task, quality inspection index data with the data type matched with the target quality inspection index data type can be accurately screened from the quality inspection index data pool, and the quality inspection index data are quality inspection index data required by the current quality inspection target task, so that the quality inspection result calculation of the current quality inspection target task can be conveniently completed.
In one or more embodiments of the present invention, in step S1, the calculating quality inspection index data according to the service data information and the predefined rule specifically includes the following steps:
s11: judging the data volume of the service data information, and determining a sampling proportion according to the data volume of the service data information;
s12: sampling the service data information according to the sampling proportion, and calculating quality inspection index data according to the service data information obtained by sampling and a predefined rule.
By sampling in a corresponding proportion according to the data volume of the service data information, the calculation amount of quality inspection index data can be reduced, the calculation efficiency is improved, and the system performance is ensured.
In practice, the data amount and the sampling ratio of the service data information can be flexibly adjusted according to actual conditions. It should be noted that, in order to ensure the uniformity of the data and not to substantially affect the calculation result, the service data information of the same data type is sampled according to the sampling ratio.
The big data quality inspection method of the present invention will be explained below by taking the purchase cost of a product as an example. The total cost of purchasing commodities is commodity purchasing cost (a) + transportation cost (b) + inventory cost (c) + interest cost (financing) (d) + management cost (e) + insurance cost (f) + other cost (g), wherein the total cost of purchasing commodities is purchasing amount (a1) × purchasing unit price (a 2).
In the embodiment of the invention, the data volume of the service data information exceeds 30 ten thousand, which is relatively large, sampling is carried out according to the proportion of 15:1, then the data volume of the obtained service data information exceeds 2 ten thousand, and then the quality inspection index data is calculated according to the service data information and the predefined rule, as follows:
a total number of purchases (a1) of the item in the month;
the transportation cost (b) is the total transportation cost of the purchased commodities in the month;
a purchase unit price (a2) which is an average price for purchasing the commodity in the month;
the inventory cost (c) is the total inventory cost of all the commodities in the month;
interest charge (d) total interest financed in the month;
the management cost (e) is the total cost generated by purchasing the commodities in the month;
insurance cost (f): the total cost of purchasing commodity insurance in the same month;
and the other cost (g) is the total cost of other purchased commodities in the month.
And after the quality inspection index data is calculated, a quality inspection index data pool can be constructed. And then, determining a corresponding quality inspection rule according to the quality inspection target task, and calculating a quality inspection result according to the quality inspection rule and the quality inspection index data.
Specifically, the early warning rule is set for the total purchase expense
(1) The total purchase cost cm ═ a1 a2+ b + c + d + e + f + g > con (constant, custom);
description of the drawings: and if the total purchasing cost cm is greater than a preset purchasing cost quality inspection threshold con, early warning is given.
(2) Stock cost (cd) ═ c > con1 (constant, custom);
description of the drawings: if the inventory cost cm is larger than a preset inventory cost quality inspection threshold con1, early warning is given;
(3) logistics and inventory costs (cld) c + b > con2 (constant, custom);
description of the drawings: if the logistics and inventory fees cld > the preset logistics and inventory fees quality inspection threshold con2, early warning is given;
(4) commodity procurement cost (cp) a1 a2> con3 (constant, custom)
Description of the drawings: and if the commodity purchasing charge cp > the preset commodity purchasing charge quality inspection threshold con2, early warning is given.
In practice, corresponding quality inspection rules are determined for different quality inspection tasks, and then quality inspection results are calculated according to the quality inspection rules and quality inspection index data, so that quality inspection of large data is completed, repeated calculation from the source of the large data in a database is not needed, quality inspection index data in a quality inspection index data pool are directly adopted, and calculation of the quality inspection results can be completed very quickly through the quality inspection index data matched with the corresponding data types by combining with the target service data types determined by the corresponding quality inspection target tasks.
As shown in fig. 2, the present invention further provides a big data quality inspection system, which includes a quality inspection index data pool module, a quality inspection rule engine module and an early warning module;
the quality inspection index data pool module is used for reading the service data information in the database, calculating quality inspection index data according to the service data information and a predefined rule, and constructing a quality inspection index data pool;
the quality inspection rule engine module is used for determining a corresponding quality inspection rule according to a quality inspection target task and calculating a quality inspection result according to the quality inspection rule and quality inspection index data in the quality inspection index data pool;
and the early warning module is used for directly comparing the quality inspection result with a corresponding preset quality inspection threshold value and generating early warning information when the quality inspection result and the preset quality inspection threshold value trigger an early warning rule.
According to the big data quality inspection system, the quality inspection index data is calculated in advance according to the service data information and the predefined rule, and the quality inspection index data pool is constructed, so that the quality inspection result can be calculated by directly adopting the quality inspection index data in the quality inspection index data pool for a specific quality inspection task subsequently, the system is plug and play, the process is simple, the repeated calculation workload is greatly reduced, the quality inspection work difficulty is obviously reduced, the data in the quality inspection index data pool can be shared, the expansibility is strong, the system is flexible and convenient, the quality inspection efficiency is improved, the performance is not influenced along with the increase of the data amount, and the quality inspection result can be directly returned, so that the system is convenient and visual.
Optionally, in one or more embodiments of the present invention, the big data quality inspection system further includes a screening and marking module, configured to screen the quality inspection index data in the quality inspection index data pool according to a quality inspection target task, and mark the screened quality inspection index data.
By screening and marking the quality inspection index data in the quality inspection index data pool, the corresponding required quality inspection index data can be conveniently screened out according to the mark aiming at a specific quality inspection task, so that the quality inspection result is quickly calculated, the quality inspection efficiency is improved, and the system performance is favorably optimized.
Specifically, in one or more embodiments of the present invention, the screening and labeling module, according to a quality inspection target task, specifically implements the following steps of:
and determining the type of target service data according to the quality inspection target task, and screening quality inspection index data of which the data type is matched with the type of the target quality inspection index data from the quality inspection index data pool according to the type of the target quality inspection index data.
By determining the data type of the quality inspection target task, quality inspection index data with the data type matched with the target quality inspection index data type can be accurately screened from the quality inspection index data pool, and the quality inspection index data are quality inspection index data required by the current quality inspection target task, so that the quality inspection result calculation of the current quality inspection target task can be conveniently completed.
In one or more embodiments of the present invention, the specific implementation of the quality inspection index data pool module calculating the quality inspection index data according to the service data information and the predefined rule is as follows:
judging the data volume of the service data information, and determining a sampling proportion according to the data volume of the service data information;
sampling the service data information according to the sampling proportion, and calculating quality inspection index data according to the service data information obtained by sampling and a predefined rule.
By sampling in a corresponding proportion according to the data volume of the service data information, the calculation amount of quality inspection index data can be reduced, the calculation efficiency is improved, and the system performance is ensured.
The invention also provides a computer readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the big data quality inspection method.
The invention also provides big data quality inspection equipment which comprises the storage medium and a processor, wherein the processor realizes the steps of the big data quality inspection method when executing the computer program on the storage medium.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A big data quality inspection method is characterized by comprising the following steps:
s1: reading service data information in a database, calculating quality inspection index data according to the service data information and a predefined rule, and constructing a quality inspection index data pool;
s2: determining a corresponding quality inspection rule according to a quality inspection target task, and calculating a quality inspection result according to the quality inspection rule and quality inspection index data in a quality inspection index data pool;
s3: and directly comparing the quality inspection result with a corresponding preset quality inspection threshold value, and generating early warning information when the quality inspection result and the preset quality inspection threshold value trigger an early warning rule.
2. The big data quality inspection method according to claim 1, wherein before the step S2, the method further comprises the steps of:
and screening the quality inspection index data in the quality inspection index data pool according to a quality inspection target task, and marking the screened quality inspection index data.
3. The big data quality inspection method according to claim 2, wherein the specific method for screening the quality inspection index data in the quality inspection index data pool according to the quality inspection target task is as follows:
and determining the type of target service data according to the quality inspection target task, and screening quality inspection index data of which the data type is matched with the type of the target quality inspection index data from the quality inspection index data pool according to the type of the target quality inspection index data.
4. The big data quality inspection method according to any one of claims 1 to 3, wherein in the step S1, the calculating quality inspection index data according to the business data information and the predefined rule specifically includes the steps of:
s11: judging the data volume of the service data information, and determining a sampling proportion according to the data volume of the service data information;
s12: sampling the service data information according to the sampling proportion, and calculating quality inspection index data according to the service data information obtained by sampling and a predefined rule.
5. A big data quality inspection system is characterized in that: the quality control system comprises a quality control index data pool module, a quality control rule engine module and an early warning module;
the quality inspection index data pool module is used for reading the service data information in the database, calculating quality inspection index data according to the service data information and a predefined rule, and constructing a quality inspection index data pool;
the quality inspection rule engine module is used for determining a corresponding quality inspection rule according to a quality inspection target task and calculating a quality inspection result according to the quality inspection rule and quality inspection index data in the quality inspection index data pool;
and the early warning module is used for directly comparing the quality inspection result with a corresponding preset quality inspection threshold value and generating early warning information when the quality inspection result and the preset quality inspection threshold value trigger an early warning rule.
6. The big data quality inspection system of claim 5, wherein: the quality inspection target task screening and marking module is used for screening the quality inspection index data in the quality inspection index data pool according to the quality inspection target task and marking the screened quality inspection index data.
7. The big data quality inspection system of claim 6, wherein: the screening and marking module screens the quality inspection index data in the quality inspection index data pool according to the quality inspection target task, and the screening and marking module specifically realizes the following steps:
and determining the type of target service data according to the quality inspection target task, and screening quality inspection index data of which the data type is matched with the type of the target quality inspection index data from the quality inspection index data pool according to the type of the target quality inspection index data.
8. The big data quality inspection system according to any one of claims 5 to 7, wherein: the quality inspection index data pool module calculates quality inspection index data according to the service data information and the predefined rule, and the specific implementation is as follows:
judging the data volume of the service data information, and determining a sampling proportion according to the data volume of the service data information;
sampling the service data information according to the sampling proportion, and calculating quality inspection index data according to the service data information obtained by sampling and a predefined rule.
9. A computer-readable storage medium, having stored thereon a computer program which, when executed by a processor, implements the big data quality inspection method of any one of claims 1 to 4.
10. A big data quality inspection apparatus comprising the storage medium of claim 9 and a processor, the processor implementing the steps of the big data quality inspection method of any one of claims 1 to 4 when executing the computer program on the storage medium.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110085577.7A CN112783882A (en) | 2021-01-22 | 2021-01-22 | Big data quality inspection method, system, storage medium and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110085577.7A CN112783882A (en) | 2021-01-22 | 2021-01-22 | Big data quality inspection method, system, storage medium and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112783882A true CN112783882A (en) | 2021-05-11 |
Family
ID=75758477
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110085577.7A Pending CN112783882A (en) | 2021-01-22 | 2021-01-22 | Big data quality inspection method, system, storage medium and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112783882A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116975044A (en) * | 2023-09-21 | 2023-10-31 | 云粒智慧科技有限公司 | Quality inspection rule determining method, quality inspection rule determining device, quality inspection rule determining equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1826601A (en) * | 2003-08-29 | 2006-08-30 | 瑞士银行股份有限公司 | Redundancy-free provision of multi-purpose data |
| CN110019566A (en) * | 2019-03-13 | 2019-07-16 | 平安信托有限责任公司 | Data checking, device, computer equipment and storage medium based on data warehouse |
| CN111026749A (en) * | 2019-11-11 | 2020-04-17 | 支付宝(杭州)信息技术有限公司 | Service alarm method and device |
| CN111414376A (en) * | 2020-03-02 | 2020-07-14 | 中国建设银行股份有限公司 | Data early warning method and device |
| CN111563074A (en) * | 2020-04-28 | 2020-08-21 | 厦门市美亚柏科信息股份有限公司 | Data quality detection method and system based on multi-dimensional label |
-
2021
- 2021-01-22 CN CN202110085577.7A patent/CN112783882A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1826601A (en) * | 2003-08-29 | 2006-08-30 | 瑞士银行股份有限公司 | Redundancy-free provision of multi-purpose data |
| CN110019566A (en) * | 2019-03-13 | 2019-07-16 | 平安信托有限责任公司 | Data checking, device, computer equipment and storage medium based on data warehouse |
| CN111026749A (en) * | 2019-11-11 | 2020-04-17 | 支付宝(杭州)信息技术有限公司 | Service alarm method and device |
| CN111414376A (en) * | 2020-03-02 | 2020-07-14 | 中国建设银行股份有限公司 | Data early warning method and device |
| CN111563074A (en) * | 2020-04-28 | 2020-08-21 | 厦门市美亚柏科信息股份有限公司 | Data quality detection method and system based on multi-dimensional label |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116975044A (en) * | 2023-09-21 | 2023-10-31 | 云粒智慧科技有限公司 | Quality inspection rule determining method, quality inspection rule determining device, quality inspection rule determining equipment and storage medium |
| CN116975044B (en) * | 2023-09-21 | 2023-12-22 | 云粒智慧科技有限公司 | Quality inspection rule determining method, quality inspection rule determining device, quality inspection rule determining equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6707564B2 (en) | Data quality analysis | |
| CN107168854B (en) | Internet advertisement abnormal click detection method, device, equipment and readable storage medium | |
| CN108734561B (en) | Electronic device, order data processing method, and computer-readable storage medium | |
| CN109191133B (en) | Payment channel selection method and terminal equipment | |
| WO2019095665A1 (en) | Interview method, server and computer-readable storage medium | |
| CN111210321B (en) | Risk early warning method and system based on contract management | |
| CN112712417A (en) | Bidding management method, system and storage medium | |
| CN112783882A (en) | Big data quality inspection method, system, storage medium and equipment | |
| CN114969040A (en) | Data display method and device, electronic equipment and storage medium | |
| CN109697203B (en) | Index transaction analysis method and device, computer storage medium, and computer device | |
| JP2020057356A (en) | Intelligent prediction of bundles of spare parts | |
| CN111680941A (en) | Premium recommendation method, device, equipment and storage medium | |
| CN115168509A (en) | Processing method and device of wind control data, storage medium and computer equipment | |
| CN109345301A (en) | A kind of data price-determining system and determining method | |
| CN112329814B (en) | Invoice data processing method and equipment | |
| CN111427900B (en) | Label library updating method, device, equipment and readable storage medium | |
| CN108109002B (en) | Data processing method and device | |
| CN111833085A (en) | Method and device for calculating price of article | |
| CN112541514A (en) | Event distribution method, server, terminal and storage medium | |
| CN113409025B (en) | Service data extraction method, device and storage medium | |
| CN111598638A (en) | Click rate determination method, device and equipment | |
| CN114996113B (en) | Real-time monitoring and early warning method and device for abnormal operation of large-data online user | |
| CN113688645B (en) | Identification method, system and equipment | |
| CN113656486B (en) | Method, device, terminal equipment and storage medium for generating visualized object | |
| KR102473175B1 (en) | Method and apparatus of valuation of business model using jump model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210511 |
|
| RJ01 | Rejection of invention patent application after publication |