CN110309125A - Data verification method, electronic device and storage medium - Google Patents
Data verification method, electronic device and storage medium Download PDFInfo
- Publication number
- CN110309125A CN110309125A CN201910559406.6A CN201910559406A CN110309125A CN 110309125 A CN110309125 A CN 110309125A CN 201910559406 A CN201910559406 A CN 201910559406A CN 110309125 A CN110309125 A CN 110309125A
- Authority
- CN
- China
- Prior art keywords
- business datum
- index
- data
- thresholds
- metrics
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Abstract
The present invention relates to technical field of data processing, provide a kind of data verification method, electronic device and computer storage medium, this method comprises: by connecting the corresponding operation system of multiple data sources, corresponding business datum is obtained from each operation system according to preset temporal frequency and carries out index allocation, the corresponding index of each business datum and index allocation information are obtained, rule configuration is carried out to the corresponding index of obtained each business datum and generates the configuration file including metrics-thresholds；The corresponding index of each business datum and index allocation information input computing engines operation are obtained into the corresponding index value of each business datum again；Finally, reading the configuration file using regulation engine parses the corresponding metrics-thresholds of each business datum, verify whether the corresponding index value of each business datum is more than corresponding metrics-thresholds.The present invention is matched by the data progress rule periodically to each operation system postpones automation verification, it is ensured that the reliability of data, to improve the quality of data.
Description
Technical field
The present invention relates to technical field of data processing more particularly to a kind of data verification methods, electronic device and computer
Readable storage medium storing program for executing.
Background technique
With popularizing for big data technology, data processing and calculating have a wide range of applications in various industries field.Its
Middle ETL (Extract-Transform-Load, data warehouse) technology has important role in large-scale Data processing.
Currently, the business datum of many companies is to report to summarize by the operation system in each place, ETL is in data
It reports, indicator-specific statistics, higher to data quality requirement in the application scenarios such as Analysis of Policy Making, the mode of tradition verification data uses part
There is the deficiencies of verification low efficiency, artificial logarithm is error-prone, the degree of automation is not high, influence the standard of data in the artificial logarithm of data
True rate causes the decision of data wrong.
Summary of the invention
In view of the foregoing, the present invention provides a kind of data verification method, electronic device and computer readable storage medium,
Main purpose is periodically to carry out automation verification to the data of each operation system, it is ensured that the reliability of data improves data
Quality.
To achieve the above object, the present invention provides a kind of data verification method, is applied to electronic device, this method comprises:
Obtaining step: connecting the corresponding operation system of multiple data sources, according to preset temporal frequency from each business system
Corresponding business datum is obtained in system；
First configuration step: index allocation is carried out to every business datum of acquisition, obtains the corresponding finger of each business datum
Mark and index allocation information；
Second configuration step: rule configuration is carried out to the corresponding index of obtained each business datum and generates configuration file, institute
Stating configuration file includes the corresponding metrics-thresholds of each business datum；
It calculates step: the corresponding index of each business datum and index allocation information input computing engines operation is transported
The corresponding index value of each business datum after calculation；And
Verification step: the configuration file is read using regulation engine, the configuration file is parsed and obtains each business datum
Corresponding metrics-thresholds verify whether the corresponding index value of each business datum is more than corresponding metrics-thresholds.
Preferably, the verification step includes:
When judging that the corresponding index value of a business datum is more than corresponding metrics-thresholds, judge that the business datum has
Accidentally, preset mark is distributed for the wrong business datum and save to early warning table.
Preferably, after the verification step further include:
Warning step: early warning table described in timing scan, when scanning is to when having the preset mark business datum, triggering is default
Early warning system generates corresponding warning information, and is sent to corresponding operation system.
Preferably, the index allocation information includes the associated data source of index, the arithmetic logic of index and setup time.
Preferably, the calculating step includes:
The arithmetic logic of index corresponding with each business datum is found out from obtained index allocation information；
SQL calculating is carried out to the corresponding index of each business datum according to the arithmetic logic found out, obtains each business datum
Corresponding index value.
In addition, to achieve the above object, the present invention also provides a kind of electronic device, which includes memory and place
Device is managed, is stored with the data verifying program that can be run on the processor, the data verifying program quilt in the memory
The processor realizes following steps when executing:
Obtaining step: connecting the corresponding operation system of multiple data sources, according to preset temporal frequency from each business system
Corresponding business datum is obtained in system；
First configuration step: index allocation is carried out to every business datum of acquisition, obtains the corresponding finger of each business datum
Mark and index allocation information；
Second configuration step: rule configuration is carried out to the corresponding index of obtained each business datum and generates configuration file, institute
Stating configuration file includes the corresponding metrics-thresholds of each business datum；
It calculates step: the corresponding index of each business datum and index allocation information input computing engines operation is transported
The corresponding index value of each business datum after calculation；And
Verification step: the configuration file is read using regulation engine, the configuration file is parsed and obtains each business datum
Corresponding metrics-thresholds verify whether the corresponding index value of each business datum is more than corresponding metrics-thresholds.
Preferably, the verification step includes:
When judging that the corresponding index value of a business datum is more than corresponding metrics-thresholds, judge that the business datum has
Accidentally, preset mark is distributed for the wrong business datum and save to early warning table.
Preferably, after the verification step further include:
Warning step: early warning table described in timing scan, when scanning is to when having the preset mark business datum, triggering is default
Early warning system generates corresponding warning information, and is sent to corresponding operation system.
Preferably, the index allocation information includes the associated data source of index, the arithmetic logic of index and setup time.
In addition, to achieve the above object, it is described computer-readable the present invention also provides a kind of computer readable storage medium
It include data verifying program in storage medium, it can be achieved that number as described above when the data verifying program is executed by processor
According to the either step in method of calibration.
Data verification method, electronic device and computer readable storage medium proposed by the present invention, by connecting multiple numbers
According to the corresponding operation system in source, corresponding business datum is obtained from each operation system according to preset temporal frequency, to obtaining
The every business datum taken carries out index allocation, the corresponding index of each business datum and index allocation information is obtained, to what is obtained
The corresponding index of each business datum carries out rule configuration and generates configuration file, and the configuration file includes that each business datum is corresponding
Metrics-thresholds；Again by the corresponding index of each business datum and index allocation information input computing engines operation, after obtaining operation
The corresponding index value of each business datum；Finally, reading the configuration file using regulation engine, parses the configuration file and obtain
The corresponding metrics-thresholds of each business datum, verify whether the corresponding index value of each business datum is more than corresponding metrics-thresholds.This
Invention is matched by the data progress rule periodically to each operation system postpones automation verification, it is ensured that the reliability of data, thus
Improve the quality of data.
Detailed description of the invention
Fig. 1 is the schematic diagram of electronic device preferred embodiment of the present invention；
Fig. 2 is the Program modual graph of one preferred embodiment of data verifying program in Fig. 1；
Fig. 3 is the flow chart of one preferred embodiment of data verification method of the present invention；
Fig. 4 is the Program modual graph of the data verifying program another preferred embodiment in Fig. 1；
Fig. 5 is the flow chart of data verification method another preferred embodiment of the present invention；
The embodiments will be further described with reference to the accompanying drawings for the realization, the function and the advantages of the object of the present invention.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to the accompanying drawings and embodiments, right
The present invention is further elaborated.It should be appreciated that described herein, specific examples are only used to explain the present invention, not
For limiting the present invention.Based on the embodiments of the present invention, those of ordinary skill in the art are not before making creative work
Every other embodiment obtained is put, shall fall within the protection scope of the present invention.
It should be noted that the description for being related to " first ", " second " etc. in the present invention is used for description purposes only, and cannot
It is interpreted as its relative importance of indication or suggestion or implicitly indicates the quantity of indicated technical characteristic.Define as a result, " the
One ", the feature of " second " can explicitly or implicitly include at least one of the features.In addition, the skill between each embodiment
Art scheme can be combined with each other, but must be based on can be realized by those of ordinary skill in the art, when technical solution
Will be understood that the combination of this technical solution is not present in conjunction with there is conflicting or cannot achieve when, also not the present invention claims
Protection scope within.
It is the schematic diagram of electronic device preferred embodiment of the present invention shown in referring to Fig.1.Electronic device 1 is that one kind can be by
According to the instruction for being previously set or storing, the automatic equipment for carrying out numerical value calculating and/or information processing.The electronic device 1 can
To be computer, be also possible to single network server, the server group that multiple network servers form or based on cloud computing
The cloud being made of a large amount of hosts or network server, wherein cloud computing is one kind of distributed computing, by the loose coupling of a group
One super virtual computer of the computer set composition of conjunction.
In the present embodiment, electronic device 1 may include, but be not limited only to, and can be in communication with each other connection by system bus
Memory 11, processor 12, network interface 13, memory 11 are stored with the data verifying program that can be run on the processor 12
10.It should be pointed out that Fig. 1 illustrates only the electronic device 1 with component 11-13 it should be appreciated that be not required for
Implement all components shown, the implementation that can be substituted is more or less component.
Wherein, memory 11 includes the readable storage medium storing program for executing of memory and at least one type.Inside save as the fortune of electronic device 1
Row provides caching；Readable storage medium storing program for executing can be for if flash memory, hard disk, multimedia card, card-type memory are (for example, SD or DX memory
Deng), random access storage device (RAM), static random-access memory (SRAM), read-only memory (ROM), electric erasable can compile
Journey read-only memory (EEPROM), programmable read only memory (PROM), magnetic storage, disk, CD etc. it is non-volatile
Storage medium.In some embodiments, readable storage medium storing program for executing can be the internal storage unit of electronic device 1, such as the electronics
The hard disk of device 1；In further embodiments, the external storage which is also possible to electronic device 1 is set
Plug-in type hard disk that is standby, such as being equipped on electronic device 1, intelligent memory card (Smart Media Card, SMC), secure digital
(Secure Digital, SD) card, flash card (Flash Card) etc..In the present embodiment, the readable storage medium storing program for executing of memory 11
Commonly used in storing in the operating system and types of applications software that are installed on electronic device 1, such as storage one embodiment of the invention
Data verifying program 10 etc..It has exported or will export each in addition, memory 11 can be also used for temporarily storing
Class data.
The processor 12 can be in some embodiments central processing unit (Central Processing Unit,
CPU), controller, microcontroller, microprocessor or other data processing chips.The processor 12 is commonly used in the control electricity
The overall operation of sub-device 1, such as execute control relevant to other equipment progress data interaction or communication and processing
Deng.In the present embodiment, the processor 12 is for running the program code stored in the memory 11 or processing data, example
Such as operation data checking routine 10.
The network interface 13 may include radio network interface or wired network interface, which is commonly used in
Communication connection is established between the electronic device 1 and other electronic equipments.
The data verifying program 10 is stored in memory 11, including the computer-readable finger being stored in memory 11
It enables, which can be executed by processor 12, the method to realize each embodiment of the application.
In one embodiment, following steps are realized when above-mentioned data verifying program 10 is executed by the processor 12:
Obtaining step: connecting the corresponding operation system of multiple data sources, according to preset temporal frequency from each business system
Corresponding business datum is obtained in system.
In the present embodiment, data source includes operation system title, IP domain name, connectivity port, database, type of database,
Data synchronizing frequency etc..It is connected to each operation system by data source, is obtained from each operation system by preset temporal frequency
Corresponding business datum is taken, and the warehouse Hive is cured to the business datum got, each operation system is not influenced and continues typing
Other data, it is ensured that the reliability of the original service data got.The Hive is a kind of Tool for Data Warehouse, can be right
It gets data to extract, conversion, load, SQL data query etc. can also be carried out.
In one embodiment, the corresponding business system of multiple data sources (such as not the IP domain name of homologous ray or connectivity port) is connected
System obtains phase according to preset temporal frequency (such as daily 8 a.m.) from each business system (such as operation system of each province, city)
The business datum (declaration form data) answered is cured to the warehouse Hive.
First configuration step: index allocation is carried out to every business datum of acquisition, obtains the corresponding finger of each business datum
Mark and index allocation information.
Specifically, index allocation is carried out to business datum according to the field attribute of every of acquisition business datum, obtained each
The corresponding index of business datum, then the corresponding index of each business datum will be obtained and be associated, simultaneously to the index allocation after association
Preset arithmetic logic is matched, index allocation information is finally obtained.
Further, when the index allocation information includes the associated data source of index, the arithmetic logic of index and configuration
Between.
The associated data source of index includes the number of the association and each business datum between the corresponding index of each business datum
According to the association between source, convenient for the source of trace back data；
The arithmetic logic of index includes generic function (function of such as sum, count, min or max) and pre-set
Custom function (function of such as mathematical formulae)；
Setup time refers to that each business datum carries out the time record of index allocation.
Second configuration step: rule configuration is carried out to the corresponding index of obtained each business datum and generates configuration file, institute
Stating configuration file includes the corresponding metrics-thresholds of each business datum.
In the present embodiment, in order to each business datum got can automatic Verification, business datum is advised in advance
It preparing and sets, i.e., rule configures the corresponding index of each business datum, generates configuration file, which is the file of drl format,
It can be parsed by Drools regulation engine, the configuration file includes the corresponding metrics-thresholds of each business datum, which is
Corresponding metrics-thresholds are generated in rule configuration according to the field attribute of each business datum and corresponding index.
It calculates step: the corresponding index of each business datum and index allocation information input computing engines operation is transported
The corresponding index value of each business datum after calculation.
In the present embodiment, computing engines use Spark engine, be it is a kind of quickly, general procedure large-scale data Spark draws
It holds up.The calculating of Spark engine is carried out according to the corresponding index of each business datum and index allocation information, including the use of Spark engine
In SQL tool calculate and/or dimension hierarchy is unfolded between the business datum of each operation system using Spark engine meter
It calculates, the corresponding index value of each business datum after being calculated, and is saved with json format to database.
SQL is the abbreviation of structured query language (Structured Query Language).Be a kind of data query,
The tool of programming and data management, the calculating to processing large-scale data, and a variety of different calculation tools are provided
There is versatility.
Further, the calculating step includes:
The arithmetic logic of index corresponding with each business datum is found out from obtained index allocation information；
SQL calculating is carried out to the corresponding index of each business datum according to the arithmetic logic found out, obtains each business datum
Corresponding index value.
Wherein, arithmetic logic include generic function (function of such as sum, count, min or max) and it is pre-set from
Defined function (function of such as mathematical formulae), the arithmetic logic being configured to using SQL according to the corresponding index of each business datum,
It completes complicated, multiplicity business datum to calculate, obtains the corresponding index value of each business datum.
Verification step: the configuration file is read using regulation engine, the configuration file is parsed and obtains each business datum
Corresponding metrics-thresholds verify whether the corresponding index value of each business datum is more than corresponding metrics-thresholds.
The present embodiment, regulation engine use Drools engine, automatic loading configuration file and can parse, for business number
According to the configuration file that rule configuration is completed, Drools engine reads the configuration file automatically, and parses each business datum pair
Whether the metrics-thresholds answered are more than corresponding metrics-thresholds, energy to verify the corresponding index value of each business datum after operation
It is enough that fast, efficiently the corresponding index value of each business datum after operation is verified, judge the correct of each business datum
Property.
Further, the verification step includes:
When judging that the corresponding index value of a business datum is more than corresponding metrics-thresholds, judge that the business datum has
Accidentally, preset mark is distributed for the wrong business datum and save to early warning table.
Further, after the verification step further include:
Warning step: early warning table described in timing scan, when scanning is to when having the preset mark business datum, triggering is default
Early warning system generates corresponding warning information, and is sent to corresponding operation system.
In one embodiment, periodically the early warning table is scanned by the preset time, when scanning is described to having
When preset mark business datum, triggers default early warning system and generate corresponding warning information, the warning information includes label
Wrong business datum, the associated data source of wrong business datum and pre-warning time etc., and the warning information is sent to pair
The operation system answered.Operation system after receiving the warning information, the operator of operation system according to warning information into
Row analysis, solves wrong business datum in time, it is ensured that the correctness of data improves the quality of data.
It is the Program modual graph of 10 1 preferred embodiment of data verifying program in Fig. 1 referring to shown in Fig. 2.
In one embodiment, data verifying program 10 includes: to obtain module 101, the first configuration module 102, second match
Set module 103, computing module 104, correction verification module 105.The functions or operations step that the module 101-105 is realized is under
It is similar to state data verification method, and will not be described here in detail, illustratively, such as wherein:
Module 101 is obtained, for connecting the corresponding operation system of multiple data sources, according to preset temporal frequency from each
Corresponding business datum is obtained in operation system；
First configuration module 102 obtains each business datum pair for carrying out index allocation to every business datum of acquisition
The index and index allocation information answered；
Second configuration module 103 generates configuration for carrying out rule configuration to the corresponding index of obtained each business datum
File, the configuration file include the corresponding metrics-thresholds of each business datum；
Computing module 104 is used for the corresponding index of each business datum and index allocation information input computing engines operation,
The corresponding index value of each business datum after obtaining operation；And
Correction verification module 105 parses the configuration file and obtains each industry for reading the configuration file using regulation engine
The corresponding metrics-thresholds of data of being engaged in, verify whether the corresponding index value of each business datum is more than corresponding metrics-thresholds.
Referring to shown in Fig. 4, it is the Program modual graph of 10 another preferred embodiment of data verifying program in Fig. 1, is verifying
After module 105, data verifying program 10 further includes warning module 106, illustratively:
Warning module 106, for early warning table described in timing scan, when scanning is to when having the preset mark business datum,
It triggers default early warning system and generates corresponding warning information, and be sent to corresponding operation system.
It is the flow chart of one preferred embodiment of data verification method of the present invention referring to shown in Fig. 3.Disclosed herein one kind
Data verification method, applied to above-mentioned electronic device, this method comprises:
Step S210: connecting the corresponding operation system of multiple data sources, according to preset temporal frequency from each business system
Corresponding business datum is obtained in system.
In the present embodiment, data source includes operation system title, IP domain name, connectivity port, database, type of database,
Data synchronizing frequency etc..It is connected to each operation system by data source, is obtained from each operation system by preset temporal frequency
Corresponding business datum is taken, and the warehouse Hive is cured to the business datum got, each operation system is not influenced and continues typing
Other data, it is ensured that the reliability of the original service data got.The Hive is a kind of Tool for Data Warehouse, can be right
It gets data to extract, conversion, load, SQL data query etc. can also be carried out.
In one embodiment, the corresponding business system of multiple data sources (such as not the IP domain name of homologous ray or connectivity port) is connected
System obtains phase according to preset temporal frequency (such as daily 8 a.m.) from each business system (such as operation system of each province, city)
The business datum (declaration form data) answered is cured to the warehouse Hive.
Step S220: carrying out index allocation to every business datum of acquisition, obtain the corresponding index of each business datum and
Index allocation information.
Specifically, index allocation is carried out to business datum according to the field attribute of every of acquisition business datum, obtained each
The corresponding index of business datum, then the corresponding index of each business datum will be obtained and be associated, simultaneously to the index allocation after association
Preset arithmetic logic is matched, index allocation information is finally obtained.
Further, when the index allocation information includes the associated data source of index, the arithmetic logic of index and configuration
Between.
The associated data source of index includes the number of the association and each business datum between the corresponding index of each business datum
According to the association between source, convenient for the source of trace back data；
The arithmetic logic of index includes generic function (function of such as sum, count, min or max) and pre-set
Custom function (function of such as mathematical formulae)；
Setup time refers to that each business datum carries out the time record of index allocation.
Step S230: carrying out rule configuration to the corresponding index of obtained each business datum and generate configuration file, described to match
Setting file includes the corresponding metrics-thresholds of each business datum.
In the present embodiment, in order to each business datum got can automatic Verification, business datum is advised in advance
It preparing and sets, i.e., rule configures the corresponding index of each business datum, generates configuration file, which is the file of drl format,
It can be parsed by Drools regulation engine, the configuration file includes the corresponding metrics-thresholds of each business datum, which is
Corresponding metrics-thresholds are generated in rule configuration according to the field attribute of each business datum and corresponding index.
Step S240: the corresponding index of each business datum and index allocation information input computing engines operation are transported
The corresponding index value of each business datum after calculation.
In the present embodiment, computing engines use Spark engine, be it is a kind of quickly, general procedure large-scale data Spark draws
It holds up.The calculating of Spark engine is carried out according to the corresponding index of each business datum and index allocation information, including the use of Spark engine
In SQL tool calculate and/or dimension hierarchy is unfolded between the business datum of each operation system using Spark engine meter
It calculates, the corresponding index value of each business datum after being calculated, and is saved with json format to database.
SQL is the abbreviation of structured query language (Structured Query Language).Be a kind of data query,
The tool of programming and data management, the calculating to processing large-scale data, and a variety of different calculation tools are provided
There is versatility.
Further, the step S240: include:
The arithmetic logic of index corresponding with each business datum is found out from obtained index allocation information；
SQL calculating is carried out to the corresponding index of each business datum according to the arithmetic logic found out, obtains each business datum
Corresponding index value.
Wherein, arithmetic logic include generic function (function of such as sum, count, min or max) and it is pre-set from
Defined function (function of such as mathematical formulae), the arithmetic logic being configured to using SQL according to the corresponding index of each business datum,
It completes complicated, multiplicity business datum to calculate, obtains the corresponding index value of each business datum.
Step S250: the configuration file is read using regulation engine, the configuration file is parsed and obtains each business datum
Corresponding metrics-thresholds verify whether the corresponding index value of each business datum is more than corresponding metrics-thresholds.
The present embodiment, regulation engine use Drools engine, automatic loading configuration file and can parse, for business number
According to the configuration file that rule configuration is completed, Drools engine reads the configuration file automatically, and parses each business datum pair
Whether the metrics-thresholds answered are more than corresponding metrics-thresholds, energy to verify the corresponding index value of each business datum after operation
It is enough that fast, efficiently the corresponding index value of each business datum after operation is verified, judge the correct of each business datum
Property.
Further, the step S250: include:
When judging that the corresponding index value of a business datum is more than corresponding metrics-thresholds, judge that the business datum has
Accidentally, preset mark is distributed for the wrong business datum and save to early warning table.
It referring to Figure 5, is the flow chart of data verification method another preferred embodiment of the present invention, in the step S250
Afterwards, this method further include:
Step S260: early warning table described in timing scan, when scanning is to when having the preset mark business datum, triggering is default
Early warning system generates corresponding warning information, and is sent to corresponding operation system.
In one embodiment, periodically the early warning table is scanned by the preset time, when scanning is described to having
When preset mark business datum, triggers default early warning system and generate corresponding warning information, the warning information includes label
Wrong business datum, the associated data source of wrong business datum and pre-warning time etc., and the warning information is sent to pair
The operation system answered.Operation system after receiving the warning information, the operator of operation system according to warning information into
Row analysis, solves wrong business datum in time, it is ensured that the correctness of data improves the quality of data.
In addition, including in the computer readable storage medium the present invention also provides a kind of computer readable storage medium
Data verifying program, it can be achieved that following operation when the data verifying program is executed by processor:
Obtaining step: connecting the corresponding operation system of multiple data sources, according to preset temporal frequency from each business system
Corresponding business datum is obtained in system；
First configuration step: index allocation is carried out to every business datum of acquisition, obtains the corresponding finger of each business datum
Mark and index allocation information；
Second configuration step: rule configuration is carried out to the corresponding index of obtained each business datum and generates configuration file, institute
Stating configuration file includes the corresponding metrics-thresholds of each business datum；
It calculates step: the corresponding index of each business datum and index allocation information input computing engines operation is transported
The corresponding index value of each business datum after calculation；And
Verification step: the configuration file is read using regulation engine, the configuration file is parsed and obtains each business datum
Corresponding metrics-thresholds verify whether the corresponding index value of each business datum is more than corresponding metrics-thresholds.
Computer readable storage medium specific embodiment of the present invention and above-mentioned data verification method and each reality of electronic device
It is essentially identical to apply example, does not make tired state herein.
The serial number of the above embodiments of the invention is only for description, does not represent the advantages or disadvantages of the embodiments.
It should be noted that, in this document, the terms "include", "comprise" or its any other variant are intended to non-row
His property includes, so that the process, device, article or the method that include a series of elements not only include those elements, and
And further include other elements that are not explicitly listed, or further include for this process, device, article or method institute it is intrinsic
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including being somebody's turn to do
There is also other identical elements in the process, device of element, article or method.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can be realized by means of software and necessary general hardware platform, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Based on this understanding, technical solution of the present invention substantially in other words does the prior art
The part contributed out can be embodied in the form of software products, which is stored in one as described above
In storage medium (such as ROM/RAM, magnetic disk, CD), including some instructions are used so that terminal device (it can be mobile phone,
Computer, server or network equipment etc.) execute method described in each embodiment of the present invention.
The above is only a preferred embodiment of the present invention, is not intended to limit the scope of the invention, all to utilize this hair
Equivalent structure or equivalent flow shift made by bright specification and accompanying drawing content is applied directly or indirectly in other relevant skills
Art field, is included within the scope of the present invention.
Claims (10)
1. a kind of data verification method is applied to electronic device, which is characterized in that this method comprises:
Obtaining step: the corresponding operation system of multiple data sources is connected, according to preset temporal frequency from each operation system
Obtain corresponding business datum；
First configuration step: carrying out index allocation to every business datum of acquisition, obtain the corresponding index of each business datum and
Index allocation information；
Second configuration step: carrying out rule configuration to the corresponding index of obtained each business datum and generate configuration file, described to match
Setting file includes the corresponding metrics-thresholds of each business datum；
Calculate step: by the corresponding index of each business datum and index allocation information input computing engines operation, after obtaining operation
The corresponding index value of each business datum；And
Verification step: reading the configuration file using regulation engine, parses the configuration file and obtains each business datum correspondence
Metrics-thresholds, verify whether the corresponding index value of each business datum is more than corresponding metrics-thresholds.
2. data verification method as described in claim 1, which is characterized in that the verification step includes:
When judging that the corresponding index value of a business datum is more than corresponding metrics-thresholds, judges that the business datum is wrong, be
The wrong business datum distribution preset mark is simultaneously saved to early warning table.
3. data verification method as claimed in claim 2, which is characterized in that after the verification step further include:
Warning step: early warning table described in timing scan, when scanning is to when having the preset mark business datum, early warning is preset in triggering
System generates corresponding warning information, and is sent to corresponding operation system.
4. data verification method as described in claim 1, which is characterized in that the index allocation information includes that index is associated
Data source, the arithmetic logic of index and setup time.
5. data verification method according to any one of claims 1-4, which is characterized in that the calculating step includes:
The arithmetic logic of index corresponding with each business datum is found out from obtained index allocation information；
SQL calculating is carried out to the corresponding index of each business datum according to the arithmetic logic found out, it is corresponding to obtain each business datum
Index value.
6. a kind of electronic device, which is characterized in that the electronic device includes memory and processor, is stored in the memory
The data verifying program that can be run on the processor is realized as follows when the data verifying program is executed by the processor
Step:
Obtaining step: the corresponding operation system of multiple data sources is connected, according to preset temporal frequency from each operation system
Obtain corresponding business datum；
First configuration step: carrying out index allocation to every business datum of acquisition, obtain the corresponding index of each business datum and
Index allocation information；
Second configuration step: carrying out rule configuration to the corresponding index of obtained each business datum and generate configuration file, described to match
Setting file includes the corresponding metrics-thresholds of each business datum；
Calculate step: by the corresponding index of each business datum and index allocation information input computing engines operation, after obtaining operation
The corresponding index value of each business datum；And
Verification step: reading the configuration file using regulation engine, parses the configuration file and obtains each business datum correspondence
Metrics-thresholds, verify whether the corresponding index value of each business datum is more than corresponding metrics-thresholds.
7. electronic device as claimed in claim 6, which is characterized in that the verification step includes:
When judging that the corresponding index value of a business datum is more than corresponding metrics-thresholds, judges that the business datum is wrong, be
The wrong business datum distribution preset mark is simultaneously saved to early warning table.
8. electronic device as claimed in claim 7, which is characterized in that after the verification step further include:
Warning step: early warning table described in timing scan, when scanning is to when having the preset mark business datum, early warning is preset in triggering
System generates corresponding warning information, and is sent to corresponding operation system.
9. electronic device as claimed in claim 6, which is characterized in that the index allocation information includes the associated data of index
Source, the arithmetic logic of index and setup time.
10. a kind of computer readable storage medium, which is characterized in that include data check in the computer readable storage medium
Program, it can be achieved that data as described in any one of claim 1 to 5 when the data verifying program is executed by processor
The step of method of calibration.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910559406.6A CN110309125B (en) | 2019-06-24 | 2019-06-24 | Data verification method, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910559406.6A CN110309125B (en) | 2019-06-24 | 2019-06-24 | Data verification method, electronic device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110309125A true CN110309125A (en) | 2019-10-08 |
| CN110309125B CN110309125B (en) | 2021-09-21 |
Family
ID=68076631
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910559406.6A Active CN110309125B (en) | 2019-06-24 | 2019-06-24 | Data verification method, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110309125B (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111325466A (en) * | 2020-02-20 | 2020-06-23 | 深圳壹账通智能科技有限公司 | Intelligent early warning method and system |
| CN112445875A (en) * | 2020-12-14 | 2021-03-05 | 招商局金融科技有限公司 | Data association and inspection method and device, electronic equipment and storage medium |
| CN112506901A (en) * | 2020-11-30 | 2021-03-16 | 深圳微众信用科技股份有限公司 | Data quality measuring method, device and medium |
| CN112540996A (en) * | 2020-11-12 | 2021-03-23 | Oppo（重庆）智能科技有限公司 | Service data verification method and device, electronic equipment and storage medium |
| CN112579376A (en) * | 2020-12-16 | 2021-03-30 | 中国建设银行股份有限公司 | Double-run-stage data verification method, system, equipment and storage medium |
| CN112734358A (en) * | 2020-12-25 | 2021-04-30 | 山东浪潮通软信息科技有限公司 | Configuration method and device of business system |
| CN112733083A (en) * | 2019-10-28 | 2021-04-30 | 中国移动通信集团湖南有限公司 | Data verification method, system and device |
| CN112767767A (en) * | 2021-01-29 | 2021-05-07 | 重庆子元科技有限公司 | Virtual training system |
| CN113128848A (en) * | 2021-04-01 | 2021-07-16 | 广州虎牙科技有限公司 | Data quality monitoring method for full-service index, electronic equipment and storage medium |
| CN113223200A (en) * | 2021-05-07 | 2021-08-06 | 广州天长信息技术有限公司 | Road stealing and escaping intelligent prevention and control method, storage medium and system based on index dimension |
| CN113515923A (en) * | 2021-09-14 | 2021-10-19 | 浙江省金融综合服务平台管理有限公司 | Index processing implementation method applied to decision engine |
| CN113553533A (en) * | 2021-06-10 | 2021-10-26 | 国网安徽省电力有限公司 | Index calculation method based on digital internal five-level market assessment system |
| CN113609283A (en) * | 2021-07-28 | 2021-11-05 | 浙江惠瀜网络科技有限公司 | Data acquisition method and system |
| CN113778535A (en) * | 2021-08-10 | 2021-12-10 | 济南浪潮数据技术有限公司 | Method, system, equipment and storage medium for operation verification of cloud host |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101135715A (en) * | 2007-05-25 | 2008-03-05 | 中兴通讯股份有限公司 | North direction interface data integrity verification method |
| US20110137939A1 (en) * | 2009-12-09 | 2011-06-09 | Linkage Technology Group Co., Ltd. | Data Supervision Based on the Configuration Rule of All Operational Indicators |
| CN105550511A (en) * | 2015-12-11 | 2016-05-04 | 北京锐软科技股份有限公司 | Data quality evaluation system and method based on data verification technique |
| CN106452933A (en) * | 2015-08-05 | 2017-02-22 | 腾讯科技（北京）有限公司 | Service data statistical method, device and system |
| CN107038200A (en) * | 2016-12-15 | 2017-08-11 | 平安科技（深圳）有限公司 | Business data processing method and system |
| CN109522318A (en) * | 2018-10-22 | 2019-03-26 | 中国银行股份有限公司 | A kind of data quality management method and system |
| CN109766370A (en) * | 2018-12-27 | 2019-05-17 | 口碑(上海)信息技术有限公司 | Data processing method, data service system and equipment |
-
2019
- 2019-06-24 CN CN201910559406.6A patent/CN110309125B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101135715A (en) * | 2007-05-25 | 2008-03-05 | 中兴通讯股份有限公司 | North direction interface data integrity verification method |
| US20110137939A1 (en) * | 2009-12-09 | 2011-06-09 | Linkage Technology Group Co., Ltd. | Data Supervision Based on the Configuration Rule of All Operational Indicators |
| CN106452933A (en) * | 2015-08-05 | 2017-02-22 | 腾讯科技（北京）有限公司 | Service data statistical method, device and system |
| CN105550511A (en) * | 2015-12-11 | 2016-05-04 | 北京锐软科技股份有限公司 | Data quality evaluation system and method based on data verification technique |
| CN107038200A (en) * | 2016-12-15 | 2017-08-11 | 平安科技（深圳）有限公司 | Business data processing method and system |
| CN109522318A (en) * | 2018-10-22 | 2019-03-26 | 中国银行股份有限公司 | A kind of data quality management method and system |
| CN109766370A (en) * | 2018-12-27 | 2019-05-17 | 口碑(上海)信息技术有限公司 | Data processing method, data service system and equipment |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112733083B (en) * | 2019-10-28 | 2023-08-18 | 中国移动通信集团湖南有限公司 | Data verification method, system and device |
| CN112733083A (en) * | 2019-10-28 | 2021-04-30 | 中国移动通信集团湖南有限公司 | Data verification method, system and device |
| CN111325466A (en) * | 2020-02-20 | 2020-06-23 | 深圳壹账通智能科技有限公司 | Intelligent early warning method and system |
| WO2021164465A1 (en) * | 2020-02-20 | 2021-08-26 | 深圳壹账通智能科技有限公司 | Intelligent early warning method and system |
| CN112540996A (en) * | 2020-11-12 | 2021-03-23 | Oppo（重庆）智能科技有限公司 | Service data verification method and device, electronic equipment and storage medium |
| CN112540996B (en) * | 2020-11-12 | 2023-11-24 | Oppo（重庆）智能科技有限公司 | Service data verification method and device, electronic equipment and storage medium |
| CN112506901A (en) * | 2020-11-30 | 2021-03-16 | 深圳微众信用科技股份有限公司 | Data quality measuring method, device and medium |
| CN112445875A (en) * | 2020-12-14 | 2021-03-05 | 招商局金融科技有限公司 | Data association and inspection method and device, electronic equipment and storage medium |
| CN112445875B (en) * | 2020-12-14 | 2024-03-01 | 招商局金融科技有限公司 | Data association and verification method and device, electronic equipment and storage medium |
| CN112579376A (en) * | 2020-12-16 | 2021-03-30 | 中国建设银行股份有限公司 | Double-run-stage data verification method, system, equipment and storage medium |
| CN112734358A (en) * | 2020-12-25 | 2021-04-30 | 山东浪潮通软信息科技有限公司 | Configuration method and device of business system |
| CN112767767A (en) * | 2021-01-29 | 2021-05-07 | 重庆子元科技有限公司 | Virtual training system |
| CN113128848A (en) * | 2021-04-01 | 2021-07-16 | 广州虎牙科技有限公司 | Data quality monitoring method for full-service index, electronic equipment and storage medium |
| CN113223200A (en) * | 2021-05-07 | 2021-08-06 | 广州天长信息技术有限公司 | Road stealing and escaping intelligent prevention and control method, storage medium and system based on index dimension |
| CN113553533A (en) * | 2021-06-10 | 2021-10-26 | 国网安徽省电力有限公司 | Index calculation method based on digital internal five-level market assessment system |
| CN113609283A (en) * | 2021-07-28 | 2021-11-05 | 浙江惠瀜网络科技有限公司 | Data acquisition method and system |
| CN113778535A (en) * | 2021-08-10 | 2021-12-10 | 济南浪潮数据技术有限公司 | Method, system, equipment and storage medium for operation verification of cloud host |
| CN113778535B (en) * | 2021-08-10 | 2024-02-23 | 济南浪潮数据技术有限公司 | Cloud host operation verification method, system, equipment and storage medium |
| CN113515923A (en) * | 2021-09-14 | 2021-10-19 | 浙江省金融综合服务平台管理有限公司 | Index processing implementation method applied to decision engine |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110309125B (en) | 2021-09-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110309125A (en) | Data verification method, electronic device and storage medium | |
| CN108268253B (en) | Interface code generation method and terminal equipment | |
| CN107145537B (en) | Table data importing method and system | |
| CN104572122A (en) | Software application data generating device and method | |
| CN108845950A (en) | Test device, the method for test and storage medium | |
| CN110275861A (en) | Date storage method and device, storage medium, electronic device | |
| CN108460068A (en) | Method, apparatus, storage medium and the terminal that report imports and exports | |
| CN110990403A (en) | Business data storage method, system, computer equipment and storage medium | |
| CN111858730A (en) | Data importing and exporting device, method, equipment and medium of graph database | |
| CN111177113A (en) | Data migration method and device, computer equipment and storage medium | |
| CN110795431B (en) | Environment monitoring data processing method, device, equipment and storage medium | |
| CN107357588B (en) | Object code generation method and device | |
| CN111475494A (en) | Mass data processing method, system, terminal and storage medium | |
| CN113327119B (en) | Virtual material coding method and system, electronic equipment and storage medium | |
| CN112395339B (en) | Intersystem data admission verification method, device, computer equipment and storage medium | |
| CN106095511A (en) | A kind of server updating method and apparatus | |
| CN106502977B (en) | A kind of sheet disposal method and form system based on B/S framework | |
| CN110019485A (en) | A kind of product data storage method, terminal device and storage medium | |
| CN104463460A (en) | Method and device for processing scheduling information for network data delivery | |
| CN113179294B (en) | Schema unified access method based on intelligent contract | |
| CN109669829A (en) | A kind of diagnosis adjustment method, device and server based on BMC | |
| CN106570004B (en) | Data management method and device | |
| CN113342647A (en) | Test data generation method and device | |
| CN113868138A (en) | Method, system, equipment and storage medium for acquiring test data | |
| CN116263717A (en) | Order service processing method and device based on event |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |