CN110515937A - A kind of data verification method and device - Google Patents
A kind of data verification method and device Download PDFInfo
- Publication number
- CN110515937A CN110515937A CN201910823356.8A CN201910823356A CN110515937A CN 110515937 A CN110515937 A CN 110515937A CN 201910823356 A CN201910823356 A CN 201910823356A CN 110515937 A CN110515937 A CN 110515937A
- Authority
- CN
- China
- Prior art keywords
- data
- verification
- verified
- check
- checking
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
Abstract
This application discloses a kind of data verification method and devices, data to be verified are obtained first, then it calls preset rules engine to treat verification data and carries out multiple check, the relational checking between table successively is verified including numeric field in data volume verification, field contents verification, interfield relational checking, table in the multiple check, includes multiple verifications rules in the regulation engine；It is carried out in checking procedure to the data to be verified, it is automatic to carry out the next item down verification if verification passes through；If a certain verification does not pass through, stores and show check results.The data verification method and device, by calling preset rules engine logarithm factually now to verify, due to the component that regulation engine is embeddable application program, what user can be convenient configures regulation engine and modifies, greatly convenient for users to use；Meanwhile the data verification method and device can carry out multi-faceted data check to data, can satisfy the application scenarios more demanding to verification.
Description
Technical field
The present invention relates to data processing techniques, and more specifically, it relates to a kind of data verification method and devices.
Background technique
It is reported and submitted in system in supervision data, according to regulatory requirements, business bank acquires according to data and standardizes, and reports silver on time
Row related data.When reporting related data, data needs are required to meet certain rule according to the quality of data, otherwise will record
Error message, and bank's reported data error situation is counted, order associated bank change or punishment etc. if mistake is more.
Therefore before data are reported supervision department, relevant data check is with regard to particularly necessary.
Existing data verification method, it is most of to be verified just for single field data, such as verification phone number
Whether ad hoc rules etc. is met；Small part can be between logical relation verification be carried out, as A field number adds B word different field data
Number of segment word meets condition equal to C field number.It can be seen that verification type ratio of the data verification method of the prior art to data
More single, verifying function is limited, and verifies rule and be directly fixed in code, be unfavorable for modifying and show.
Summary of the invention
In view of this, the present invention provides a kind of data verification method and device, to overcome data check in the prior art
Method verifying function is limited and verification rule is not easy the problem of modifying.
To achieve the above object, the invention provides the following technical scheme:
A kind of data verification method, comprising:
Obtain data to be verified；
It calls preset rules engine to carry out multiple check to the data to be verified, successively includes number in the multiple check
The relational checking between table, the rule are verified according to numeric field in amount verification, field contents verification, interfield relational checking, table
It include multiple verification rules in engine；
It is carried out in checking procedure to the data to be verified, it is automatic to carry out the next item down verification if verification passes through；If certain
Item verification does not pass through, then stores and show check results.
Optionally, further includes:
It carries out in checking procedure to the data to be verified, is automatically updated in the verification rule according to data frequency
Threshold value.
Optionally, the threshold value automatically updated according to data frequency in the verification rule, comprising:
The verification rule are updated according to history mean value, exponent-weighted average value, logistic regression numerical value and early warning distribution probability
Threshold value in then；
Or,
Dynamic update is carried out by the early warning situation between association analysis difference table.
Optionally, the data to be verified are Table A, carry out the data volume verification to the data to be verified, comprising:
Count in the Table A data volume of of that month data, and by the threshold value configured in the data volume and regulation engine and
Historical data amount is compared, the early warning if finding differences more than threshold value.
Optionally, relational checking carrying out the table to the data to be verified, comprising:
It verifies and whether meets the first preset relation between the field in two tables of data.
Optionally, before the calling preset rules engine carries out multiple check to the data to be verified, further includes:
The data to be verified are pre-processed.
It is optionally, described that the data to be verified are pre-processed, comprising:
The data to be verified are carried out to screen date data, screening field to be verified and generation auxiliary examination letter to be verified
The processing of breath.
Optionally, after the calling preset rules engine carries out multiple check to the data to be verified, further includes:
Standardization processing is carried out to the check results according to preset rules；
It is then described to store and show check results, comprising:
It stores and shows the check results by standardization processing.
A kind of data calibration device, comprising:
Data acquisition module, for obtaining data to be verified；
Multiple check module is described more for calling preset rules engine to carry out multiple check to the data to be verified
Re-graduation test in successively include data volume verification, field contents verification, in interfield relational checking, table numeric field verify between table
Relational checking includes that multiple verifications are regular in the regulation engine；
Process control module, for carrying out in checking procedure to the data to be verified, if verification passes through, control is certainly
It is dynamic to carry out the next item down verification；If a certain verification does not pass through, controls storage and show check results.
Optionally, further includes:
Threshold value update module, for being carried out in checking procedure in the multiple check module to the data to be verified,
The threshold value in the verification rule is automatically updated according to data frequency.
It can be seen via above technical scheme that compared with prior art, the embodiment of the invention discloses a kind of data checks
Method and device obtains data to be verified first, and preset rules engine is then called to carry out multiple school to the data to be verified
It tests, successively including numeric field in data volume verification, field contents verification, interfield relational checking, table in the multiple check
The relational checking between table is verified, includes multiple verifications rules in the regulation engine；It is verified to the data to be verified
In the process, automatic to carry out the next item down verification if verification passes through；If a certain verification does not pass through, stores and show check results.
The data verification method and device, by call preset rules engine logarithm factually now verify, due to regulation engine be can be embedding
Enter the component of application program, thus user can be convenient regulation engine is configured and is modified, have great convenience for the user
Use；Meanwhile the data verification method and device can carry out multi-faceted data check to data, find data in time
Quality problems can satisfy the application scenarios more demanding to verification.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
The embodiment of invention for those of ordinary skill in the art without creative efforts, can also basis
The attached drawing of offer obtains other attached drawings.
Fig. 1 is a kind of flow chart of data verification method disclosed by the embodiments of the present invention；
Fig. 2 is the basic flow chart that rule-based engine disclosed by the embodiments of the present invention carries out data check；
Fig. 3 is the flow chart of another data verification method disclosed by the embodiments of the present invention；
Fig. 4 is the flow chart of the third data verification method disclosed by the embodiments of the present invention；
Fig. 5 is the flow chart of the 4th kind of data verification method disclosed by the embodiments of the present invention；
Fig. 6 is a kind of structural schematic diagram of data calibration device disclosed by the embodiments of the present invention；
Fig. 7 is the structural schematic diagram of another data calibration device disclosed by the embodiments of the present invention；
Fig. 8 is the structural schematic diagram of the third data calibration device disclosed by the embodiments of the present invention；
Fig. 9 is the structure chart of the 4th kind of data calibration device disclosed by the embodiments of the present invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall within the protection scope of the present invention.
Fig. 1 is a kind of flow chart of data verification method disclosed by the embodiments of the present invention, shown in Figure 1, data check
Method may include:
Step 101: obtaining data to be verified.
Wherein, the data to be verified can be list data, may include multiple fields, each field in list data
It can be indicated with different types of data, such as numeric data, lteral data, string data.
The data to be verified can be obtained directly from operation system, or have the function of data storage and processing from other
System in obtain.
Step 102: call preset rules engine to carry out multiple check to the data to be verified, in the multiple check according to
It is secondary to verify the relational checking between table including numeric field in data volume verification, field contents verification, interfield relational checking, table,
It include multiple verification rules in the regulation engine.
Wherein, the regulation engine is developed by inference engine, is a kind of component being embedded in the application, is realized
Operational decision making separated from application code, and writes operational decision making using predefined semantic modules.Rule
The process of engines handle data includes: to receive data input, explains business rule, and make operational decision making according to business rule.
It may include multiple verifications rule in the present embodiment, in the regulation engine, the multiple verification rule can be by
User is pre-configured with, and when needs are treated verification data and verified, the regulation engine can be called directly, according to the rule
Then the verification rule in engine verifies the data to be verified.
Step 103: it is carried out in checking procedure to the data to be verified, it is automatic to carry out the next item down if verification passes through
Verification；If a certain verification does not pass through, stores and show check results.
In the present embodiment, multiple check is carried out due to needing to treat verification data, the check results of each single item verification have
By with not by two kinds of situations, when a certain check results be by when, directly carry out the next item down verification；When a certain check results
It to be obstructed out-of-date, then does not continue to carry out subsequent check operation, but directly stores and show check results, it is subsequent to related work
After making the relevant data modifications correction that personnel go wrong verification, the data after correction can be restarted to verify,
It can also be verified from the beginning since last time verifies unacceptable verification.
In the present embodiment, the data verification method is by calling preset rules engine logarithm factually now to verify, due to rule
Then engine be embeddable application program component, therefore user can be convenient regulation engine is configured and is modified, greatly
It is convenient for users to use；Meanwhile the data verification method and device can carry out multi-faceted data check to data,
Discovery data quality problem in time can satisfy the application scenarios more demanding to verification.
Fig. 2 is the basic flow chart that rule-based engine disclosed by the embodiments of the present invention carries out data check, it illustrates
The process of data check may include: as shown in connection with fig. 2 in one specific implementation
S1, beginning；
S2, Table A this month data are obtained first, and pre-processed；
The pretreatment can there are many implementations, in the following embodiments, will divide pretreated different realize
It does not describe in detail, herein no longer excessive description.
The data volume of S3, statistical form A this month data, and the threshold value and history that will be configured in the data volume and regulation engine
Data volume is compared, the early warning if finding differences more than threshold value；
S4, the field contents verification for carrying out Table A data, field contents verification refer to whether the data of field meet specific rule
Model, if whether certain field is not sky, whether phone number length is 11 bit digitals, and whether country code is within the scope of dictionary etc.；
If the verification of S5, field contents passes through, interfield relational checking is carried out, otherwise, storage shows check results.Word
Whether section relational checking meets particular kind of relationship between referring to check field, as field A has to be larger than field B；
If S6, interfield relational checking pass through, carries out numeric field in table and verify, numeric field verification refers to logical in table
It crosses logarithm type-word section and carries out statistics calculating, and dynamic updates threshold value, if current-period data is more than specific threshold value, carry out pre-
It is alert；
S7, relational checking between table is carried out, relational checking, which refers to, between table meets specific relationship between field.
The data of the table needed for needing to introduce other before carrying out this step, and the data have already been through content authentication and
The verification such as interfield relationship；Relational checking, which refers to, between table needs data in two tables to meet the data check of particular kind of relationship；
S8, check results are carried out with standardization processing and shows check results；
S9, end.
In the present embodiment, introduction is made that the verification sequence and verification content of multinomial data check, through this embodiment
Content is introduced it is found that checking process disclosed in the present embodiment can first verify data itself according to from inside to outside, it is errorless after again
Carry out the associated check between data and other data or two tables, logic is reasonable, and can be realized treat verification data into
A plurality of types of verifications of row, meet practical application scene demand.
On the basis of the above disclosed embodiments of the present invention, Fig. 3 is another data school disclosed by the embodiments of the present invention
The flow chart of proved recipe method, shown in Figure 3, data verification method may include:
Step 301: obtaining data to be verified.
Step 302: call preset rules engine to carry out multiple check to the data to be verified, in the multiple check according to
It is secondary to verify the relational checking between table including numeric field in data volume verification, field contents verification, interfield relational checking, table,
It include multiple verification rules in the regulation engine.
Step 303: it is carried out in checking procedure to the data to be verified, it is automatic to carry out the next item down if verification passes through
Verification；If a certain verification does not pass through, stores and show check results.
Step 304: the data to be verified being carried out to automatically update the verification according to data frequency in checking procedure
Threshold value in rule.
Since data can update at any time, the dependent thresholds in corresponding verification rule are also required to the change with data
Change and real-time update, just can guarantee the accuracy of check results in this way.
For example, in the embodiment depicted in figure 2, after S2 step obtains the of that month data that need to be verified, S3 data check step is first
Data volume statistics first is carried out to of that month data, establishes data statistic, and compared with the history average in data statistic
Compared with, when being more than a certain threshold value, progress early warning.In the present embodiment, it can be automatically updated according to data frequency in the verification rule
Threshold value, specific update method can there are many, e.g., it is described according to data frequency automatically update it is described verification rule in threshold
Value may include: to update the school according to history mean value, exponent-weighted average value, logistic regression numerical value and early warning distribution probability
The threshold value in rule is tested, or, carrying out dynamic update by the early warning situation between association analysis difference table.
Fig. 4 is the flow chart of the third data verification method disclosed by the embodiments of the present invention, shown in Figure 4, data school
Proved recipe method may include:
Step 401: obtaining data to be verified.
Step 402: the data to be verified are pre-processed.
In the concrete realization, the data verification method can be realized by verification server.Specifically, rule is utilized
The advantage of engine operational decision making and code dehind, general program are deployed on verification server, verify server first from data
Library obtains data to be verified, and pre-processes to data.
Described to pre-process to the data to be verified, there are many different realizations.
For example, pretreatment includes screening date data, screening field to be verified and generation auxiliary examination information to be verified.
It for another example, is empty situation for data in database table, if it is character string that data, which are empty field type,
Empty data are converted to null character string, null character string and sky difference by pretreatment stage, and sky indicates that there is nothing, null character string list
Show that it is a character string, but there is no content.If there are the interference informations such as space before and after character types data content, rank is pre-processed
Section can be removed, and key message is only retained.
For relational checking between table, the splicing in pretreatment stage carry out table is needed, for example Table A is as follows:
Table B is as follows:
Pretreatment stage splices according to the incidence relation that field B is equal to field 2, because only that the field B content of Table A is as evidence
Field 2 of the content of certificate company in table B exists, and is as follows: after all final splicings
Step 403: call preset rules engine to carry out multiple check to the data to be verified, in the multiple check according to
It is secondary to verify the relational checking between table including numeric field in data volume verification, field contents verification, interfield relational checking, table,
It include multiple verification rules in the regulation engine.
Step 404: it is carried out in checking procedure to the data to be verified, it is automatic to carry out the next item down if verification passes through
Verification；If a certain verification does not pass through, stores and show check results.
In the concrete realization, after obtaining data to be verified, verification server obtains the dynamic threshold automatically updated, and will count
It is further processed according to regulation engine is sent to by interface with dynamic threshold.After regulation engine receives data, acquisition is matched
Set verification rule, will verification rule translate into executable program treat verification data verify.
It verifies between data field, data sheet field, between table and is realized by configuring different rule items, due to regulation engine
The limitation of itself also needs to realize special verifying function using modes such as pre-defined functions.
Using verifying rule needed for natural language flexible configuration in regulation engine, in rule setting, need and dynamic
Threshold value is used cooperatively.General program obtains data automatically according to customer parameter, and automatically selects corresponding rule items and carry out school
It tests, verification types a variety of to multiple tables can verify simultaneously simultaneously.
In the present embodiment, by treat verification data pretreatment so that subsequent verifying work more smoothly efficiently
It executes, the data verification method, by calling preset rules engine logarithm factually now to verify, meanwhile, the data check side
Method and device can carry out multi-faceted data check to data, find data quality problem in time, can satisfy and want to verification
Seek higher application scenarios.
Fig. 5 is the flow chart of the 4th kind of data verification method disclosed by the embodiments of the present invention, as shown in figure 5, data check
Method may include:
Step 501: obtaining data to be verified.
Step 502: call preset rules engine to carry out multiple check to the data to be verified, in the multiple check according to
It is secondary to verify the relational checking between table including numeric field in data volume verification, field contents verification, interfield relational checking, table,
It include multiple verification rules in the regulation engine.
Step 503: standardization processing being carried out to the check results according to preset rules.
Since a data may be related to a plurality of mistake or warning information simultaneously, need to carry out at standardization check results
Reason is to check.Such as in reporting and submitting system, there is a verification rule to have following error description in Z03 verification rule:
" E0305:Z0305 is required item, value range 10≤Z0305≤100；E0307:Z0307 be required item and
" Z0302 investor code " cannot be identical with " Z0307 investee Institution Code "；U0305: investor's right to vote proportional numerical value
It is abnormal, it please verify；".
Specific standardization processing process may include: that check results are saved as text data first, by check results
With branch "；" it is that separator is split as a plurality of check results, corresponding initial data is retained in every check results；Then to every
One check results is that separator is split as error code and error description with colon ": "；Finally by error description according to mistake
Code classification is divided into multiple classifications such as early warning, mistake, interception.It is raising efficiency, using Hadoop to text when overabundance of data
Part is handled, and processing result is stored into multiple files according to classification.
Step 504: it is carried out in checking procedure to the data to be verified, it is automatic to carry out the next item down if verification passes through
Verification；If a certain verification does not pass through, stores and show the check results by standardization processing.
Check results after standardization processing are imported in database.Different classes of error message individually shows, same
The a plurality of error message of data is individually shown respectively, is that Table A check results show example as follows:
Table A error message is presented below:
Table A warning information is presented below:
Relational checking error message is presented below between Table A table B table:
In the present embodiment, describes in detail and standardization processing is carried out to check results, and store and show check results
Content, using disclosed in the present embodiment standardization check results and show relevant programme, be able to use family quickly, clearly
Understand the information that check results are included.
For the various method embodiments described above, for simple description, therefore, it is stated as a series of action combinations, but
Be those skilled in the art should understand that, the present invention is not limited by the sequence of acts described because according to the present invention, certain
A little steps can be performed in other orders or simultaneously.Secondly, those skilled in the art should also know that, it is retouched in specification
The embodiment stated belongs to preferred embodiment, and related actions and modules are not necessarily necessary for the present invention.
Method is described in detail in aforementioned present invention disclosed embodiment, diversified forms can be used for method of the invention
Device realize that therefore the invention also discloses a kind of devices, and specific embodiment is given below and is described in detail.
Fig. 6 is a kind of structural schematic diagram of data calibration device disclosed by the embodiments of the present invention, as shown in fig. 6, data school
Experiment device 60 may include:
Data acquisition module 601, for obtaining data to be verified.
Wherein, the data to be verified can be list data, may include multiple fields, each field in list data
It can be indicated with different types of data, such as numeric data, lteral data, string data.
Multiple check module 602, it is described for calling preset rules engine to carry out multiple check to the data to be verified
Successively including numeric field verification and table in data volume verification, field contents verification, interfield relational checking, table in multiple check
Between relational checking, include that multiple verifications are regular in the regulation engine.
It may include multiple verifications rule in the present embodiment, in the regulation engine, the multiple verification rule can be by
User is pre-configured with, and when needs are treated verification data and verified, the regulation engine can be called directly, according to the rule
Then the verification rule in engine verifies the data to be verified.
Process control module 603, if verification passes through, is controlled for carrying out in checking procedure to the data to be verified
System is automatic to carry out the next item down verification；If a certain verification does not pass through, controls storage and show check results.
In the present embodiment, multiple check is carried out due to needing to treat verification data, the check results of each single item verification have
By with not by two kinds of situations, when a certain check results be by when, directly carry out the next item down verification；When a certain check results
It to be obstructed out-of-date, then does not continue to carry out subsequent check operation, but directly stores and show check results, it is subsequent to related work
After making the relevant data modifications correction that personnel go wrong verification, the data after correction can be restarted to verify,
It can also be verified from the beginning since last time verifies unacceptable verification.
In the present embodiment, the data calibration device is by calling preset rules engine logarithm factually now to verify, due to rule
Then engine be embeddable application program component, therefore user can be convenient regulation engine is configured and is modified, greatly
It is convenient for users to use；Meanwhile the data verification method and device can carry out multi-faceted data check to data,
Discovery data quality problem in time can satisfy the application scenarios more demanding to verification.
Fig. 7 is the structural schematic diagram of another data calibration device disclosed by the embodiments of the present invention, shown in Figure 7, number
May include: according to calibration equipment 70
Data acquisition module 601, for obtaining data to be verified.
Multiple check module 602, it is described for calling preset rules engine to carry out multiple check to the data to be verified
Successively including numeric field verification and table in data volume verification, field contents verification, interfield relational checking, table in multiple check
Between relational checking, include that multiple verifications are regular in the regulation engine.
Process control module 603, if verification passes through, is controlled for carrying out in checking procedure to the data to be verified
System is automatic to carry out the next item down verification；If a certain verification does not pass through, controls storage and show check results.
Threshold value update module 701, for carrying out checking procedure to the data to be verified in the multiple check module
In, the threshold value in the verification rule is automatically updated according to data frequency.
Since data can update at any time, the dependent thresholds in corresponding verification rule are also required to the change with data
Change and real-time update, just can guarantee the accuracy of check results in this way.
Fig. 8 is the structural schematic diagram of the third data calibration device disclosed by the embodiments of the present invention, as shown in figure 8, data
Calibration equipment 80 may include:
Data acquisition module 601, for obtaining data to be verified.
Preprocessing module 801, for being pre-processed to the data to be verified.
Concrete function achieved by the preprocessing module 801 is different.
For example, pretreatment includes screening date data, screening field to be verified and generation auxiliary examination information to be verified.
It for another example, is empty situation for data in database table, if it is character string that data, which are empty field type,
Empty data are converted to null character string, null character string and sky difference by pretreatment stage, and sky indicates that there is nothing, null character string list
Show that it is a character string, but there is no content.If there are the interference informations such as space before and after character types data content, rank is pre-processed
Section can be removed, and key message is only retained.
It can additionally be spliced according to content of the field to two tables.
Multiple check module 602, it is described for calling preset rules engine to carry out multiple check to the data to be verified
Successively including numeric field verification and table in data volume verification, field contents verification, interfield relational checking, table in multiple check
Between relational checking, include that multiple verifications are regular in the regulation engine.
Process control module 603, if verification passes through, is controlled for carrying out in checking procedure to the data to be verified
System is automatic to carry out the next item down verification；If a certain verification does not pass through, controls storage and show check results.
Fig. 9 is the structure chart of the 4th kind of data calibration device disclosed by the embodiments of the present invention, as shown in figure 9, data check
Device 90 may include:
Data acquisition module 601, for obtaining data to be verified.
Multiple check module 602, it is described for calling preset rules engine to carry out multiple check to the data to be verified
Successively including numeric field verification and table in data volume verification, field contents verification, interfield relational checking, table in multiple check
Between relational checking, include that multiple verifications are regular in the regulation engine.
Specification handles module 901, for carrying out standardization processing to the check results according to preset rules.
Specific standardization processing process may include: that check results are saved as text data first, by check results
With branch "；" it is that separator is split as a plurality of check results, corresponding initial data is retained in every check results；Then to every
One check results is that separator is split as error code and error description with colon ": "；Finally by error description according to mistake
Code classification is divided into multiple classifications such as early warning, mistake, interception.It is raising efficiency, using Hadoop to text when overabundance of data
Part is handled, and processing result is stored into multiple files according to classification.
Process control module 603, if verification passes through, is controlled for carrying out in checking procedure to the data to be verified
System is automatic to carry out the next item down verification；If a certain verification does not pass through, controls storage and show the school by standardization processing
Test result.
In the present embodiment, describes in detail and standardization processing is carried out to check results, and store and show check results
Content, using disclosed in the present embodiment standardization check results and show relevant programme, be able to use family quickly, clearly
Understand the information that check results are included.
Each embodiment in this specification is described in a progressive manner, the highlights of each of the examples are with other
The difference of embodiment, the same or similar parts in each embodiment may refer to each other.For device disclosed in embodiment
For, since it is corresponded to the methods disclosed in the examples, so being described relatively simple, related place is said referring to method part
It is bright.
It should also be noted that, herein, relational terms such as first and second and the like are used merely to one
Entity or operation are distinguished with another entity or operation, without necessarily requiring or implying between these entities or operation
There are any actual relationship or orders.Moreover, the terms "include", "comprise" or its any other variant are intended to contain
Lid non-exclusive inclusion, so that the process, method, article or equipment including a series of elements is not only wanted including those
Element, but also including other elements that are not explicitly listed, or further include for this process, method, article or equipment
Intrinsic element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that
There is also other identical elements in process, method, article or equipment including the element.
The step of method described in conjunction with the examples disclosed in this document or algorithm, can directly be held with hardware, processor
The combination of capable software module or the two is implemented.Software module can be placed in random access memory (RAM), memory, read-only deposit
Reservoir (ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or technology
In any other form of storage medium well known in field.
The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present invention.
Various modifications to these embodiments will be readily apparent to those skilled in the art, as defined herein
General Principle can be realized in other embodiments without departing from the spirit or scope of the present invention.Therefore, of the invention
It is not intended to be limited to the embodiments shown herein, and is to fit to and the principles and novel features disclosed herein phase one
The widest scope of cause.
Claims (10)
1. a kind of data verification method characterized by comprising
Obtain data to be verified；
It calls preset rules engine to carry out multiple check to the data to be verified, successively includes data volume in the multiple check
Verification, field contents verification, numeric field verification and relational checking between table, the regulation engine in interfield relational checking, table
In include that multiple verifications are regular；
It is carried out in checking procedure to the data to be verified, it is automatic to carry out the next item down verification if verification passes through；If a certain school
It tests and does not pass through, then store and show check results.
2. data verification method according to claim 1, which is characterized in that further include:
It is carried out in checking procedure to the data to be verified, the threshold in the verification rule is automatically updated according to data frequency
Value.
3. data verification method according to claim 2, which is characterized in that it is described automatically updated according to data frequency it is described
Threshold value in verification rule, comprising:
It is updated in the verification rule according to history mean value, exponent-weighted average value, logistic regression numerical value and early warning distribution probability
Threshold value；
Or,
Dynamic update is carried out by the early warning situation between association analysis difference table.
4. data verification method according to claim 1, which is characterized in that the data to be verified be Table A, to it is described to
It verifies data and carries out the data volume verification, comprising:
Count the data volume of of that month data in the Table A, and the threshold value and history that will be configured in the data volume and regulation engine
Data volume is compared, the early warning if finding differences more than threshold value.
5. data verification method according to claim 1, which is characterized in that carrying out the table to the data to be verified
Relational checking, comprising:
It verifies and whether meets the first preset relation between the field in two tables of data.
6. data verification method according to claim 1, which is characterized in that in the calling preset rules engine to described
Before data to be verified carry out multiple check, further includes:
The data to be verified are pre-processed.
7. data verification method according to claim 6, which is characterized in that described to be located in advance to the data to be verified
Reason, comprising:
Screen date data to be verified, screening field to be verified to the data to be verified and generates auxiliary examination information
Processing.
8. data verification method according to claim 1, which is characterized in that in the calling preset rules engine to described
After data to be verified carry out multiple check, further includes:
Standardization processing is carried out to the check results according to preset rules；
It is then described to store and show check results, comprising:
It stores and shows the check results by standardization processing.
9. a kind of data calibration device characterized by comprising
Data acquisition module, for obtaining data to be verified；
Multiple check module, for calling preset rules engine to carry out multiple check, the multiple school to the data to be verified
It tests successively including numeric field verification in data volume verification, field contents verification, interfield relational checking, table and relationship between table
It verifies, includes multiple verifications rules in the regulation engine；
Process control module, for being carried out in checking procedure to the data to be verified, if verification passes through, control automatically into
The verification of row the next item down；If a certain verification does not pass through, controls storage and show check results.
10. data calibration device according to claim 9, which is characterized in that further include:
Threshold value update module, for being carried out in checking procedure in the multiple check module to the data to be verified, according to
Data frequency automatically updates the threshold value in the verification rule.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910823356.8A CN110515937A (en) | 2019-09-02 | 2019-09-02 | A kind of data verification method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910823356.8A CN110515937A (en) | 2019-09-02 | 2019-09-02 | A kind of data verification method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110515937A true CN110515937A (en) | 2019-11-29 |
Family
ID=68629128
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910823356.8A Pending CN110515937A (en) | 2019-09-02 | 2019-09-02 | A kind of data verification method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110515937A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111258998A (en) * | 2020-01-16 | 2020-06-09 | 北京字节跳动网络技术有限公司 | Data verification method, device, medium and electronic equipment |
| CN111338817A (en) * | 2020-02-21 | 2020-06-26 | 中国农业银行股份有限公司 | Interface preprocessing method and device |
| CN111722940A (en) * | 2020-05-22 | 2020-09-29 | 百富计算机技术（深圳）有限公司 | Message transmission method, terminal equipment and transmission system based on asynchronous serial port |
| CN112015728A (en) * | 2020-09-08 | 2020-12-01 | 浙江惠瀜网络科技有限公司 | Method for automatically checking acquired data |
| CN112231312A (en) * | 2020-10-29 | 2021-01-15 | 山东超越数控电子股份有限公司 | Data quality verification method based on process |
| CN112597194A (en) * | 2020-12-04 | 2021-04-02 | 中广核工程有限公司 | Method and system for calibrating data information of nuclear power instrument |
| CN112597143A (en) * | 2020-12-28 | 2021-04-02 | 中国农业银行股份有限公司 | Data detection method and device and electronic equipment |
| CN112632143A (en) * | 2020-12-30 | 2021-04-09 | 中国农业银行股份有限公司 | Data label generation method and device |
| CN112817953A (en) * | 2021-01-22 | 2021-05-18 | 深圳依时货拉拉科技有限公司 | Data verification method and device, computer equipment and computer-readable storage medium |
| CN112948429A (en) * | 2021-02-02 | 2021-06-11 | 中国工商银行股份有限公司 | Data reporting method, device and equipment |
| CN113641530A (en) * | 2021-06-23 | 2021-11-12 | 地平线(上海)人工智能技术有限公司 | Data processing method and device |
| CN114840295A (en) * | 2022-05-10 | 2022-08-02 | 网易（杭州）网络有限公司 | Information display method, display device, equipment and medium |
| CN115620851A (en) * | 2022-12-19 | 2023-01-17 | 一临云(深圳)科技有限公司 | Data verification method and device, electronic equipment and readable storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106101090A (en) * | 2016-06-07 | 2016-11-09 | 中国建设银行股份有限公司 | Operational approach and rule engine system for regulation engine |
| CN106254045A (en) * | 2016-08-09 | 2016-12-21 | 中国银行股份有限公司 | A kind of data verification method and device |
| CN107103025A (en) * | 2017-01-05 | 2017-08-29 | 北京亚信智慧数据科技有限公司 | A kind of data processing method and data processing platform (DPP) |
| CN107818509A (en) * | 2017-11-24 | 2018-03-20 | 泰康保险集团股份有限公司 | Business datum method of calibration, device, storage medium and electronic equipment |
| CN107908725A (en) * | 2017-11-14 | 2018-04-13 | 中国银行股份有限公司 | A kind of batch data method of calibration, device and system |
| US20180234957A1 (en) * | 2007-06-15 | 2018-08-16 | Samsung Electronics Co., Ltd. | Method and apparatus for allocating and acquiring ack/nack resourcs in a mobile communication system |
| CN110020381A (en) * | 2018-02-23 | 2019-07-16 | 中国平安财产保险股份有限公司 | Method of calibration, device, equipment and computer storage medium based on configuration file |
-
2019
- 2019-09-02 CN CN201910823356.8A patent/CN110515937A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180234957A1 (en) * | 2007-06-15 | 2018-08-16 | Samsung Electronics Co., Ltd. | Method and apparatus for allocating and acquiring ack/nack resourcs in a mobile communication system |
| CN106101090A (en) * | 2016-06-07 | 2016-11-09 | 中国建设银行股份有限公司 | Operational approach and rule engine system for regulation engine |
| CN106254045A (en) * | 2016-08-09 | 2016-12-21 | 中国银行股份有限公司 | A kind of data verification method and device |
| CN107103025A (en) * | 2017-01-05 | 2017-08-29 | 北京亚信智慧数据科技有限公司 | A kind of data processing method and data processing platform (DPP) |
| CN107908725A (en) * | 2017-11-14 | 2018-04-13 | 中国银行股份有限公司 | A kind of batch data method of calibration, device and system |
| CN107818509A (en) * | 2017-11-24 | 2018-03-20 | 泰康保险集团股份有限公司 | Business datum method of calibration, device, storage medium and electronic equipment |
| CN110020381A (en) * | 2018-02-23 | 2019-07-16 | 中国平安财产保险股份有限公司 | Method of calibration, device, equipment and computer storage medium based on configuration file |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111258998A (en) * | 2020-01-16 | 2020-06-09 | 北京字节跳动网络技术有限公司 | Data verification method, device, medium and electronic equipment |
| CN111338817B (en) * | 2020-02-21 | 2023-11-03 | 中国农业银行股份有限公司 | Interface preprocessing method and device |
| CN111338817A (en) * | 2020-02-21 | 2020-06-26 | 中国农业银行股份有限公司 | Interface preprocessing method and device |
| CN111722940A (en) * | 2020-05-22 | 2020-09-29 | 百富计算机技术（深圳）有限公司 | Message transmission method, terminal equipment and transmission system based on asynchronous serial port |
| CN111722940B (en) * | 2020-05-22 | 2024-04-16 | 百富计算机技术（深圳）有限公司 | Message transmission method, terminal equipment and transmission system based on asynchronous serial port |
| CN112015728A (en) * | 2020-09-08 | 2020-12-01 | 浙江惠瀜网络科技有限公司 | Method for automatically checking acquired data |
| CN112231312A (en) * | 2020-10-29 | 2021-01-15 | 山东超越数控电子股份有限公司 | Data quality verification method based on process |
| CN112597194A (en) * | 2020-12-04 | 2021-04-02 | 中广核工程有限公司 | Method and system for calibrating data information of nuclear power instrument |
| CN112597143A (en) * | 2020-12-28 | 2021-04-02 | 中国农业银行股份有限公司 | Data detection method and device and electronic equipment |
| CN112632143A (en) * | 2020-12-30 | 2021-04-09 | 中国农业银行股份有限公司 | Data label generation method and device |
| CN112817953A (en) * | 2021-01-22 | 2021-05-18 | 深圳依时货拉拉科技有限公司 | Data verification method and device, computer equipment and computer-readable storage medium |
| CN112948429A (en) * | 2021-02-02 | 2021-06-11 | 中国工商银行股份有限公司 | Data reporting method, device and equipment |
| CN112948429B (en) * | 2021-02-02 | 2024-04-26 | 中国工商银行股份有限公司 | Data reporting method, device and equipment |
| CN113641530A (en) * | 2021-06-23 | 2021-11-12 | 地平线(上海)人工智能技术有限公司 | Data processing method and device |
| CN114840295A (en) * | 2022-05-10 | 2022-08-02 | 网易（杭州）网络有限公司 | Information display method, display device, equipment and medium |
| CN114840295B (en) * | 2022-05-10 | 2023-09-15 | 网易（杭州）网络有限公司 | Information display method, display device, equipment and medium |
| CN115620851A (en) * | 2022-12-19 | 2023-01-17 | 一临云(深圳)科技有限公司 | Data verification method and device, electronic equipment and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110515937A (en) | A kind of data verification method and device | |
| US7386526B1 (en) | Method of and system for rules-based population of a knowledge base used for medical claims processing | |
| US10482006B2 (en) | System and method for automatically categorizing test cases for model based testing | |
| US8176003B2 (en) | Automatic designation of XBRL taxonomy tags | |
| US9390176B2 (en) | System and method for recursively traversing the internet and other sources to identify, gather, curate, adjudicate, and qualify business identity and related data | |
| US20160162456A1 (en) | Methods for generating natural language processing systems | |
| US20170039062A1 (en) | Scalable continuous integration and delivery systems and methods | |
| EP3640814A1 (en) | User-friendly explanation production using generative adversarial networks | |
| US7822621B1 (en) | Method of and system for populating knowledge bases using rule based systems and object-oriented software | |
| US10977290B2 (en) | Transaction categorization system | |
| WO2014070070A1 (en) | Method, apparatus and computer program for detecting deviations in data sources | |
| CN109960719A (en) | A kind of document handling method and relevant apparatus | |
| EP3985526A1 (en) | System and method for auto-mapping source and target data attributes based on metadata information | |
| CN114564624A (en) | Feature matching rule construction method, feature matching device, feature matching equipment and feature matching medium | |
| CN109284975A (en) | Policy information batch modification method, apparatus, computer equipment and storage medium | |
| CN109657240A (en) | Determine the method, apparatus, equipment and medium of fault type | |
| US10705810B2 (en) | Automatic code generation | |
| CN109002355B (en) | Distribution method, device and equipment for processing requests | |
| CN110516258A (en) | Data verification method and device, storage medium, electronic device | |
| CN109324963A (en) | The method and terminal device of automatic test profitable result | |
| US20220138584A1 (en) | Artificial content identification using subset scanning over generative model activations | |
| CN113641823A (en) | Text classification model training method, text classification device, text classification equipment and medium | |
| CN113127359A (en) | Method and device for obtaining test data | |
| CN112783775A (en) | Special character input testing method and device | |
| US20220374914A1 (en) | Regulatory obligation identifier |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191129 |
|
| RJ01 | Rejection of invention patent application after publication |