CN107918651B - Method and system for detecting data consistency - Google Patents
Method and system for detecting data consistency Download PDFInfo
- Publication number
- CN107918651B CN107918651B CN201711122754.4A CN201711122754A CN107918651B CN 107918651 B CN107918651 B CN 107918651B CN 201711122754 A CN201711122754 A CN 201711122754A CN 107918651 B CN107918651 B CN 107918651B
- Authority
- CN
- China
- Prior art keywords
- data
- detected
- line
- file
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0751—Error or fault detection not based on redundancy
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computer Security & Cryptography (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a method and a system for detecting data consistency, wherein the method comprises the following steps: acquiring all row data of a file to be detected, and displaying all row data and a pre-stored extraction rule; receiving a target extraction rule generated according to all the line data and a pre-stored extraction rule and a detection rule generated according to all the line data; extracting first data to be detected and second data to be detected for each line of data according to a target extraction rule; and carrying out consistency detection on the first data to be detected and the second data to be detected according to a detection rule, and determining the consistency of the line data according to a detection result. The invention improves the detection efficiency and reduces the time consumed by detection.
Description
Technical Field
The invention belongs to the technical field of data processing, and particularly relates to a method and a system for detecting data consistency.
Background
The mobile CRM data consistency platform judges the consistency of files stored by the platform, and when invalid data in the files are inconsistent, the platform cannot judge the consistency. When the amount of the stored files is large, the method in the prior art firstly splits the files and then compares the split files. A large amount of time is consumed during splitting, and the judgment efficiency is low; and the individual file cannot be split, so that the judgment on the consistency of the individual file cannot be completed.
Disclosure of Invention
The technical problem to be solved by the present invention is to provide a method and a system for detecting data consistency, aiming at the defects of the prior art.
The technical scheme for solving the technical problems is as follows: a method of detecting data consistency, comprising:
acquiring all row data of a file to be detected, and displaying all row data and a pre-stored extraction rule;
receiving a target extraction rule generated according to all the line data and a pre-stored extraction rule and a detection rule generated according to all the line data;
extracting first data to be detected and second data to be detected for each line of data according to a target extraction rule;
and carrying out consistency detection on the first data to be detected and the second data to be detected according to a detection rule, and determining the consistency of the line data according to a detection result.
The invention has the beneficial effects that: through showing the extraction rule that prestores and all row data of the file that awaits measuring that acquire to can receive the target extraction rule that generates in view of the above, extract each row of line data according to the target extraction rule, thereby need not split the file and can acquire corresponding data that await measuring, avoided the time that the split consumed, also avoided the unable circumstances that detects that the unable split of file caused, improved detection efficiency. And detecting corresponding data to be detected through a detection rule generated by all the line data, thereby determining the consistency of the line data of each line.
Another technical solution of the present invention for solving the above technical problems is as follows: a system for detecting data consistency, comprising:
the acquisition module is used for acquiring all the line data of the file to be detected and displaying all the line data and the pre-stored extraction rules;
the receiving module is used for receiving a target extraction rule generated according to all the line data and a pre-stored extraction rule and a detection rule generated according to all the line data;
the processing module is used for extracting first data to be detected and second data to be detected from each line of data according to a target extraction rule;
and the detection module is used for carrying out consistency detection on the first data to be detected and the second data to be detected according to the detection rule and determining the consistency of the line data according to the detection result.
The invention has the beneficial effects that: through showing the extraction rule that prestores and all row data of the file that awaits measuring that acquire to can receive the target extraction rule that generates in view of the above, extract each row of line data according to the target extraction rule, thereby need not split the file and can acquire corresponding data that await measuring, avoided the time that the split consumed, also avoided the unable circumstances that detects that the unable split of file caused, improved detection efficiency. And detecting corresponding data to be detected through a detection rule generated by all the line data, thereby determining the consistency of the line data of each line.
Drawings
Fig. 1 is a schematic flowchart illustrating a method for detecting data consistency according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a method for detecting data consistency according to another embodiment of the present invention;
fig. 3 is a schematic structural diagram of a system for detecting data consistency according to an embodiment of the present invention.
Detailed Description
The principles and features of this invention are described below in conjunction with the following drawings, which are set forth by way of illustration only and are not intended to limit the scope of the invention.
As shown in fig. 1, a method for detecting data consistency according to an embodiment of the present invention includes:
s110, acquiring all line data of the file to be detected, and displaying all line data and pre-stored extraction rules.
And S120, receiving a target extraction rule generated according to all the line data and the pre-stored extraction rule and a detection rule generated according to all the line data.
Specifically, the target extraction rule may be selected from pre-stored extraction rules according to the data characteristics of all the line data, or may be generated according to the data characteristics of all the line data. The data characteristics may include: data storage location, data storage manner, etc. The new target extraction rules can be stored together with the pre-stored extraction rules for use when encountering the files to be detected with the same data characteristics.
And S130, extracting the first data to be detected and the second data to be detected for each line of data according to a target extraction rule.
And S140, carrying out consistency detection on the first data to be detected and the second data to be detected according to the detection rule, and determining the consistency of the line data according to the detection result.
According to the method for detecting the data consistency, the pre-stored extraction rule and all the obtained row data of the file to be detected are displayed, so that the target extraction rule generated based on the pre-stored extraction rule can be received, each row of data is extracted according to the target extraction rule, the corresponding data to be detected can be obtained without splitting the file, time consumed by splitting is avoided, the condition that the file cannot be split to cause the failure in detection is avoided, and the detection efficiency is improved. And detecting corresponding data to be detected through a detection rule generated by all the line data, thereby determining the consistency of the line data of each line.
Optionally, as shown in fig. 2, in the embodiment of the present invention, the S130 includes:
s231, extracting a first target field and a second target field for each row of data according to a target extraction rule.
S232, respectively acquiring first data to be detected and second data to be detected from the first target field and the second target field.
In this embodiment, the number of the first target fields and the number of the second target fields may be at least one, and the number of the first target fields is the same as the number of the second target fields.
The embodiment of the invention is illustrated by a specific example: for each row of data, the first field and the third field are first target fields, and the seventh field and the ninth field are second target fields, wherein the first field and the seventh field describe data of the same object, and the third field and the ninth field describe another data of the same object. The method comprises the steps of firstly extracting a first field and a seventh field according to a target extraction rule, respectively obtaining corresponding first data to be detected and second data to be detected from the first field and the seventh field for detection, determining consistency, then extracting a third field and a ninth field according to the target extraction rule, respectively obtaining corresponding first data to be detected and second data to be detected from the third field and the ninth field for detection, and determining consistency.
In the embodiment, the first target field and the second target field are extracted from each row of data according to the target extraction rule, and the first data to be detected and the second data to be detected are respectively obtained from the first target field and the second target field, so that the corresponding data to be detected can be obtained without splitting a file, the time consumed by splitting is avoided, the condition that the file cannot be split to cause the incapability of detection is also avoided, and the detection efficiency is improved.
Optionally, in an embodiment of the present invention, the S140 includes:
and judging whether the first data to be detected is the same as the second data to be detected or not or whether the first data to be detected is in an inclusion relationship with the second data to be detected, determining that the data of the line data is consistent when the first data to be detected is the same as the second data to be detected or the first data to be detected is in the inclusion relationship with the second data to be detected, and otherwise, determining that the data of the line data is inconsistent.
In the above embodiment, it is determined whether the first data to be detected and the second data to be detected are the same or not, or whether the first data to be detected and the second data to be detected are in an inclusion relationship, if yes, the line data are consistent, otherwise, the line data are inconsistent, so that the line data consistency is detected, and the detection mode of the data consistency is enriched.
Optionally, in an embodiment of the present invention, the method further includes:
and when the data of the line data are inconsistent, storing the first to-be-detected data and the second to-be-detected data of the line data to a first preset position.
Specifically, the first to-be-detected data and the second to-be-detected data of the line data with inconsistent data are stored in the first preset position, so that the inconsistent line data can be quickly acquired from the first preset position.
And when the line data can not be judged to be consistent, storing the line data to a second preset position.
Specifically, when it cannot be determined whether the line data are consistent, the generated target extraction rule and detection rule are not applicable to the line data, so that the line data are stored in the second preset position, the line data in the second preset position are detected by adopting other existing extraction rules and detection rules, and then the consistency of the line data in the second preset position is determined. The reasons for the failure to determine whether the line data is consistent may include: each field in the row data is separated by a space, but one field contains a space; or the logic of the row data is particularly complex, etc.
In the above embodiment, when the row data are inconsistent, the inconsistent row data are stored in the first preset position, so that a user can determine the problem according to the data in the preset position, the experience of the user is improved, and when the row data cannot be judged to be consistent, the row data are stored in the second preset position, so that the row data in the second preset position are detected by using the existing detection method, and the consistency of all the row data is detected.
Optionally, in the embodiment of the present invention, acquiring all the line data of the file to be detected includes:
and acquiring the file to be detected.
Specifically, files needing to be determined for data consistency are stored in the mobile CRM data consistency platform, and the files to be detected are obtained from the mobile CRM data consistency platform, wherein the files to be detected are the files needing to be determined for data consistency. The file to be detected can be obtained according to the storage path of the file to be detected in the mobile CRM data consistency platform. The files to be detected record data of the same service.
And extracting all the line data in the file to be detected from the file to be detected by using the file stream.
In the embodiment, all the line data in the file to be detected are extracted from the file to be detected by using the file stream through obtaining the file to be detected, so that all the line data are quickly obtained, and the detection efficiency is improved.
As shown in fig. 3, a system for detecting data consistency according to an embodiment of the present invention includes:
and the acquisition module is used for acquiring all the line data of the file to be detected and displaying all the line data and the pre-stored extraction rules.
And the receiving module is used for receiving the target extraction rule generated according to all the line data and the pre-stored extraction rule and the detection rule generated according to all the line data.
And the processing module is used for extracting the first data to be detected and the second data to be detected from each line of data according to the target extraction rule.
And the detection module is used for carrying out consistency detection on the first data to be detected and the second data to be detected according to the detection rule and determining the consistency of the line data according to the detection result.
According to the system for detecting the data consistency, the pre-stored extraction rule and all the obtained row data of the file to be detected are displayed, so that the target extraction rule generated based on the pre-stored extraction rule can be received, each row of data is extracted according to the target extraction rule, the corresponding data to be detected can be obtained without splitting the file, time consumed by splitting is avoided, the condition that the file cannot be split to cause the failure in detection is avoided, and the detection efficiency is improved. And detecting corresponding data to be detected through a detection rule generated by all the line data, thereby determining the consistency of the line data of each line.
Optionally, in the embodiment of the present invention, the processing module is specifically configured to:
and extracting a first target field and a second target field for each row of data according to a target extraction rule.
And respectively acquiring first data to be detected and second data to be detected from the first target field and the second target field.
In the embodiment, the first target field and the second target field are extracted from each row of data according to the target extraction rule, and the first data to be detected and the second data to be detected are respectively obtained from the first target field and the second target field, so that the corresponding data to be detected can be obtained without splitting a file, the time consumed by splitting is avoided, the condition that the file cannot be split to cause the incapability of detection is also avoided, and the detection efficiency is improved.
Optionally, in the embodiment of the present invention, the detection module is specifically configured to:
and judging whether the first data to be detected is the same as the second data to be detected or not or whether the first data to be detected is in an inclusion relationship with the second data to be detected, determining that the data of the line data is consistent when the first data to be detected is the same as the second data to be detected or the first data to be detected is in the inclusion relationship with the second data to be detected, and otherwise, determining that the data of the line data is inconsistent.
In the above embodiment, it is determined whether the first data to be detected and the second data to be detected are the same or not, or whether the first data to be detected and the second data to be detected are in an inclusion relationship, if yes, the line data are consistent, otherwise, the line data are inconsistent, so that the line data consistency is detected, and the detection mode of the data consistency is enriched.
Optionally, in the embodiment of the present invention, the detection module is further specifically configured to:
and when the data of the line data are inconsistent, storing the first to-be-detected data and the second to-be-detected data of the line data to a first preset position.
And when the line data can not be judged to be consistent, storing the line data to a second preset position.
In the above embodiment, when the row data are inconsistent, the inconsistent row data are stored in the first preset position, so that a user can determine the problem according to the data in the preset position, the experience of the user is improved, and when the row data cannot be judged to be consistent, the row data are stored in the second preset position, so that the row data in the second preset position are detected by using the existing detection method, and the consistency of all the row data is detected.
Optionally, in the embodiment of the present invention, the obtaining module is specifically configured to:
and acquiring the file to be detected.
And extracting all the line data of the file to be detected from the file to be detected by using the file stream.
In the embodiment, all the line data in the file to be detected are extracted from the file to be detected by using the file stream through obtaining the file to be detected, so that all the line data are quickly obtained, and the detection efficiency is improved.
The functions executed by the modules of the system have been described in detail in the method for detecting data consistency in the above embodiment, and are not described again here.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (6)
1. A method for detecting data consistency, comprising:
acquiring all line data of a file to be detected, and displaying all the line data and a prestored extraction rule;
receiving a target extraction rule generated according to all the line data and the prestored extraction rule and a detection rule generated according to all the line data;
extracting first data to be detected and second data to be detected for each line of data according to the target extraction rule;
the consistency detection is carried out on the first data to be detected and the second data to be detected according to the detection rule, and the consistency of the row of data is determined according to the detection result, which specifically comprises the following steps:
judging whether the first data to be detected is the same as the second data to be detected or not or whether the first data to be detected is in an inclusion relationship with the second data to be detected, and determining that the data of the line data is consistent when the first data to be detected is the same as the second data to be detected or is in the inclusion relationship with the second data to be detected, otherwise, determining that the data of the line data is inconsistent;
when the data of the line data are inconsistent, storing first data to be detected and second data to be detected of the line data to a first preset position;
and when the line data can not be judged to be consistent, storing the line data to a second preset position.
2. The method according to claim 1, wherein the extracting the first data to be detected and the second data to be detected according to the target extraction rule for each row of data comprises:
extracting a first target field and a second target field for each row of data according to the target extraction rule;
and respectively acquiring first data to be detected and second data to be detected from the first target field and the second target field.
3. The method according to any one of claims 1-2, wherein the acquiring all line data of the file to be detected comprises: acquiring a file to be detected; and extracting all the line data in the file to be detected from the file to be detected by using the file stream.
4. A system for detecting data consistency, comprising:
the acquisition module is used for acquiring all line data of the file to be detected and displaying all the line data and a pre-stored extraction rule;
the receiving module is used for receiving a target extraction rule generated according to all the line data and the pre-stored extraction rule and a detection rule generated according to all the line data;
the processing module is used for extracting first data to be detected and second data to be detected from each line of data according to the target extraction rule;
the detection module is configured to perform consistency detection on the first data to be detected and the second data to be detected according to the detection rule, and determine consistency of the line data according to a detection result, and specifically configured to:
judging whether the first data to be detected is the same as the second data to be detected or not or whether the first data to be detected is in an inclusion relationship with the second data to be detected, and determining that the data of the line data is consistent when the first data to be detected is the same as the second data to be detected or is in the inclusion relationship with the second data to be detected, otherwise, determining that the data of the line data is inconsistent;
when the data of the line data are inconsistent, storing first data to be detected and second data to be detected of the line data to a first preset position;
and when the line data can not be judged to be consistent, storing the line data to a second preset position.
5. The system of claim 4, wherein the processing module is specifically configured to:
extracting a first target field and a second target field for each row of data according to the target extraction rule;
and respectively acquiring first data to be detected and second data to be detected from the first target field and the second target field.
6. The system according to any one of claims 4 to 5, wherein the acquisition module is specifically configured to:
acquiring a file to be detected; and extracting all the line data of the file to be detected from the file to be detected by using a file stream.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711122754.4A CN107918651B (en) | 2017-11-14 | 2017-11-14 | Method and system for detecting data consistency |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711122754.4A CN107918651B (en) | 2017-11-14 | 2017-11-14 | Method and system for detecting data consistency |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107918651A CN107918651A (en) | 2018-04-17 |
| CN107918651B true CN107918651B (en) | 2021-12-24 |
Family
ID=61896221
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711122754.4A Active CN107918651B (en) | 2017-11-14 | 2017-11-14 | Method and system for detecting data consistency |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107918651B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111177165B (en) * | 2019-12-23 | 2020-12-18 | 拉扎斯网络科技（上海）有限公司 | Method, device and equipment for detecting data consistency |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104731858A (en) * | 2015-01-29 | 2015-06-24 | 无锡科技职业学院 | Multi-client-side parallel detection database content consistency system and method |
| CN105095507A (en) * | 2015-08-31 | 2015-11-25 | 浪潮(北京)电子信息产业有限公司 | DB2 database migration method, device and system |
| CN105989194A (en) * | 2015-03-20 | 2016-10-05 | 国际商业机器公司 | Method and system of table data comparison |
| CN106484908A (en) * | 2016-10-31 | 2017-03-08 | 成都轻松益家电子商务有限公司 | A kind of multisystem comparison of data consistency method based on middleware |
| CN107066601A (en) * | 2017-04-20 | 2017-08-18 | 北京古盘创世科技发展有限公司 | File contrasts management method and system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10747776B2 (en) * | 2012-12-04 | 2020-08-18 | International Business Machines Corporation | Replication control using eventually consistent meta-data |
| US9778973B2 (en) * | 2015-10-28 | 2017-10-03 | International Business Machines Corporation | Early diagnosis of hardware, software or configuration problems in data warehouse system utilizing grouping of queries based on query parameters |
-
2017
- 2017-11-14 CN CN201711122754.4A patent/CN107918651B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104731858A (en) * | 2015-01-29 | 2015-06-24 | 无锡科技职业学院 | Multi-client-side parallel detection database content consistency system and method |
| CN105989194A (en) * | 2015-03-20 | 2016-10-05 | 国际商业机器公司 | Method and system of table data comparison |
| CN105095507A (en) * | 2015-08-31 | 2015-11-25 | 浪潮(北京)电子信息产业有限公司 | DB2 database migration method, device and system |
| CN106484908A (en) * | 2016-10-31 | 2017-03-08 | 成都轻松益家电子商务有限公司 | A kind of multisystem comparison of data consistency method based on middleware |
| CN107066601A (en) * | 2017-04-20 | 2017-08-18 | 北京古盘创世科技发展有限公司 | File contrasts management method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107918651A (en) | 2018-04-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9666161B2 (en) | Method and system for replacing theme of a mobile terminal on the basis of a user's clothes color | |
| EP2161678A3 (en) | Image processing apparatus, imaging apparatus, image processing method, and program | |
| CN107948255B (en) | The method for pushing and computer readable storage medium of APP | |
| EP4270355A3 (en) | Method for processing hazard reports from vehicles | |
| EP2685404A3 (en) | Method and system for determining image similarity | |
| CN107329398A (en) | Car-mounted terminal time keeping system and method | |
| EP2458335A3 (en) | Road estimation device and method for estimating road | |
| RU2016150428A (en) | FILLING IN THE USER CONTACTS | |
| RU2008113194A (en) | EXPERT ANALYSIS OF THE SYSTEM AND GRAPHIC DISPLAY OF ROUTES FOR INCREASING PRIVILEGES IN A COMPUTER ENVIRONMENT | |
| CN106412573A (en) | Method and device for detecting lens stain | |
| CN103150631A (en) | Image-matching-based automatic article management method and system | |
| CN105117644B (en) | Acquire Android plug-in program method and system | |
| CA2519908A1 (en) | Target detection improvements using temporal integrations and spatial fusion | |
| EP2458332A3 (en) | Road estimation device and method for estimating road | |
| CN111124888A (en) | Method and device for generating recording script and electronic device | |
| CN103315700A (en) | Method for vision detection by using mobile intelligent terminal and terminal | |
| EP2458330A3 (en) | Road estimation device and method for estimating road | |
| CN102611691A (en) | Method, system and gateway device for detecting phishing websites | |
| CN107918651B (en) | Method and system for detecting data consistency | |
| CN108629310B (en) | Engineering management supervision method and device | |
| CN103049530B (en) | Deeply merge video checking method and system | |
| US10368331B2 (en) | Method for registering on voice-over-LTE network, device and terminal, and computer storage medium | |
| CN110619211A (en) | Malicious software identification method, system and related device based on dynamic characteristics | |
| CN107145412B (en) | Hard disk state detection method and detection device | |
| CN110166769B (en) | Method, device and system for detecting output dislocation of camera module and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |