CN107301203B - Mass data comparison method and system - Google Patents
Mass data comparison method and system Download PDFInfo
- Publication number
- CN107301203B CN107301203B CN201710390393.5A CN201710390393A CN107301203B CN 107301203 B CN107301203 B CN 107301203B CN 201710390393 A CN201710390393 A CN 201710390393A CN 107301203 B CN107301203 B CN 107301203B
- Authority
- CN
- China
- Prior art keywords
- data
- comparison
- files
- reference data
- mass
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2474—Sequence data queries, e.g. querying versioned data
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a comparison method and a comparison system for mass data, wherein the method comprises the following steps: acquiring mass data files to be compared; sequencing the mass data files according to a pre-stored quick sequencing algorithm to obtain a plurality of subdata files; and carrying out data consistency comparison on the plurality of subdata files. According to the comparison method and system for the mass data, provided by the invention, the mass data file is divided into the plurality of subdata files, and the plurality of subdata files are compared respectively, so that the phenomenon that the conventional single machine has limited I/O and cannot perform multi-thread comparison is effectively solved, and the comparison efficiency of the data can be improved.
Description
Technical Field
The invention relates to the field of data processing, in particular to a mass data comparison method and system.
Background
The data comparison refers to comparing two or more different data, quickly finding out detailed differences of the different data and effectively processing the differences.
The conventional comparison method is mainly to arrange data to be compared in a certain order, each data record has two fields, the first field is an index field, and the second field is an attribute field, for example, the following two groups of data to be compared are:
wherein, the letter is an index field and the number is an attribute field.
Then, the alignment method is as follows: and obtaining data of the first row for comparison, if the index field and the attribute field are the same, continuing to compare the next row, if the index field of the second row is the same, but the attribute field is different, recording the index value B of the index field and the numerical values 2 and 0 of the attribute field, continuing to compare the next row, and so on.
The traditional comparison method is that large files are imported into a database, then the large files are sequenced in the database, then data are read from the database and compared in an internal memory, the method has too much pressure on the database, the database is easy to falsely die when the quantity of concurrent data is large, and the comparison efficiency can not meet the requirement all the time because the I/O of one host is limited.
Disclosure of Invention
The invention aims to solve the technical problem of providing a method and a system for comparing mass data aiming at the defects of the prior art.
The technical scheme for solving the technical problems is as follows:
a comparison method of mass data comprises the following steps:
step 1, acquiring mass data files to be compared;
step 2, sequencing the mass data files according to a pre-stored quick sequencing algorithm to obtain a plurality of subdata files;
and 3, comparing the data consistency of the plurality of subdata files.
The invention has the beneficial effects that: according to the comparison method for the mass data, provided by the invention, the mass data file is divided into the plurality of subdata files, and the plurality of subdata files are compared respectively, so that the phenomenon that the conventional single machine has limited I/O and cannot perform multi-thread comparison is effectively solved, and the comparison efficiency of the data can be improved.
On the basis of the technical scheme, the invention can be further improved as follows.
Further, step 2 specifically includes:
step 2.1, selecting N data with different values from the mass data file, and taking the values as first reference data;
2.2, sequentially arranging the first reference data from large to small or from small to large;
step 2.3, comparing and classifying the data in the mass data file with the first reference data in sequence to respectively obtain a data set smaller than the minimum first reference data, a data set larger than the maximum first reference data and a data set between any two adjacent first reference data, and putting the data with the same value as the first reference data in the mass data file into a previous data set adjacent to the first reference data or a next data set adjacent to the first reference data;
and 2.4, storing the classified data set into a file form to obtain N +1 sub-data files.
Further, step 3 specifically includes:
step 3.1, extracting the index field and the attribute field of the data in the sub data file, and acquiring the source file information of the data;
step 3.2, selecting all data with different index fields from the subdata files as second reference data;
step 3.3, comparing the attribute field of the data to be compared, which is the same as the index field of the second reference data but different in source file information, with the attribute field of the second reference data in sequence to obtain a comparison result;
and 3.4, outputting and storing the data to be compared according to the comparison result.
Further, in step 3, a plurality of comparison units respectively perform data consistency comparison on the plurality of sub-data files.
Further, the alignment method further comprises:
and 4, when any comparison unit fails, sending the data compared by the current comparison unit to other comparison units for comparison again.
Another technical solution of the present invention for solving the above technical problems is as follows:
a mass data comparison system comprises:
the acquisition module is used for acquiring mass data files to be compared;
the sorting module is used for sorting the mass data files according to a pre-stored quick sorting algorithm to obtain a plurality of subdata files;
and the comparison module is used for carrying out data consistency comparison on the plurality of subdata files.
Further, the ranking module includes:
the first selection unit is used for selecting N data with different values from the mass data files and taking the values as first reference data;
the sorting unit is used for sequentially sorting the first reference data from large to small or from small to large;
the classification unit is used for comparing and classifying the data in the mass data file with the first reference data in sequence to respectively obtain a data set smaller than the minimum first reference data, a data set larger than the maximum first reference data and a data set between any two adjacent first reference data, and putting the data with the same value as the first reference data in the mass data file into a previous data set adjacent to the first reference data or a next data set adjacent to the first reference data;
and the storage unit is used for storing the classified data set into a file form to obtain N +1 sub-data files.
Further, the alignment module comprises:
the extracting unit is used for extracting the index field and the attribute field of the data in the sub data file and acquiring the source file information of the data;
the second selection unit is used for selecting all data with different index fields from the sub data files as second reference data;
the comparison unit is used for sequentially comparing the attribute field of the data to be compared, which is the same as the index field of the second reference data but has different source file information, with the attribute field of the second reference data to obtain a comparison result;
and the output unit is used for outputting and storing the data to be compared according to the comparison result.
Further, the comparison module is specifically configured to perform data consistency comparison on the plurality of sub-data files through the plurality of comparison units respectively.
Further, the alignment system further comprises:
and the detection module is used for sending the data compared by the current comparison unit to other comparison units for comparison again when any comparison unit has a fault.
Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
Fig. 1 is a schematic flow chart of a comparison method for mass data according to an embodiment of the present invention;
fig. 2 is a flowchart of a mass data comparison method according to another embodiment of the present invention;
fig. 3 is a structural diagram of a comparison system for mass data according to another embodiment of the present invention;
fig. 4 is a schematic structural diagram of a comparison module provided in the embodiment shown in fig. 3.
Detailed Description
The principles and features of this invention are described below in conjunction with the following drawings, which are set forth by way of illustration only and are not intended to limit the scope of the invention.
As shown in fig. 1, a schematic flow chart of a method for comparing mass data according to an embodiment of the present invention is provided, where the method includes the following steps:
s101, acquiring mass data files to be compared;
s102, sequencing the mass data files according to a pre-stored quick sequencing algorithm to obtain a plurality of sub data files;
s103, carrying out data consistency comparison on the plurality of subdata files.
According to the comparison method for the mass data, the mass data file is divided into the plurality of subdata files, and the plurality of subdata files are compared respectively, so that the problems that the existing single machine is limited in I/O and cannot perform multi-thread comparison are effectively solved, and the comparison efficiency of the data can be improved.
In another embodiment, as shown in fig. 2, a flowchart of a method for comparing mass data according to another embodiment of the present invention is provided, where the method includes the following steps:
s201, acquiring data files A and B to be compared;
s202, collectively selecting 100 data with different values from the data files a and B, and taking the values of the 100 data with different values as reference data, it should be noted that how many reference data are specifically selected, which may be determined according to actual requirements, and which is not a fixed value, where 100 reference data are selected, which is only for convenience of describing the present embodiment, and is not the number of the reference data that must be selected;
s203, arranging the 100 data in sequence from small to big;
s204, comparing the data in the data files a and B with the 100 reference data respectively, and dividing the total data in the data files a and B into 101 parts, which is illustrated by taking a simplified example as an example, for example, if there are 3 data in the data file a, 1, and 3 respectively, and 4 data in the data file B, 2, 3, 4, and 5 respectively, then 2 and 4 can be selected as the reference data, where the reference data 2 is from the data file a, and the reference data 4 is from the data file B, and then by comparing the data 1, 2, 3, 4, and 5 in the data files a and B with the reference data 2 and 4 respectively, 3 data sets can be obtained, where the data set [1, 1] is less than 2, the data set [3, 3] is greater than 2 and less than 4, and the data set [5] is greater than 4, then, data 2 and 4 with the same value as the reference data 2 and 4 are respectively placed into the previous data set adjacent to the data 2, that is, the data 2 is placed into the data set smaller than 2, and the data 4 is placed into the data set larger than 2 and smaller than 4, so as to obtain final data sets [1, 1, 2], [3, 3, 4] and [5], thus completing the splitting of the data files A and B, in this simplified example, data 5 from the data file B can also be selected as the reference data, so that the data set larger than the reference data 5 is an empty set;
s205, storing the 101 data sets obtained after classification into a file form to obtain 101 sub-data files;
s206, extracting an index field and an attribute field of data in the sub-data file, and obtaining source file information of the data, where the index field records index information of the data, the attribute field records a data body, and the source file information of the data refers to a source of the data, in this embodiment, the source of the data is a data file a or a data file B, for example, a data form in the data file a may be < index2, 2014070512435, a >, and a data form in the data file B may be < index2, 2014070512435, B >, where index2 of the two pieces of data is the index field, 2014070512435 is the attribute field, a indicates that the source file is the data file a, and B indicates that the source file is the data file B;
s207, selecting comparison data from the 101 sub-data files respectively, wherein the index fields of the selected comparison data are different in each sub-data file, and a simplified example is taken as an example, for example, there are 2 data in the sub-data file 1, which are respectively < index2, 5, A > and < index2, 5, B >, and there are 4 data in the sub-data file 2, which are respectively < index2, 6, A >, < index2, 6, B >, < index3, 6, A > and < index3, 6, B >, then in the sub-data file 1, < index2, 5, A > or < index2, 5, B > can be selected as the comparison data, and in the sub-data file 2, < index2, 6, A > and < index3, 6, A > can be selected as the comparison data, or < index2, 6, A > and < index 637, 6, B > and 3 can be selected as the comparison data, 6, a > is used as comparison data, or < index2, 6, B > and < index3, 6, B > are selected as comparison data, and only one data cannot be selected as comparison data, for example, only < index2, 6, a > is selected as comparison data, which causes that the data with the index field of index3 cannot be compared, and at the same time, the index fields of < index2, 6, a > and < index2, 6, B > are all index2 because the index fields are the same, and if selected, the data with the index field of index3 cannot be compared, that is, the index fields of the comparison data are different from each other in any sub-data file;
s208, sequentially comparing the data in each sub-data file, which is the same as the index field of the selected comparison data but different from the source file, to obtain a comparison result, where the comparison result is whether the attribute fields of the two data are the same, for example, if the attribute fields of the two data are the same and the values are both 5, < index2, 5, a > and < index2, 5, B > are compared, the comparison result is obtained, and the records of the two data are the same;
and S209, outputting the obtained comparison result, wherein the two pieces of data are the same and are output to a single folder, and the two pieces of data records are output to another single folder if the two pieces of data have different attribute fields.
In another embodiment, a method for comparing data when a plurality of data files to be compared are provided.
Taking the comparison method shown in fig. 2 as an example, when there are a plurality of data files to be compared, after each data file is divided into a plurality of subdata files in a sequence, the data from different data files are compared with the comparison data in sequence.
In another embodiment, as shown in fig. 3, a structure diagram of a mass data comparison system according to another embodiment of the present invention is provided, where the system includes:
an obtaining module 310, configured to obtain a mass data file to be compared;
the sorting module 320 is configured to sort the mass data files according to a pre-stored fast sorting algorithm to obtain a plurality of sub data files;
the comparison module 330 is configured to perform data consistency comparison on the multiple sub data files.
Preferably, the sorting module 320 includes:
a first selecting unit 321, configured to select, from the mass data file, N data with different values, and use the values as first reference data;
a sorting unit 322, configured to sequentially sort the first reference data in descending order or descending order;
a classifying unit 323, configured to compare and classify data in the mass data file with the first reference data in sequence, to obtain a data set smaller than the minimum first reference data, a data set larger than the maximum first reference data, and a data set between any two adjacent first reference data, and put data in the mass data file having the same value as the first reference data into a previous data set adjacent to the first reference data, or put a next data set adjacent to the first reference data;
the storage unit 324 is configured to store the classified data set into a file format, so as to obtain N +1 sub-data files.
Preferably, the alignment module 330 includes:
the extracting unit 331 is configured to extract an index field and an attribute field of data in the sub data file, and obtain source file information of the data;
a second selecting unit 332, configured to select at least one data from the sub-data files as a second reference data, where index fields of the plurality of second reference data are different when the plurality of second reference data are multiple;
a comparing unit 333, configured to compare the attribute field of the to-be-compared data that is the same as the index field of the second reference data but different from the source file information with the attribute field of the second reference data in sequence, to obtain a comparison result;
and an output unit 334, configured to output and store the data to be compared according to the comparison result.
Preferably, as shown in fig. 4, the comparing module 330 is specifically configured to perform data consistency comparison on the plurality of sub data files respectively through the plurality of comparing units 333.
Preferably, the alignment system further comprises:
the detecting module 340 is configured to, when any comparing unit 333 fails, send the data compared by the current comparing unit 333 to other comparing units 333 for comparison again.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system and method may be implemented in other ways. For example, the above-described system embodiments are merely illustrative, and for example, a division of modules is merely a logical division, and an actual implementation may have another division, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may also be an electric, mechanical or other form of connection.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (8)
1. A comparison method of mass data is characterized by comprising the following steps:
step 1, acquiring mass data files to be compared;
step 2, sequencing the mass data files according to a pre-stored quick sequencing algorithm to obtain a plurality of subdata files;
step 3, comparing the data consistency of the plurality of subdata files;
in step 3, the method specifically comprises the following steps:
step 3.1, extracting the index field and the attribute field of the data in the sub data file, and acquiring the source file information of the data;
step 3.2, selecting all data with different index fields from the subdata files as second reference data;
step 3.3, comparing the attribute field of the data to be compared, which is the same as the index field of the second reference data but different in source file information, with the attribute field of the second reference data in sequence to obtain a comparison result;
and 3.4, outputting and storing the data to be compared according to the comparison result.
2. The method according to claim 1, wherein step 2 comprises:
step 2.1, selecting N data with different values from the mass data file, and taking the values as first reference data;
2.2, sequentially arranging the first reference data from large to small or from small to large;
step 2.3, comparing and classifying the data in the mass data file with the first reference data in sequence to respectively obtain a data set smaller than the minimum first reference data, a data set larger than the maximum first reference data and a data set between any two adjacent first reference data, and putting the data with the same value as the first reference data in the mass data file into a previous data set adjacent to the first reference data or a next data set adjacent to the first reference data;
and 2.4, storing the classified data set into a file form to obtain N +1 sub-data files.
3. The comparison method according to claim 1 or 2, wherein in step 3, a plurality of comparison units are used to perform data consistency comparison on the plurality of sub-data files respectively.
4. The alignment method of claim 3, further comprising:
and 4, when any comparison unit fails, sending the data compared by the current comparison unit to other comparison units for comparison again.
5. A mass data comparison system, comprising:
the acquisition module is used for acquiring mass data files to be compared;
the sorting module is used for sorting the mass data files according to a pre-stored quick sorting algorithm to obtain a plurality of subdata files;
the comparison module is used for carrying out data consistency comparison on the plurality of subdata files;
the alignment module comprises:
the extracting unit is used for extracting the index field and the attribute field of the data in the sub data file and acquiring the source file information of the data;
the second selection unit is used for selecting all data with different index fields from the sub data files as second reference data;
the comparison unit is used for sequentially comparing the attribute field of the data to be compared, which is the same as the index field of the second reference data but has different source file information, with the attribute field of the second reference data to obtain a comparison result;
and the output unit is used for outputting and storing the data to be compared according to the comparison result.
6. The alignment system of claim 5, wherein the ranking module comprises:
the first selection unit is used for selecting N data with different values from the mass data files and taking the values as first reference data;
the sorting unit is used for sequentially sorting the first reference data from large to small or from small to large;
the classification unit is used for comparing and classifying the data in the mass data file with the first reference data in sequence to respectively obtain a data set smaller than the minimum first reference data, a data set larger than the maximum first reference data and a data set between any two adjacent first reference data, and putting the data with the same value as the first reference data in the mass data file into a previous data set adjacent to the first reference data or a next data set adjacent to the first reference data;
and the storage unit is used for storing the classified data set into a file form to obtain N +1 sub-data files.
7. The comparison system according to claim 5 or 6, wherein the comparison module is specifically configured to perform data consistency comparison on the plurality of sub data files respectively through the plurality of comparison units.
8. The alignment system of claim 7, further comprising:
and the detection module is used for sending the data compared by the current comparison unit to other comparison units for comparison again when any comparison unit has a fault.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710390393.5A CN107301203B (en) | 2017-05-27 | 2017-05-27 | Mass data comparison method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710390393.5A CN107301203B (en) | 2017-05-27 | 2017-05-27 | Mass data comparison method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107301203A CN107301203A (en) | 2017-10-27 |
| CN107301203B true CN107301203B (en) | 2020-10-02 |
Family
ID=60137202
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710390393.5A Active CN107301203B (en) | 2017-05-27 | 2017-05-27 | Mass data comparison method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107301203B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110515964A (en) * | 2019-08-30 | 2019-11-29 | 百度在线网络技术（北京）有限公司 | A kind of file updating method, device, electronic equipment and medium |
| CN110908972B (en) * | 2019-11-19 | 2022-09-02 | 加和（北京）信息科技有限公司 | Log data preprocessing method and device, electronic equipment and storage medium |
| CN112395276B (en) * | 2020-11-13 | 2024-05-28 | 中国人寿保险股份有限公司 | Data comparison method and related equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101236554A (en) * | 2007-11-29 | 2008-08-06 | 中兴通讯股份有限公司 | Database mass data comparison process |
| CN102184190A (en) * | 2011-04-19 | 2011-09-14 | 北京神州数码思特奇信息技术股份有限公司 | Data comparison method |
| CN103309847A (en) * | 2012-03-06 | 2013-09-18 | 百度在线网络技术（北京）有限公司 | Method and equipment for realizing file comparison |
| CN104317942A (en) * | 2014-10-31 | 2015-01-28 | 北京思特奇信息技术股份有限公司 | Massive data comparison method and system based on hadoop cloud platform |
-
2017
- 2017-05-27 CN CN201710390393.5A patent/CN107301203B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101236554A (en) * | 2007-11-29 | 2008-08-06 | 中兴通讯股份有限公司 | Database mass data comparison process |
| CN102184190A (en) * | 2011-04-19 | 2011-09-14 | 北京神州数码思特奇信息技术股份有限公司 | Data comparison method |
| CN103309847A (en) * | 2012-03-06 | 2013-09-18 | 百度在线网络技术（北京）有限公司 | Method and equipment for realizing file comparison |
| CN104317942A (en) * | 2014-10-31 | 2015-01-28 | 北京思特奇信息技术股份有限公司 | Massive data comparison method and system based on hadoop cloud platform |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107301203A (en) | 2017-10-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102882838B (en) | Authentication method and system applying verification code mechanism | |
| CN107301203B (en) | Mass data comparison method and system | |
| CN107515931B (en) | Repeated data detection method based on clustering | |
| CN104731896B (en) | A kind of data processing method and system | |
| CN106973332A (en) | A kind of barrage message treatment method, analytic method and system | |
| CN105718218B (en) | Compressed storage method and system applicable for load records of electric energy meter or concentrator | |
| CN106294222A (en) | A kind of method and device determining PCIE device and slot corresponding relation | |
| CN102750379B (en) | Fast character string matching method based on filtering type | |
| EP2804115B1 (en) | Index scan device and index scan method | |
| CN103345496A (en) | Multimedia information searching method and system | |
| CN104298736A (en) | Method and device for aggregating and connecting data as well as database system | |
| US20110238677A1 (en) | Dynamic Sort-Based Parallelism | |
| CN103207889A (en) | Method for retrieving massive face images based on Hadoop | |
| CN111625342B (en) | Data tracing method, device and server | |
| CN110222074A (en) | It indexes lookup method, search device, electronic equipment and storage medium | |
| CN103631769A (en) | Method and device for judging consistency between file content and title | |
| CN105488176A (en) | Data processing method and device | |
| CN101515286A (en) | Image matching method based on image feature multi-level filtration | |
| CN102314464A (en) | Lyrics searching method and lyrics searching engine | |
| CN102169499A (en) | Repeated ticket removing method | |
| CN102479211A (en) | Mass data processing system and method on basis of database | |
| CN110222015A (en) | A kind of reading of file data, querying method, device and readable storage medium storing program for executing | |
| CN103455491A (en) | Method and device for classifying search terms | |
| CN104699701A (en) | Parallel nearest node computing method and distributed system based on sensitive hashing | |
| CN106599326B (en) | Recorded data duplication eliminating processing method and system under cloud architecture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |