CN104376055B - A kind of large-sized model data comparing method based on allocation methods - Google Patents

Abstract

The invention discloses a method for comparing large model data based on fragmentation technology, which comprises the following steps: setting fragmentation parameters; taking out all keywords of the reference data source, sorting them from small to large, and storing them in a keyword array Medium; calculate the number of fragments fragment_num and the number of records in each fragment, and then obtain the first and last keyword values of each fragment from the keyword array in sequence; start a worker thread for each fragment, each The worker threads obtain the corresponding data content from the benchmark data source and the data source to be compared respectively; each worker thread compares the data content assigned to itself line by line, and records the difference results; after all the worker threads are processed, fragment_num differences are obtained As a result, all diff results are merged into a final diff result. The application of the present invention in two systems or two databases can greatly improve the comparison efficiency of large model data.

Description

A Large Model Data Comparison Method Based on Fragmentation Technology

technical field

The invention relates to a large model data comparison method based on fragmentation technology, and belongs to the technical field of power system automation distribution network model management.

Background technique

The data volume of the distribution network model is relatively large, and the number of records in a model table is likely to reach the million level. For tables of this order of magnitude, the traditional single-workflow comparison method may have the problem that the comparison process takes a long time.

Contents of the invention

In view of the deficiencies in the prior art, the purpose of the present invention is to provide a method for comparing large model data based on fragmentation technology that can greatly improve the efficiency of large model data comparison when applied in two systems or two databases.

In order to achieve the above object, the present invention is achieved through the following technical solutions:

A method for comparing large model data based on fragmentation technology of the present invention specifically includes the following steps:

(1) Fragmentation parameters are set, and the fragmentation parameters support two kinds of setting modes: setting by the number of records and setting by the data block size; The length of each record in the comparison data source is k, and the number of records contained in each fragment is set to be n at most, then n=m/k can be obtained; if the fragmentation parameter is set to the number of records, then n is the The maximum number of records contained in a shard;

(2) Take out all the keywords of the benchmark data source, arrange them in ascending order, and store them in the keyword array, the size of the keyword array is the total number of records record_sum in the benchmark data source;

(3) Calculate the number of fragments fragment_num and the number of records in each fragment, and then obtain the first and last keyword values of each fragment from the keyword array in order, that is, obtain the fragment information;

fragment_num=record_sum/n+(record_sum%n!=0)

If the total number of records record_sum is an integer multiple of n, the number of records obtained in each shard is n;

If the total number of records record_sum is not an integer multiple of n, in the first fragment_num-1 fragments, the number of records in each fragment is n, and the remaining number of records is allocated in the last fragment;

(4) Start a worker thread for each slice, and according to the corresponding slice information, each worker thread obtains corresponding data content from the benchmark data source and the data source to be compared;

(5) Each working thread compares the data content assigned to itself line by line by field, and records the difference results;

(6) After all the worker threads are processed, fragment_num difference results are obtained, and all the difference results are merged into one result according to the keywords from small to large, which is the final difference result.

The above-mentioned difference results include a difference identification and a description of the difference content; the difference identification includes three types of identification: insert, update, and delete; if a certain record does not exist in the data source to be compared but exists in the reference data source, then the difference identification is inserted; if If a record exists in the data source to be compared but not in the reference data source, the difference is marked as deleted; if a record has the same keyword in the data source to be compared and the reference data source, but the content is inconsistent, the difference is marked as Update; the difference content is described as the corresponding data records in the benchmark data source and the data source to be compared.

In the present invention, the flexibility of fragmentation is ensured by providing two fragmentation parameter setting modes of setting according to the number of records and setting according to the size of data blocks; dividing each fragment according to keywords ensures that the fragmentation is disjoint Integrity and integrity, thus ensuring the non-redundancy of the comparison process and the integrity of the difference results; multiple worker threads simultaneously read and compare the data content according to their respective fragmentation information, and the comparison work is performed concurrently to improve the efficiency Overall comparison efficiency; use the difference identification and difference content description to record the difference between the data source record to be compared and the reference data source record, so that the SQL statement to be synchronized can be easily organized according to the difference result.

Description of drawings

Fig. 1 is a working flow chart of the method for comparing large model data based on sharding technology in the present invention.

detailed description

In order to make the technical means, creative features, goals and effects achieved by the present invention easy to understand, the present invention will be further described below in conjunction with specific embodiments.

A large model data comparison method based on fragmentation technology of the present invention. The model tables in the distribution network system generally have keywords, which provides the possibility of fragment comparison by keywords. The present invention mainly aims at the model data table with a large number of records, sets the fragmentation parameters according to the needs, and then obtains the fragmentation content from the reference data source and the data source to be compared according to the fragmentation parameters, and compares multiple fragments at the same time, and finally Get diff results. The difference result is composed of a difference identifier and a difference content description. The difference identifier has three identifiers: insert, update, and delete. The difference content description is the content information of the corresponding records of the reference data source and the data source to be compared. Difference results are generated for the data source being compared against the benchmark data source.

Referring to Figure 1, this method specifically includes the following steps:

(1) Specify the data source for comparison and the model table to be compared. The types supported by the data source include databases and data files, and keywords must be included in the model table. Set fragmentation parameters as required, either by the number of records or by the size of the data block.

If the fragmentation parameter is set to the size of the data block, assuming that the data block size is set to m, the length of each record in the table is k, and the number of records contained in each corresponding fragment is at most n, then n= m/k; if the fragmentation parameter is set to the number of records, then n is the value set for this.

(2) Obtain all the keywords of the benchmark data source, arrange them in descending order, and store them in the keyword array. The size of the array is the total number of records record_sum in the benchmark data source.

Combining with the keyword array, obtain fragmentation information according to fragmentation parameters, including the number of fragments and the first and last keyword values of each fragment.

(3) Calculate the number of shards and the number of records in each shard, and then obtain the first and last keyword values of each shard from the keyword array in sequence;

Fragment number fragment_num value should be:

fragment_num=record_sum/n+(record_sum%n!=0)

If the total number of records record_sum is an integer multiple of n, then the number of records obtained in each shard is n;

The total number of records record_sum is not an integer multiple of n, then in the first fragment_num-1 fragments, the number of records in each fragment is n, and the remaining records are allocated in the last fragment.

(4) Start a worker thread for each slice, and obtain the corresponding content of the benchmark data source and the data source to be compared according to the corresponding slice information;

(5) Each worker thread compares the data content allocated to itself line by line, and records the difference results. The difference result includes difference identification and difference content description. The difference identification includes insert, update, and delete three types of identification. The difference content description is the content information of the corresponding records in the reference data source and the data source to be compared.

(6) After the comparison of all working threads is completed, fragment_num difference results are obtained, and all the difference results are merged into one result, which is the final difference result.

Working principle of the present invention is:

The present invention is mainly aimed at the comparison of large model data tables with large isomorphic records in different systems or different databases and obtaining difference results. Set the sharding information according to the keywords of the benchmark data source, and then obtain the shard content from the benchmark data source and the data source to be compared, and compare multiple shards at the same time, and finally obtain the difference result. The method of the invention realizes the slice comparison technology, and greatly improves the comparison efficiency of large model data with a large number of records.

In the present invention, the setting according to the number of records and the setting according to the data block size guarantees the flexibility of fragmentation to a certain extent. The division of each slice according to the keyword ensures the disjointness and integrity of the slices, thereby ensuring the non-redundancy of the comparison process and the integrity of the difference results. Multiple worker threads simultaneously read data content and compare according to their respective shard information, and the comparison work is performed concurrently to improve the overall comparison efficiency. Use the difference identification and difference content description to record the difference between the data source record to be compared and the reference data source record, so that the SQL statement to be synchronized can be easily organized according to the difference result.

By adopting the method of the invention, large model tables are compared using fragmentation technology, which can greatly improve the comparison efficiency. Without considering machine performance and resource usage, the number of shards is close to the speed after sharding compared to before sharding.

The basic principles and main features of the present invention and the advantages of the present invention have been shown and described above. Those skilled in the industry should understand that the present invention is not limited by the above-mentioned embodiments. What are described in the above-mentioned embodiments and the description only illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will also have Variations and improvements all fall within the scope of the claimed invention. The protection scope of the present invention is defined by the appended claims and their equivalents.

Claims (2)

1. A large model data comparison method based on fragmentation technology, characterized in that, specifically comprising the following steps: (1) Fragmentation parameters are set, and the fragmentation parameters support two setting methods: setting by the number of records and setting by the size of the data block; If the fragmentation parameter is set to be based on data block size, set the data block size to m, set the length of each record in the data source to be compared to k, and set the maximum number of records contained in each slice to n, then n can be obtained =m/k; If the sharding parameter is set to the number of records, then n is the maximum number of records contained in each shard; The value of n in the two setting methods is obtained by the tail removal method to obtain an integer value; (2) Take out all the keywords of the benchmark data source, arrange them in ascending order, and store them in the keyword array, the size of the keyword array is the total number of records record_sum in the benchmark data source; (3) Calculate the number of fragments fragment_num and the number of records in each fragment, and then obtain the first and last keyword values of each fragment from the keyword array in order, that is, obtain the fragment information; If the total number of records record_sum is an integer multiple of n, then the number of records in each fragment is n, and the calculation method for the number of fragments is: fragment_num=record_sum/n; If the total number of records record_sum is not an integer multiple of n, then in the first fragment_num-1 fragments, the number of records in each fragment is n, and the remaining records are allocated in the last fragment. The calculation method is: fragment_num=record_sum/n+1, where the value of record_sum/n obtains the corresponding integer value according to the tail removal method; (4) Start a worker thread for each slice, and according to the corresponding slice information, each worker thread obtains corresponding data content from the benchmark data source and the data source to be compared; (5) Each working thread compares the data content assigned to itself line by line by field, and records the difference results; (6) After all the worker threads are processed, fragment_num difference results are obtained, and all the difference results are merged into one result according to the keywords from small to large, which is the final difference result. 2. the large model data comparison method based on fragmentation technology according to claim 1, is characterized in that, The difference result includes difference identification and difference content description; The difference mark includes insert, update, and delete three marks; if there is no record in the data source to be compared but there is in the reference data source, then the difference mark is inserted; if a record exists in the data source to be compared, the reference If there is no record in the data source, the difference is marked as deleted; if a record has the same keyword in the data source to be compared and the reference data source, but the content is inconsistent, the difference is marked as update; The difference content is described as the corresponding data records in the benchmark data source and the data source to be compared.