CN109690521B - Database merging method and device - Google Patents
Database merging method and device Download PDFInfo
- Publication number
- CN109690521B CN109690521B CN201780037501.9A CN201780037501A CN109690521B CN 109690521 B CN109690521 B CN 109690521B CN 201780037501 A CN201780037501 A CN 201780037501A CN 109690521 B CN109690521 B CN 109690521B
- Authority
- CN
- China
- Prior art keywords
- data table
- target data
- records
- fields
- structured
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application discloses a method and a device for merging databases, wherein the method comprises the following steps: the method comprises the steps that a target data table and a source data table are respectively obtained from a target database and a source database, wherein the target data table comprises a plurality of fields and a plurality of records, the source data table also comprises a plurality of fields and a plurality of records, and each record comprises a field value corresponding to each field; performing co-structuring processing on the target data table according to field differences of the target data table and the source data table to obtain a co-structured target data table; the method can realize rapid combination of databases and improve the combination mode of the prior art.
Description
Technical Field
The present disclosure relates to the field of databases, and in particular, to a method and apparatus for merging databases.
Background
Databases have become the preferred way of data storage due to their superior performance of high load, high response, multiple concurrency, support for atomic operations, support for multi-version control, etc. Many software and websites today, up to cloud computing centers, down to instant chat software, use databases to store data. In using databases, it is often necessary to update a target database using a source database. At this point, the databases need to be consolidated. In addition, when there are other data to be integrated, the databases need to be merged.
Common consolidated databases are typically done manually by database maintenance personnel. The maintainer respectively exports the two databases to be combined, opens the databases in a certain view, and combines the databases by means of other comparison tools, so that the degree of automation is lower. Some commercial databases may provide associated merge tools, but are only applicable to their associated database products. Moreover, since some relational databases are used as free software, developers do not provide related tools, and therefore the merging tools cannot be applied to the occasions of some relational databases. There are also some database service vendors that typically prefer to offer a complete set of database services, requiring a high service fee. Accordingly, such products are also often bulky, slow to react, and difficult to maintain.
Disclosure of Invention
The application provides a database merging method and device, which can be used for quickly and safely merging databases through a light-weight method.
The application adopts a technical scheme that: there is provided a method of database merging comprising the steps of: the method comprises the steps that a target data table and a source data table are respectively obtained from a target database and a source database, wherein the target data table comprises a plurality of fields and a plurality of records, the source data table also comprises a plurality of fields and a plurality of records, and each record comprises a field value corresponding to each field; performing co-structuring processing on the target data table according to field differences of the target data table and the source data table to obtain a co-structured target data table; and carrying out data merging on the basis of the source data table and the co-structured target data table.
In order to solve the technical problems, another technical scheme adopted by the application is as follows: an apparatus for database merging, comprising: the processor is connected with the memory, and the processor executes the following steps by running a program stored in the memory: the method comprises the steps that a target data table and a source data table are respectively obtained from a target database and a source database, wherein the target data table comprises a plurality of fields and a plurality of records, the source data table also comprises a plurality of fields and a plurality of records, and each record comprises a field value corresponding to each field; performing co-structuring processing on the target data table according to field differences of the target data table and the source data table to obtain a co-structured target data table; and carrying out data merging on the basis of the source data table and the co-structured target data table.
The beneficial effects of this application are: the method and the device for merging the databases are provided, and the databases are quickly and safely merged by a lightweight method by merging the source data table and the target data table which are respectively obtained from the source database and the target database in a structuring way.
Drawings
FIG. 1 is a flow chart of an embodiment of a method for database merging of the present application
FIG. 2 is a schematic view of a sub-flow of an embodiment of step S10 of FIG. 1, which is a method for database merging according to the present application;
FIG. 3 is a schematic view showing a sub-flow of an embodiment of step S11 of FIG. 1, which is a method for database merging according to the present application;
FIG. 4 is a schematic illustration of the co-structure of FIG. 3;
FIG. 5 is a schematic view showing a sub-flow of an embodiment of step S12 of FIG. 1, which is a method for database merging according to the present application;
FIG. 6 is a schematic diagram of two structured target data tables to be merged to a source data table of the database merging method of the present application;
FIG. 7 is a schematic view of a further sub-process of the embodiment of step S12 of FIG. 1;
FIG. 8 is a schematic diagram of the co-structured target data table of FIG. 6 after merging according to the method of FIG. 7;
FIG. 9 is a schematic view of another sub-flow chart of an embodiment of step S12 of FIG. 1, which is a method for database merging according to the present application;
FIG. 10 is a schematic diagram of the co-structured target data table of FIG. 6 after merging according to the method of FIG. 9;
FIG. 11 is a schematic view of another sub-flow of an embodiment of step S12 of FIG. 1, which is a method for database merging according to the present application;
FIG. 12 is a schematic diagram of the co-structured target data table of FIG. 6 after merging according to the method of FIG. 11;
FIG. 13 is a schematic view of another sub-process of the embodiment of step S12 of FIG. 1, which is a method for database merging according to the present application;
FIG. 14 is a schematic diagram illustrating an embodiment of a database merging apparatus according to the present application.
Description of the embodiments
The following description of the technical solutions in the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
Referring to fig. 1, a flowchart of an embodiment of the method for merging databases in fig. 1 is shown, and a specific gap detection method includes the following steps:
s10, respectively acquiring a target data table and a source data table from a target database and a source database.
In this embodiment, since the specific operation object is each data table in the database, the target data table and the source data table are acquired from the target database and the source database, respectively. Specifically, the table names of all tables stored in the target database are traversed, and simultaneously the table names of all tables stored in the source database are traversed, so that two tables corresponding to the table names are selected from the two databases respectively and used as the target data table and the source data table. Each of the target data table and the source data table includes a plurality of fields and a plurality of records, and includes a plurality of field values. Preferably, a field refers to a column of a table, a record refers to a row of a table, and a field value refers to a specific value corresponding to the field and the record.
In particular, in this embodiment, both the destination data table and the source data table have a unique identification field. All field values under this identification field are unique to a table, such as number, student status number, etc.
S11, performing co-structuring processing on the target data table according to field differences of the target data table and the source data table to obtain a co-structured target data table.
Before merging, the target data table and the source data table need to be subjected to the same structure operation, that is, the target data table and the source data table have the same field. In this embodiment, the target data table is operated, so that the target database can be backed up and the source database can be read-only declared. In order to make the target data table and the source data table have the same field, the target data table and the source data table need to be traversed to obtain redundant or missing fields of the target data table relative to the source data table, and then a temporary table is created. Wherein the temporary table adds missing fields or deletes redundant fields relative to the target data table, or adds both missing and redundant fields. In addition, the co-structured target data table may be further filled with records as needed, so that the same field value as the co-structured target data table field in the records of the target data table remains in the co-structured target data table. The field values of the same field as the temporary table in the record of the target data table can be filled into the temporary table, and the field values of the corresponding fields of the record of the temporary table, which are different from the target data table, are filled by the field values of the corresponding fields of the corresponding record in the source data table. And finally, setting the temporary table as a co-structured target data table, so as to obtain the co-structured target data table after the co-structuring treatment.
S12, data merging is carried out on the basis of the source data table and the same structured target data table.
And after the co-structured target data table and the source data table are obtained, merging the obtained source data table and the co-structured target data table. However, in actual operation, the user may need to protect part of the data, so in the actual merging process, the protection data including the protection record and the field may be determined first, or may be selected to be unprotected, which is not limited herein, and then different methods are used according to different choices of the user.
Referring to fig. 2, fig. 2 is a schematic flow chart illustrating a sub-process of an embodiment of step S10 of the method for merging databases in the present application, and this embodiment is a further extension of step S10 of the first embodiment of the present application, which specifically includes the following sub-steps in this embodiment.
S101, traversing table names of tables stored in target data and original data respectively.
In the actual operation of the database, the tables stored in the database are actually operated, so that the source data table and the target data table which need to be combined need to be found out. In a particular operation, for a table in the database that needs to be merged, there is also such a corresponding table in the source data. Specifically, table names have a corresponding relationship, and the table names can be the same or similar, so that the table names of each table stored in the target data are traversed respectively, and the source database is traversed by the same method.
S102, taking tables with table names corresponding to and respectively stored in the target database and the source database as a target data table and a source data table.
For one table in the target data, if one table with the table name corresponding to the table name exists in the source data, the table in the target database is taken as the target data table, and the table with the table name corresponding to the table name in the source data is taken as the source data table. The correspondence here may be the same or similar or other correspondence.
Referring to fig. 3, fig. 3 is a schematic flow chart illustrating a sub-process of step S11 of the method for merging databases in the present application, and this embodiment is a further extension of step S11 in the first embodiment of the present application, and this embodiment specifically includes the following sub-steps.
S111, traversing the fields in the target data table and the source data table respectively to determine the missing fields and redundant fields of the target data table relative to the source data table.
Because of the possible fields in the source data table and the destination data table, data errors can occur during merging. Therefore, the target data table is first structured identically to the source data table, i.e., so that the target data table has the same fields as the source data table. Therefore, the fields in the target data table and the source data table are traversed, and some fields and redundant fields which are missing in the target data table relative to the source data table are obtained and recorded.
S112, a temporary table is established and field setting is carried out, so that the temporary table is added with missing fields relative to the target data table and redundant fields are deleted.
In a specific database operation, in order to protect data and prevent data from being deleted, lost, etc. during the operation, a method of backing up and creating a temporary table is generally adopted. Therefore, a temporary table is firstly established, and then a field is set for the temporary table, so that the temporary table is added with the missing field recorded in the previous step relative to the target data table, and the redundant field recorded in the previous step is deleted. That is, this temporary table has the same field as the source data table.
S113, filling the field value of the same field as the temporary table in the record of the target data table into the temporary table.
Since only the temporary table is provided with fields, it is also necessary to fill the temporary table with records, including filling the same fields of the temporary table as the target data table. And for the same field in the temporary table as the field in the target database, sequentially filling the field value corresponding to the field in the target data table into the field corresponding to the temporary table. In particular, since the target data table and the source data table both have an identification field, the temporary table also has an identification field.
S114, filling the field value corresponding to the field which is different from the target data table in the record of the temporary table by the field value of the corresponding field of the corresponding record in the source data table.
Because the temporary table may be more than a certain field with respect to the target data table, and because the fields are from the source data table, there are no field values corresponding to the fields in the target data table, so the field values of the fields need to be obtained from the source database. In practice, for example, a field refers to the age, and a record refers to the information of the nail, and the value of this field of the age of the nail is determined by the field and the record together. It will thus be appreciated that for non-existent fields, such as a wage without a first in the target data table, but one more wage field in the temporary table, the wage for a first would need to be obtained from the source database if it were to be filled. Here, a field value of a unique identification field is needed to determine, and if the field value of the unique identification field in the source data table has a first, the payroll corresponding to the source data table is filled into the payroll corresponding to the temporary table, and if the field value of the unique identification field in the source data table does not have a first, the filling is not performed, assuming that the unique identification field in the table is a name.
S115, setting the temporary table as the same-structured target data table.
The temporary table has the same field as the source data table, namely the same structure, and is filled with data, namely the same structuring is completed, and the temporary table is set as the same structuring target data table.
Referring specifically to fig. 4, fig. 4 is a schematic diagram of the same structure as that of fig. 3. Including a source data table, a target data table, and a temporary table. In this embodiment, the name is used as the identification field.
It can be seen that in the temporary table, the job number and age in the field exist in the source data table and the target data table at the same time, so the field value is acquired from the target data table, but in the payroll field, the first is the identification field value shared by the target data table and the source data table, so the payroll of the first can be filled by the payroll of the source data table corresponding to the first, but in the case of the fourth, the filling cannot be performed because the identification field in the source data table, namely the last name, has no field value below, and is empty.
Referring to fig. 5, fig. 5 is a schematic flow chart illustrating a sub-process of step S12 of the method for merging databases in the present application, and this embodiment is a further extension of step S12 in the first embodiment of the present application, and this embodiment specifically includes the following sub-steps.
S121, a protected record or a protected field is specified in the co-structured target data table.
In actual operation, the user may need to protect some data, so that the user needs to determine the data to be protected, and skip the data to be protected in the subsequent merging process, where the protected data may be fields or records, and determine the data to be protected, where the protected data specifically specifies a target data table with the same structure.
S122, merging the data of the same structured target data table and the source data table on the basis of skipping the protected record or the protected field.
After the protected fields or hand-protected records are determined, the data is skipped during subsequent merging, i.e., the records or fields do not participate in the merging, nor are modified, and then the co-structured data table and the source data table are merged.
Referring to fig. 6, fig. 6 is a schematic diagram of two structured target data tables to be merged to source data tables according to the database merging method of the present application.
Wherein the identification field of the same structured data table and the source data table is a number. In other embodiments, the identification field herein is not limited and may be a job number, a name, etc. Wherein 1, 2, 3 are the field values of the identification fields of the same structured data table, the row in which 1 is located is the record corresponding to the identification fields, and the third is the field value of the name field in the same structured data table, namely the field value of the record corresponding to 1.
Referring to fig. 7, fig. 7 is a schematic diagram of another sub-flow of an embodiment of step S12 of fig. 1, which is a method for database merging according to the present application. And this embodiment is a further extension of step S12 of the first embodiment of the present application, too
One of the three combining modes, the implementation mode specifically comprises the following substeps.
S121a, comparing field values corresponding to identification fields playing an identification role in records of the structured target data table and the source data table.
In this embodiment, the identification fields are required to be used for comparison and merging, that is, to compare field values of the identification fields of the structured target data table and the source data table. If the identification fields of the source data table and the structured target data table are numbers, each number value of the structured target data table is compared with each number value of the source data table.
S122a, determining missing records and/or redundant records of the same structured target data table relative to the source data table.
After the comparison, records corresponding to field values of different identification fields of the same structured data table relative to the source data table are determined. That is, like the structured data table and the identification field of the source data table are numbers, if the number values of the numbers in the structured data table are 1, 2, and 3, and the number values of the numbers in the source data table are 3, 4, and 5, the structured data table lacks 4 and 5, that is, lacks records corresponding to 4 and 5, and is superfluous by 1 and 2, that is, superfluous records corresponding to 1 and 2, so as to determine and record the structured data table.
And S123a, adding missing records and/or deleting redundant records in the structuring target data table.
The determined missing records are added to the co-structured target data table. That is, as described above, the records corresponding to the records missing from the structured target data table with respect to the source data, namely, 4 and 5, are added, and the redundant records are deleted, that is, the records corresponding to the records missing from the structured target data table with respect to the source data, namely, 1 and 2, are deleted.
Referring further to fig. 8, fig. 8 is a schematic diagram of the co-structured target data table of fig. 6 after merging according to the method of fig. 7.
The structured target data table is redundant of records corresponding to 1 and 2 relative to the source data table, but lacks records corresponding to 4 and 5. Therefore, in the merged data table, the records corresponding to 1 and 2 are deleted, the records corresponding to 4 and 5 are added, and the record corresponding to 3 is not modified, so as to obtain fig. 8.
Referring to fig. 9, fig. 9 is a schematic view of a further sub-flow of an embodiment of the method of database merging of the present application, step S12 of fig. 1, and this embodiment is a further extension of the first embodiment of the present application, step S12, and is also
One of the three combining modes. This embodiment specifically includes the following substeps.
S121b, comparing field values in records of the structured target data table and the source data table.
In this embodiment, the identification fields are required to be used for comparison and merging, that is, to compare field values of the identification fields of the structured target data table and the source data table. If the identification fields of the source data table and the structured target data table are numbers, each number value of the structured target data table is compared with each number value of the source data table.
S122b, determining records with the same field value corresponding to the identification field playing the role of identification and at least partially different field values corresponding to the rest fields.
That is, like the structured data table and the identification field of the source data table are numbered, it is assumed that the numbered values of the same structured data table have numbers 1, 2, and 3, and the numbered values of the numbers in the source data table have numbers 3, 4, and 5, and then have a common field value of 3. In determining 3 records corresponding to the structured target data table and 3 records corresponding to the source data table, field values different from the two records are determined.
And S123b, updating corresponding records in the structured target data table by using the records determined in the source data table.
That is, if a field value in the record corresponding to 3 of the source data table is different from a field value in the record corresponding to 3 of the structured target data table, the field value in the structured target data table is replaced by the field value of the source data table, and the others remain unchanged.
Referring further to fig. 10, fig. 10 is a schematic diagram of the co-structured target data table of fig. 6 after merging according to the method of fig. 9.
The structured target data table is 1, 2 and 3 in the identification fields of the structured target data table and 3, 4 and 5 in the identification fields of the source data table, so that the identification fields of the two data tables share the same 3. In the structured data table, the record corresponding to 3 is: 3. xie Li, 3K. In the source data table, the record corresponding to 3 is: 3. wangdi, 6K. It is obvious that, if the field value of the record corresponding to 3 in the structured data table is different from the field value of the record corresponding to 3 in the source data table, the record corresponding to 3 in the structured data table is replaced with the record corresponding to 3 in the source data table, so as to obtain fig. 10.
Referring to fig. 11, fig. 11 is a schematic view of a further sub-flow of an embodiment of the method for merging databases in fig. 1, step S12, and this embodiment is a further extension on the basis of step S12 in the first embodiment of the present application, and is one of three merging modes, and this embodiment specifically includes the following sub-steps.
S121c, replacing all records in the co-structured target data table with records in the source data table.
That is, the structured target data table is emptied and all records of the source data table are filled such that the records of the source data table overlay the structured target data table.
With further reference to fig. 12, fig. 12 is a schematic diagram of the co-structured target data table of fig. 6 after merging according to the method of fig. 11.
The source data table is fully overlaid with the structured data table, i.e., all records in the structured target data table are replaced with records in the source data table, resulting in FIG. 12.
Referring to fig. 13, fig. 13 is a schematic view of a further sub-flow of an embodiment of the method of database merging of the present application in step S12 of fig. 1, and this embodiment is a further extension of step S12 of the first embodiment of the present application. It should be noted that this embodiment is a schematic diagram of implementation of the implementation method of the embodiment of fig. 7, 9 and 11, and specifically includes the following steps:
s121d, circularly reading records from the source data table by using the first thread and storing the records in the first cache area.
In this embodiment, the implementation of merging is mainly described by a specific implementation method. This is achieved by using a reference to a plurality of threads, first a row of records is read from the source data table and stored as a list of strings using a first thread loop, and this list of strings is stored in the first buffer, and the above operation is looped until the first buffer is full.
S122d, circularly reading records from the first buffer area by using the second thread, and storing the records in the second buffer area.
A record is fetched from the first cache using a second thread loop and placed in the second cache. It should be noted that the second buffer is smaller than the first buffer. If the first buffer is empty or the second buffer is full, the second thread goes to sleep, i.e. is not active. After the writing succeeds, the first buffer is written by using the first thread because the first buffer is empty by one position.
S123d, circularly reading records from the structuring target data table by utilizing a third thread, and storing the records in a third cache area.
After operating on the source data table, it is necessary here to operate on the co-structured target data table. First, a row of records is read from the structured target data table using a third thread loop and stored as a list of strings, and the list of strings is stored in a third buffer. The above operation is looped until the first buffer is full.
And S124d, circularly reading records from the third buffer area by using the fourth thread, and storing the records in the fourth buffer area.
And taking out a record from the third buffer area by adopting a fourth linear cycle, and placing the record in the fourth buffer area. It should be noted that the fourth buffer is smaller than the third buffer, and if the third buffer is empty or the fourth buffer is full, the fourth thread goes to sleep, i.e. is not active. And after the writing is successful, the third buffer area is written by using a third thread because the third buffer area is empty by one position.
And S125d, comparing the record stored in the second buffer area with the record stored in the fourth buffer area by utilizing the fifth thread, and executing corresponding data merging operation according to the comparison result.
That is, when the second buffer area and the fourth buffer area are not empty, the fifth thread is adopted to compare the record stored in the second buffer area with the record stored in the fourth buffer area, and the operations of adding, deleting, replacing and the like are performed through the comparison result and the selected combination mode. After the comparison and operation are completed, the second buffer area and the fourth buffer area are cleared, and the second thread and the fourth thread are awakened to work.
Referring to fig. 14, fig. 14 is a schematic structural diagram of an embodiment of a database merging device according to the present application.
In this embodiment, the apparatus includes a processor 10 and a memory 11, where the processor 10 is connected to the memory 11.
The processor 10 is configured to determine protection data, set temporary tables, and fill temporary tables, traverse databases and data tables, compare and merge data tables using multi-threading and multi-cache.
The processor 10 may also be referred to as a CPU (Central Processing Unit ). The processor 10 may be an integrated circuit chip having signal processing capabilities. The processor 10 may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The respective module terminals of the above apparatus may execute the corresponding steps in the above method embodiments, so that the detailed description of the respective modules will be omitted herein.
The storage means 20 of the present application stores a program file capable of implementing all the methods described above, which may be stored in the form of a software product in the storage means, while also recording data of various calculations, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the methods of the embodiments of the present application. The aforementioned storage device includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, an optical disk, or other various media capable of storing program codes, or a terminal device such as a computer, a server, a mobile phone, a tablet, or the like.
In summary, as will be readily understood by those skilled in the art, the present application provides a method and an apparatus for merging databases, which take two data tables from two databases respectively in a traversal manner, then perform structuring on the two data tables, and after determining protection data, merge the two data tables in multiple manners by multithreading and a buffer area manner, so as to quickly complete merging.
The foregoing is only the embodiments of the present application, and not the patent scope of the present application is limited by the foregoing description, but all equivalent structures or equivalent processes using the contents of the present application and the accompanying drawings, or directly or indirectly applied to other related technical fields, which are included in the patent protection scope of the present application.
Claims (17)
1. A method of merging databases, comprising the steps of:
the method comprises the steps that a target data table and a source data table are respectively obtained from a target database and a source database, wherein the target data table comprises a plurality of fields and a plurality of records, the source data table also comprises a plurality of fields and a plurality of records, and each record comprises a field value corresponding to each field;
performing co-structuring processing on the target data table according to field differences of the target data table and the source data table to obtain a co-structured target data table;
data merging is carried out on the basis of the source data table and the co-structured target data table;
the step of performing the co-structuring processing on the target data table according to the field differences of the target data table and the source data table comprises the following steps:
performing field addition and/or field deletion on the target data table according to the field difference between the target data table and the source data table to obtain the co-structured target data table, wherein the fields in the co-structured target data table are identical to the fields in the source data table;
the step of merging data based on the source data table and the co-structured target data table comprises:
circularly reading records from the source data table by using a first thread and storing the records in a first cache area;
circularly reading records from the first cache area by using a second thread and storing the records in a second cache area, wherein the number of the records stored in the first cache area is larger than that of the records stored in the second cache area;
circularly reading records from the co-structured target data table by using a third thread, and storing the records in a third cache area;
circularly reading records from a third buffer area by using a fourth thread, and writing the records into a fourth buffer area, wherein the number of the records stored in the third buffer area is larger than that of the records stored in the fourth buffer area, and if the third buffer area is empty or the fourth buffer area is full, the fourth thread enters a sleep state;
when the fourth thread is used for successfully writing the fourth buffer, the third thread is used for writing the third buffer;
comparing the record stored in the second buffer area with the record stored in the fourth buffer area by using a fifth thread, and executing corresponding data merging operation according to a comparison result;
and when the second buffer area and the fourth buffer area are not empty, comparing the record stored in the second buffer area with the record stored in the fourth buffer area by adopting the fifth thread, and performing the operations of adding, deleting and replacing based on the comparison result and the selected combination mode.
2. The method of claim 1, wherein the step of obtaining the target data table and the source data table from the target database and the source database, respectively, comprises:
traversing table names of tables stored in the target database and the source database respectively, and taking the tables corresponding to the table names and stored in the target database and the source database respectively as the target data table and the source data table.
3. The method of claim 1, wherein the step of co-structuring the target data table based on field differences between the target data table and the source data table further comprises:
and filling the record of the co-structured target data table so as to keep the field values which are the same as the fields of the co-structured target data table in the record of the target data table in the co-structured target data table, wherein the field values corresponding to the other fields in the record of the co-structured target data table are filled by the field values of the corresponding fields of the corresponding record in the source data table.
4. The method according to claim 1, wherein the step of adding and/or deleting fields to the target data table according to the field differences between the target data table and the source data table comprises:
traversing the fields in the target data table and the source data table respectively to determine the missing fields and redundant fields of the target data table relative to the source data table;
establishing a temporary table and setting fields so that the temporary table adds the missing fields and deletes the redundant fields relative to the target data table;
the step of performing record filling on the co-structured target data table comprises the following steps:
filling field values of fields which are the same as the temporary table in the records of the target data table into the temporary table, and filling field values of fields which are different from the target data table in the records of the temporary table by field values of corresponding fields of corresponding records in the source data table;
setting the temporary table as the co-structured target data table.
5. The method of claim 1, wherein the step of merging data based on the co-structured target data table and the source data table comprises:
comparing field values corresponding to identification fields playing an identification role in records of the same structured target data table with those of the source data table to determine missing records and/or redundant records of the same structured target data table relative to the source data table;
adding the missing record and/or deleting the redundant record in the co-structured target data table.
6. The method of claim 1, wherein the step of merging data based on the co-structured target data table and the source data table comprises:
comparing the field values in the records of the structuring target data table with the field values in the records of the source data table to determine records with the same field value corresponding to the marking field playing a role in marking and at least partially different field values corresponding to the rest fields;
and updating the corresponding record in the co-structured target data table by using the record determined in the source data table.
7. The method of claim 1, wherein the step of merging data based on the co-structured target data table and the source data table comprises:
and replacing all records in the co-structured target data table with the records in the source data table.
8. The method of claim 1, wherein the step of merging data based on the co-structured target data table and the source data table comprises:
specifying a protected record or protected field in the co-structured target data table;
and merging the data of the co-structured target data table and the source data table on the basis of skipping the protected record or the protected field.
9. An apparatus for merging databases, the apparatus comprising: the processor is connected with the memory, and the processor executes the following steps by running a program stored in the memory:
the method comprises the steps that a target data table and a source data table are respectively obtained from a target database and a source database, wherein the target data table comprises a plurality of fields and a plurality of records, the source data table also comprises a plurality of fields and a plurality of records, and each record comprises a field value corresponding to each field;
performing co-structuring processing on the target data table according to field differences of the target data table and the source data table to obtain a co-structured target data table;
data merging is carried out on the basis of the source data table and the co-structured target data table;
the step of performing the co-structuring processing on the target data table according to the field differences of the target data table and the source data table comprises the following steps:
performing field addition and/or field deletion on the target data table according to the field difference between the target data table and the source data table to obtain the co-structured target data table, wherein the fields in the co-structured target data table are identical to the fields in the source data table;
the step of merging data based on the source data table and the co-structured target data table comprises:
circularly reading records from the source data table by using a first thread and storing the records in a first cache area;
circularly reading records from the first cache area by using a second thread and storing the records in a second cache area, wherein the number of the records stored in the first cache area is larger than that of the records stored in the second cache area;
circularly reading records from the co-structured target data table by using a third thread, and storing the records in a third cache area;
circularly reading records from a third buffer area by using a fourth thread, and writing the records into a fourth buffer area, wherein the number of the records stored in the third buffer area is larger than that of the records stored in the fourth buffer area, and if the third buffer area is empty or the fourth buffer area is full, the fourth thread enters a sleep state;
when the fourth thread is used for successfully writing the fourth buffer, the third thread is used for writing the third buffer;
comparing the record stored in the second buffer area with the record stored in the fourth buffer area by using a fifth thread, and executing corresponding data merging operation according to a comparison result;
and when the second buffer area and the fourth buffer area are not empty, comparing the record stored in the second buffer area with the record stored in the fourth buffer area by adopting the fifth thread, and performing the operations of adding, deleting and replacing based on the comparison result and the selected combination mode.
10. The apparatus of claim 9, wherein the step of obtaining the target data table and the source data table from the target database and the source database, respectively, comprises:
traversing table names of tables stored in the target database and the source database respectively, and taking the tables corresponding to the table names and stored in the target database and the source database respectively as the target data table and the source data table.
11. The apparatus of claim 9, wherein the step of co-structuring the target data table based on field differences between the target data table and the source data table comprises:
and performing field addition and/or field deletion on the target data table according to the field difference of the target data table and the source data table to obtain the co-structured target data table, wherein the fields in the co-structured target data table are identical to the fields in the source data table.
12. The apparatus of claim 9, wherein the step of co-structuring the target data table based on field differences between the target data table and the source data table further comprises:
and filling the record of the co-structured target data table so as to keep the field values which are the same as the fields of the co-structured target data table in the record of the target data table in the co-structured target data table, wherein the field values corresponding to the other fields in the record of the co-structured target data table are filled by the field values of the corresponding fields of the corresponding record in the source data table.
13. The apparatus according to claim 9, wherein the step of adding and/or deleting fields to the target data table according to the field differences between the target data table and the source data table comprises:
traversing the fields in the target data table and the source data table respectively to determine the missing fields and redundant fields of the target data table relative to the source data table;
establishing a temporary table and setting fields so that the temporary table adds the missing fields and deletes the redundant fields relative to the target data table;
the step of performing record filling on the co-structured target data table comprises the following steps:
filling field values of fields which are the same as the temporary table in the records of the target data table into the temporary table, and filling field values of fields which are different from the target data table in the records of the temporary table by field values of corresponding fields of corresponding records in the source data table;
setting the temporary table as the co-structured target data table.
14. The apparatus of claim 9, wherein the step of merging data based on the co-structured target data table and the source data table comprises:
comparing field values corresponding to identification fields playing an identification role in records of the same structured target data table with those of the source data table to determine missing records and/or redundant records of the same structured target data table relative to the source data table;
adding the missing record and/or deleting the redundant record in the co-structured target data table.
15. The apparatus of claim 9, wherein the step of merging data based on the co-structured target data table and the source data table comprises:
comparing the field values in the records of the structuring target data table with the field values in the records of the source data table to determine records with the same field value corresponding to the marking field playing a role in marking and at least partially different field values corresponding to the rest fields;
and updating the corresponding record in the co-structured target data table by using the record determined in the source data table.
16. The apparatus of claim 9, wherein the step of merging data based on the co-structured target data table and the source data table comprises:
and replacing all records in the co-structured target data table with the records in the source data table.
17. The apparatus of claim 9, wherein the step of merging data based on the co-structured target data table and the source data table comprises:
specifying a protected record or protected field in the co-structured target data table;
and merging the data of the co-structured target data table and the source data table on the basis of skipping the protected record or the protected field.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2017/119588 WO2019127282A1 (en) | 2017-12-28 | 2017-12-28 | Database merging method and apparatus |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109690521A CN109690521A (en) | 2019-04-26 |
| CN109690521B true CN109690521B (en) | 2023-07-04 |
Family
ID=66184335
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201780037501.9A Active CN109690521B (en) | 2017-12-28 | 2017-12-28 | Database merging method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN109690521B (en) |
| WO (1) | WO2019127282A1 (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110490724A (en) * | 2019-07-03 | 2019-11-22 | 平安证券股份有限公司 | The storage method and device of account data |
| CN110502519B (en) * | 2019-08-26 | 2022-04-29 | 北京启迪区块链科技发展有限公司 | Data aggregation method, device, equipment and storage medium |
| CN110502563B (en) * | 2019-08-26 | 2023-09-29 | 腾讯科技（深圳）有限公司 | Data processing method and device of multiple data sources and storage medium |
| CN110765162A (en) * | 2019-09-06 | 2020-02-07 | 上海陆家嘴国际金融资产交易市场股份有限公司 | Data comparison method and device, computer equipment and storage medium |
| CN111143350A (en) * | 2019-11-27 | 2020-05-12 | 深圳壹账通智能科技有限公司 | Enterprise data monitoring method and device, computer equipment and storage medium |
| CN112988698A (en) * | 2019-12-02 | 2021-06-18 | 阿里巴巴集团控股有限公司 | Data processing method and device |
| CN113642309A (en) * | 2020-04-27 | 2021-11-12 | 北京国双科技有限公司 | Data comparison method and related device |
| CN111723210A (en) * | 2020-06-29 | 2020-09-29 | 深圳壹账通智能科技有限公司 | Method and device for storing data table, computer equipment and readable storage medium |
| CN111858760B (en) * | 2020-07-13 | 2024-03-22 | 中国工商银行股份有限公司 | Data processing method and device for heterogeneous database |
| CN112862579A (en) * | 2021-01-21 | 2021-05-28 | 金蝶软件（中国）有限公司 | Material detail data adjusting method and related equipment thereof |
| CN112948435A (en) * | 2021-02-25 | 2021-06-11 | 苏宁金融科技(南京)有限公司 | Data processing method and device and electronic equipment |
| CN113111101A (en) * | 2021-03-18 | 2021-07-13 | 武汉中地众思科技有限公司 | Database conversion method |
| CN113362007A (en) * | 2021-06-24 | 2021-09-07 | 北京天健源达科技股份有限公司 | Method for processing zero stock record |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8510270B2 (en) * | 2010-07-27 | 2013-08-13 | Oracle International Corporation | MYSQL database heterogeneous log based replication |
| CN102542007B (en) * | 2011-12-13 | 2014-06-25 | 中国电子科技集团公司第十五研究所 | Method and system for synchronization of relational databases |
| CN102722540B (en) * | 2012-05-23 | 2018-08-17 | 南京中兴新软件有限责任公司 | Data processing method and device in real-time memory database system |
| CN103778136A (en) * | 2012-10-19 | 2014-05-07 | 阿里巴巴集团控股有限公司 | Cross-room database synchronization method and system |
| US10210207B2 (en) * | 2015-12-31 | 2019-02-19 | International Business Machines Corporation | Parallel execution of merge operations |
| CN106325933B (en) * | 2016-08-24 | 2019-07-02 | 明算科技（北京）股份有限公司 | Batch data synchronous method and device |
-
2017
- 2017-12-28 CN CN201780037501.9A patent/CN109690521B/en active Active
- 2017-12-28 WO PCT/CN2017/119588 patent/WO2019127282A1/en active Application Filing
Also Published As
| Publication number | Publication date |
|---|---|
| CN109690521A (en) | 2019-04-26 |
| WO2019127282A1 (en) | 2019-07-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109690521B (en) | Database merging method and device | |
| US10896157B2 (en) | Clone file backup and restore | |
| CN107122355B (en) | Data migration system and method | |
| CN107122360B (en) | Data migration system and method | |
| US8341128B1 (en) | Concurrency control using an effective change stack and tenant-based isolation | |
| CN106776855B (en) | Processing method for reading Kafka data based on Spark Streaming | |
| US9971827B2 (en) | Subscription for integrating external data from external system | |
| CN107122361B (en) | Data migration system and method | |
| EP3734526A1 (en) | High performance blockchain architecture for logistics services | |
| US20170228396A1 (en) | Pre-Processing Of Geo-Spatial Sensor Data | |
| US6279025B1 (en) | Parallel processing procedure selecting apparatus and method selecting and implementing parallel-executable processing procedures | |
| US20150112947A1 (en) | System and method for database flow management | |
| JP2005302030A (en) | Multi-process access method and device to link list | |
| US10942912B1 (en) | Chain logging using key-value data storage | |
| JP6737063B2 (en) | Software asset management device, software asset management method, and software asset management program | |
| CN106648550B (en) | Method and device for concurrently executing tasks | |
| JP2018085042A (en) | Database management device, information processing system, database management method and database management program | |
| JP6688433B2 (en) | Computer system | |
| US20080155352A1 (en) | Method and system for carrying out an operation based on a log record of a computer program | |
| JP2002197114A (en) | Database management system, customer management system and storage medium | |
| JP4925711B2 (en) | Batch registration information generation apparatus, batch registration information generation method, and batch registration information generation program | |
| JP2008009966A (en) | Device and method for setting business process | |
| JPH07175641A (en) | Distributed program development integration update managing system | |
| JP2002175110A (en) | Production planning device and storage medium | |
| US20090164997A1 (en) | System and method for processing workflow tasks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |