CN109690521A - A kind of method and device of database combining - Google Patents
A kind of method and device of database combining Download PDFInfo
- Publication number
- CN109690521A CN109690521A CN201780037501.9A CN201780037501A CN109690521A CN 109690521 A CN109690521 A CN 109690521A CN 201780037501 A CN201780037501 A CN 201780037501A CN 109690521 A CN109690521 A CN 109690521A
- Authority
- CN
- China
- Prior art keywords
- data
- record
- field
- structured objects
- target matrix
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application discloses a kind of method of database combining and devices, method includes: to obtain target matrix and source data table from target database and source database respectively, wherein the target matrix includes multiple fields and a plurality of record, the source data table also includes multiple fields and a plurality of record, and every record includes field value corresponding with the field respectively；The target matrix handle with structuring according to the field difference of the target matrix and the source data table, to obtain with structured objects tables of data；Data merging is carried out on the basis of the source data table and the same structured objects tables of data, by the above method, the application can be realized the quick merging to database, improve the merging mode of the prior art.
Description
Technical field
This application involves database field is arrived, more particularly to the method and device of a kind of database combining.
Background technique
Database is due to excellent with high load, height response, more concurrent, support atomic operations, support multi version control etc.
Performance, it has also become the preferred manner of data storage.Many softwares and website now, up to cloud computing center, down toward instant chat
Software all carrys out storing data using database.During using database, it is often necessary to be gone more using a source database
A new target database.And at this point, just needing database combining.In addition, when needing integral data there are also other, it is also desirable to
By database combining.
Common merging data library is usually manually completed by database maintenance personnel.Maintenance personnel, which respectively exports, to be merged
Two databases with certain view opening, then compare tool by others and come merging data library, and the degree of automation is relatively low
Under.Certain business databases may provide associated merging tool, but be suitable only for its relevant database product.And
Since some relational databases are as freeware, developer does not also provide related tool, so the merging tool can not
Apply the occasion in some relational databases.There are also certain database service manufacturers, normally tend to provide a whole set of database
Service, asks for high service charge.Correspondingly, such product is generally also huge too fat to move, delay of response, is difficult to safeguard.
Summary of the invention
The application provides the method and device of a kind of database combining, is carried out by light-weighted method to database fast
The merging of speed and safety.
The technical solution that the application uses is: a kind of method of database combining is provided, comprising steps of respectively from mesh
Mark database and source database and obtain target matrix and source data table, wherein the target matrix include multiple fields and
A plurality of record, the source data table also include multiple fields and a plurality of record, every record include respectively with the field pair
The field value answered；The target matrix is carried out with knot according to the field difference of the target matrix and the source data table
Structureization processing, to obtain with structured objects tables of data；In the base of the source data table and the same structured objects tables of data
Data merging is carried out on plinth.
In order to solve the above technical problems, another technical solution that the application uses is: a kind of device of database combining,
It include: processor, memory, the processor connects the memory, and the processor is by running the memory memory
The program of storage executes following steps: target matrix and source data table are obtained from target database and source database respectively, wherein
The target matrix includes multiple fields and a plurality of record, and the source data table also includes multiple fields and a plurality of note
Record, every record include field value corresponding with the field respectively；According to the target matrix and the source data table
Field difference handle with structuring to the target matrix, to obtain with structured objects tables of data；In the source number
According to progress data merging on the basis of table and the same structured objects tables of data.
The beneficial effect of the application is: the mode and device of a kind of database combining is provided, by will be respectively from source
Source data table and the target matrix progress obtained in database and target database is being merged with structuring, thus
It realizes and quick and safe merging is carried out to database by light-weighted method.
Detailed description of the invention
Fig. 1 is the flow diagram of one embodiment of method of the application database combining
Fig. 2 is the sub-process schematic diagram of mono- embodiment of method Fig. 1 step S10 of the application database combining；
Fig. 3 is the sub-process schematic diagram of mono- embodiment of method Fig. 1 step S11 of the application database combining；
Fig. 4 is specific schematic diagram of the Fig. 3 with structuring；
Fig. 5 is the sub-process schematic diagram of mono- embodiment of method Fig. 1 step S12 of the application database combining；
Fig. 6 is that two of the method for the application database combining need combined structured objects tables of data with source data table
Schematic diagram；
Fig. 7 is the another sub-process schematic diagram of mono- embodiment of Fig. 1 step S12；
Fig. 8 is the schematic diagram of the same structured objects tables of data after Fig. 6 merges according to the method for Fig. 7；
Fig. 9 is the another sub-process schematic diagram of mono- embodiment of method Fig. 1 step S12 of the application database combining；
Figure 10 is the schematic diagram of the same structured objects tables of data after Fig. 6 merges according to the method for Fig. 9；
Figure 11 is the another sub-process schematic diagram of mono- embodiment of method Fig. 1 step S12 of the application database combining；
Figure 12 is the schematic diagram of the same structured objects tables of data after Fig. 6 merges according to the method for Figure 11；
Figure 13 is the another sub-process schematic diagram of mono- embodiment of method Fig. 1 step S12 of the application database combining；
Figure 14 is a kind of structural schematic diagram of one embodiment of device of database combining of the application.
Embodiment
Below in conjunction with the attached drawing in the embodiment of the present application, technical solutions in the embodiments of the present application carries out clear, complete
Site preparation description, it is clear that described embodiment is only a part of the embodiment of the application, instead of all the embodiments.It is based on
Embodiment in the application, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall in the protection scope of this application.
Referring to Fig. 1, the flow diagram of one embodiment of method of Fig. 1 the application database combining, and specific seam
Detection method includes the following steps for gap:
S10 obtains target matrix and source data table from target database and source database respectively.
In the present embodiment, due to each tables of data that specific operation object is in database, so first from number of targets
According to target matrix and source data table are obtained in library and in source database respectively.Particular by being stored in target data
The table name of all tables in library is traversed, while being traversed to the table name for all tables being stored in source database, respectively
Corresponding two tables of table name are selected from two databases, as target matrix and source data table.Each target matrix
All include multiple fields and a plurality of record with source data table, and includes multiple field values.Preferably, a field refers to one
One column of a table, one records a line for referring to a table, and field value refers to corresponding to field and record specific one
Value.
Particularly, in the present embodiment, target matrix and source data table have a unique identification field.For one
In a table, all field values under this identification field are all unique, such as number, student status number etc..
S11 handle with structuring to target matrix according to the field difference of target matrix and source data table, with
It obtains with structured objects tables of data.
Before merging, need to carry out same structure operation to target matrix and source data table, that is, make number of targets
According to table and source data table field having the same.It in the present embodiment, is to be operated to target matrix, it is possible to mesh
Mark database carries out backup and carries out read-only statement to source database.It is identical to have target matrix and source data table
Field, need to traverse to target matrix and source data table, it is extra with respect to source data table to obtain target matrix
Or the field lacked, then create an interim table.Wherein interim table relative target tables of data increase the field lacked or
Extra field is deleted, or i.e. increase the field lacked deletes extra field again.In addition, can also be right as needed
Record filling is carried out with structured objects tables of data, so that retaining the note of target matrix in same structured objects tables of data
Field value identical with same structured objects data sheet field in record.It can will be identical as interim table in the record of target matrix
The field value of field is filled into interim table, and by field corresponding to field different from target matrix in the record of interim table
Value is filled by the field value of the corresponding field of corresponding record in source data table.Finally, setting same structuring for interim table
Target matrix, has thus obtained that treated with structured objects tables of data with structuring.
S12 carries out in source data table and on the basis of with structured objects tables of data data merging.
After obtaining same structured objects tables of data and source data table, then to obtained source data table and with structuring mesh
Mark tables of data merges.But in actual operation, user may need to protect partial data, so actual
In merging process, first determines protection data, including protection record and field, also can choose and do not protect, here without limitation,
Then different methods is used according to the different selections of user.
Referring to fig. 2, Fig. 2 is the sub-process signal of mono- embodiment of method Fig. 1 step S10 of the application database combining
Figure, and the embodiment is further expanding on the basis of the application first embodiment step S10, the step is in this embodiment party
Following sub-step is specifically included in formula.
S101 respectively traverses the table name of the table stored in target data and former data.
It is in fact exactly to be operated to the table of database purchase, so needing institute in the practical operation to database
Combined source data table and target matrix is needed to find out.In specific operating process, for needing combined data
A table in library, also with the presence of so corresponding table in source data.Specifically there are corresponding relationships for table name, can be
Table name is same or similar, so traverse respectively to the table name of each table being stored in target data, while with identical
Method traverse source database.
S102 using table name correspondence and is stored respectively in the table in target database and source database as target matrix
With source data table.
For a table in target data, if there are a table names corresponding with the table name of the table one in source data
A table, then using this table in target database as target matrix, using table corresponding to table name in source data as source number
According to table.Here correspondence can be same or similar or other corresponding relationships.
Referring to the sub-process signal that Fig. 3, Fig. 3 are mono- embodiment of method Fig. 1 step S11 of the application database combining
Figure, and the embodiment is further expanding on the basis of the application first embodiment step S11, which specifically wraps
Include following sub-step.
S111 respectively traverses the field in target matrix and source data table, to determine that target matrix is opposite
In field and extra field that source data table lacks.
Since field possible in source data table and target matrix is different, it will lead to data occur when merging in this way
Mistake.So first have to carry out target matrix with the same structure of source data table, that is, make target matrix with
Source data table has identical field.Therefore, the field in target matrix and source data table is traversed, obtains mesh
It marks in tables of data relative to some fields and extra some fields lacked in source data table, and records.
S112 establishes an interim table and carries out field setting so that interim table increased relative to target matrix it is scarce
Few field simultaneously deletes extra field.
In specific database manipulation, accidentally deleting for data is prevented in order to protect data and in operating process, lose etc.
Deng the method for being generally used backup and establishing interim table.So initially setting up an interim table, then set to this interim table
Field is set, so that increasing the field of previous step record lacked for interim table relative target tables of data, deleting previous
The extra field that step is recorded.That is, this interim table and source data table have identical field.
Field value in the record of target matrix with interim table same field is filled into interim table by S113.
Due to being only provided with field to interim table, recorded so also needing to fill interim table, including to interim table
Field identical with target matrix carries out record filling.For, with field identical in target database, just will in interim table
Field value corresponding to this field is successively filled into sequence in the corresponding field of interim table in target matrix.Wherein,
Particularly, since target matrix and source data table have an identification field, so must also have identification field in interim table.
S114, by field value corresponding to field different from target matrix in the record of interim table by right in source data table
The field value for the corresponding field that should be recorded is filled.
Due to interim table relative target tables of data can extra certain field, and these fields due to come from source data table, institute
Field value corresponding to these fields to be not present in target matrix, so the field value of these fields is needed from source data
It is obtained in library.In actual operation, for example a field refers to the age, and one records the information for referring to first, this first
This field value at age is determined together by field and record.So as to learn, for the field being not present, such as exist
There is no the wage of first in target matrix, but the more wage fields of interim table, then the wage of first is if necessary to fill
If, it needs to obtain from source database.Just need the field value of a unique identification field to determine, it is assumed that in table here
Unique identification field be exactly name, for the wage of first, if the field value of the unique identification field in source data table
There is first, then wage corresponding to source data table first is filled into wage corresponding to interim table first, if in source data table
First is not present in the field value of unique identification field, then without filling.
Interim table is set same structured objects tables of data by S115.
Present interim table has had field identical with source data table, that is, has the same structure, simultaneously
Also data are filled with to interim table, that is, are completed with structuring, set same structured objects tables of data for interim table.
It is being please specific schematic diagram of the Fig. 3 with structuring referring particularly to Fig. 4, Fig. 4.Including source data table, number of targets
According to table and interim table.Wherein, in the present embodiment, name is as identification field.
As can be seen that in interim table, for first, in field work number and age exist simultaneously with source data table with
And in target matrix, so obtain field value from target matrix, but for wage field, as identification field,
First is the identification field values that target matrix and source data table share, so the wage of first can correspond to first by source data table
Wage is filled, but for fourth, due in source data table identification field i.e. name below there is no this
Field value is then sky so can not be filled.
Referring to the sub-process signal that Fig. 5, Fig. 5 are mono- embodiment of method Fig. 1 step S12 of the application database combining
Figure, and the embodiment is further expanding on the basis of the application first embodiment step S12, which specifically wraps
Include following sub-step.
S121 specifies protected record or protected field in same structured objects tables of data.
In actual operation, user may need to protect certain data, so it needs to be determined that the number of user's protection
According to needing data to be protected to skip in subsequent merging process, data of these protections can be field, can also be with
It is record, determination needs data to be protected, and protection data here are specifically to specify to the target matrix of same structuring.
S122, to same structured objects tables of data and source data on the basis of skipping protected record or protected field
Table carries out data merging.
It is determining after by protection field or hand protection record, in subsequent merging process, is skipping these data, also
It is in subsequent merging process, these records or field are not involved in merging, also without any change, then to same structure
Change tables of data and source data table merges.
Referring to Fig. 6, Fig. 6 be two of method of the application database combining need combined structured objects tables of data with
The schematic diagram of source data table.
It wherein, is number with the identification field of structural data table and source data table.In other embodiments, here
Identification field with no restriction, can be work number, student number, name etc..Wherein, 1,2,3 be the identification field with structural data table
Field value, a line where 1 is exactly the record corresponding to it, and Zhang San is the field of name field in same structural data table
Value, also with regard to a field value of record corresponding to 1.
Referring to Fig. 7, Fig. 7 is that the another sub-process of mono- embodiment of method Fig. 1 step S12 of the application database combining is shown
It is intended to.And the embodiment is further expanding on the basis of the application first embodiment step S12, and
The one of which of three kinds of merging modes, the embodiment specifically include following sub-step.
S121a will play the identification field institute of mark action in the record of same structured objects tables of data and source data table
Corresponding field value compares.
In the present embodiment, it needs to compare and combine by identification field, that is, carrys out comparison structure number of targets
According to the field value of the identification field of table and source data table.As the identification field of source data table and structured objects tables of data is
Number, then each number value of structured objects tables of data is compared with each number value of source data table.
S122a determines the record and/or extra record lacked with structured objects tables of data relative to source data table.
After being compared, the field value institute with the structural data table identification field different relative to source data table is determined
Corresponding record.That is, as the identification field of structural data table and source data table is number, in same structuring number
Number value according to table hypothesis number has 1,2,3, and the number value of the number in source data table has 3,4,5, then same structural data
Table has lacked 4,5 for source data table, that is, record corresponding to having lacked 4,5, and extra 1,2, that is, it is more
Record corresponding to remaining 1,2, is determined it and records.
S123a adds the record/lacked in same structured objects tables of data or deletes the extra record.
The record lacked determined by being added in same structured objects tables of data.Namely as above-mentioned steps, add
With structured objects tables of data relative to namely 4 and 5 corresponding records are recorded lacking in source data, extra note is deleted
Record, that is, delete with structured objects tables of data relative to recording namely 1 and 2 corresponding notes lacking in source data
Record.
It is the signal of the same structured objects tables of data after Fig. 6 merges according to the method for Fig. 7 further referring to Fig. 8, Fig. 8
Figure.
Structured objects tables of data is for source data table, record corresponding to extra 1 and 2, but is the absence of
Record corresponding to 4 and 5.So for target matrix, it is right to delete 1 and 2 institutes in merging data table
The record answered, be added to 4 and 5 corresponding to record, do not modify to record corresponding to 3, obtain Fig. 8.
Referring to Fig. 9, Fig. 9 is that the another sub-process of mono- embodiment of method Fig. 1 step S12 of the application database combining is shown
It is intended to, and the embodiment is further expanding on the basis of the application first embodiment step S12, and
The one of which of three kinds of merging modes.The embodiment specifically includes following sub-step.
S121b compares same structured objects tables of data and the field value in the record of source data table.
In the present embodiment, it needs to compare and combine by identification field, that is, carrys out comparison structure number of targets
According to the field value of the identification field of table and source data table.As the identification field of source data table and structured objects tables of data is
Number, then each number value of structured objects tables of data is compared with each number value of source data table.
S122b, determine play that field value corresponding to the identification field of mark action is identical but remaining field corresponding to
At least partly different record of field value.
That is, as the identification field of structural data table and source data table is number, in same structural data
Table assumes that the number value of number has 1,2,3, and the number value of the number in source data table has 3,4,5, then has a common word
Segment value 3.Determining 3 records corresponding in structured objects tables of data and 3 records corresponding in source data table,
Determine that the two record different field values.
S123b is updated using the record determined in source data table with the corresponding record in structured objects tables of data.
That is, source data table is different from structured objects number the certain field value in record corresponding to 3
According to the certain field value in record corresponding to table 3, then this field value in same structured objects tables of data is substituted for source
This field value of tables of data, other are remained unchanged.
It is showing for the same structured objects tables of data after Fig. 6 merges according to the method for Fig. 9 further referring to Figure 10, Figure 10
It is intended to.
For structured objects tables of data for source data table, the identification field of structured objects tables of data has 1,2,3,
The identification field of source data table has 3,4,5, then the identification field of two tables of data shares one identical 3.In structural data
In table, record corresponding to 3 are as follows: 3, Xie Li, 3K.And in source data table, record corresponding to 3 are as follows: 3, king two, 6K.Then show
So, 3 in structural data table corresponding to record field value and source data table in record corresponding to 3 field value not
Together, then by record corresponding to 3 in structural data table be substituted for 3 in source data table corresponding to record, obtain Figure 10.
Referring to Figure 11, Figure 11 is the another sub-process of mono- embodiment of method Fig. 1 step S12 of the application database combining
Schematic diagram, and the embodiment is further expanding on the basis of the application first embodiment step S12 and three kinds of merging
The one of which of mode, the embodiment specifically include following sub-step.
S121c, the record record in same structured objects tables of data being all substituted in source data table.
That is, structured objects tables of data is emptied, and the record of all source data tables is filled, so that
The record overlay structure target matrix of source data table.
Further referring to the same structured objects tables of data that Figure 12, Figure 12 are after Fig. 6 merges according to the method for Figure 11
Schematic diagram.
Source data table is completely covered into structural data table, that is, the record in same structured objects tables of data is complete
Portion is substituted for the record in source data table, obtains Figure 12.
Referring to Figure 13, Figure 13 is the another sub-process of mono- embodiment of method Fig. 1 step S12 of the application database combining
Schematic diagram, and the embodiment is further expanding on the basis of the application first embodiment step S12.And it needs to pay attention to
, the present embodiment is the schematic diagram to the embodiment of the concrete methods of realizing of Fig. 7, Fig. 9 and Figure 11 embodiment, and specifically has
Following step:
S121d is cyclically read from source data table using first thread and is recorded, and is stored to the first buffer area.
In this embodiment, combined realization is mainly illustrated by concrete implementation method.Here by using reference
Multiple threads realize, first using reading a line record in the slave source data table of first thread circulation, and are stored as a word
Tandem table is accorded with, and this character string list is stored in the first buffer area, recycles aforesaid operations always, until the first caching
Area is fully written.
S122d using the second thread loops reads record from the first buffer area, and stores to the second buffer area.
A record is taken out from the first buffer area using the second thread loops, and is placed in the second buffer area.It needs to infuse
Meaning, the second buffer area are smaller than the first buffer area.If first buffering area be empty or the second buffer area be it is full,
Second thread enters sleep state, that is, does not work.After being written successfully, due to the again empty position of the first buffer area,
Then the first buffer area is written using first thread.
S123d using third thread loops reads record from same structured objects tables of data, and stores slow to third
Deposit area.
After operating source data table, need exist for operating to same structured objects tables of data.It adopts first
A line record is read from same structured objects tables of data with third thread loops, and is stored as a character string list, and
This character string list is stored in third buffer area.Aforesaid operations are recycled always, until the first buffer area is fully written.
S124d is read record from third buffer area using the 4th thread loops, and stored to the 4th buffer area.
One record is taken out using the slave third buffer area of the 4th thread loops, and is placed in the 4th buffer area.It needs to infuse
Meaning, the 4th buffer area is smaller than third buffer area, if third buffer area be empty or the 4th buffer area be it is full,
4th thread enters sleep state, that is, does not work.After being written successfully, due to the again empty position of third buffer area,
Then third buffer area is written using third thread.
S125d, what the record and the 4th buffer area stored using the 5th thread to the second buffer area was stored
Record is compared, and executes corresponding data union operation according to comparison result.
That is, when the second buffer area and the 4th buffer area are not skies, then using the 5th thread to second
The record that the record that buffer area is stored is stored with the 4th buffer area compares, and passes through the result and selection compared
Merging mode is increased, deletes, replaces etc. operation.After the completion of comparing and operating, remove the second buffer area and
4th buffer area, and the second thread of wake-up and the 4th thread work.
4, Figure 14 is a kind of structural schematic diagram of one embodiment of device of database combining of the application refering to fig. 1.
In the present embodiment, which includes processor 10 and memory 11, and processor 10 connects memory 11.
Wherein, processor 10 is arranged interim table and is filled to interim table, to database for determining protection data
And tables of data is traversed, and tables of data is compared and is merged by using the mode of multithreading and more buffer areas.
Wherein, processor 10 can also be known as CPU (Central Processing Unit, central processing unit).Processing
Device 10 may be a kind of IC chip, the processing capacity with signal.Processor 10 can also be general processor, number
Signal processor (DSP), specific integrated circuit (ASIC), ready-made programmable gate array (FPGA) or other programmable logic devices
Part, discrete gate or transistor logic, discrete hardware components.General processor can be microprocessor or the processor
It is also possible to any conventional processor etc..
The modules terminal of above equipment can execute step corresponding in above method embodiment respectively, therefore not right herein
Each module is repeated, and please refers to the explanation of the above corresponding step in detail.
The storage device storage 20 of the application can be realized the methodical command file of above-mentioned institute, which can be with
It is stored in above-mentioned storage device in the form of software products, while still recording the data of various calculating, including some instructions
With so that a computer equipment (can be personal computer, server or the network equipment etc.) or processor
(processor) all or part of the steps of each embodiment method of the application is executed.And storage device above-mentioned includes: U
Disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access
Memory), the various media that can store program code such as magnetic or disk or computer, server, mobile phone, flat
The terminal devices such as plate.
In conclusion it should be readily apparent to one skilled in the art that the application provides a kind of method and device in merging data library,
It takes out two tables of data from two databases respectively by way of traversal, same structure then is being carried out to two tables of data
Change, after determining protection data, by way of multithreading and buffer area two tables of data carried out with the merging of various ways,
To quickly finish merging, and in this application, there are perfect data protection schemes, not will cause in merging process
The loss or damage of data.
The above is only presently filed embodiments, are not intended to limit the scope of the patents of the application, all to utilize the application
Equivalent structure or equivalent flow shift made by specification and accompanying drawing content is applied directly or indirectly in other relevant technologies
Field similarly includes in the scope of patent protection of the application.
Claims (20)
1. a kind of method in merging data library, which comprises the steps of:
Target matrix and source data table are obtained from target database and source database respectively, wherein the target matrix includes
Multiple fields and a plurality of record, the source data table also include multiple fields and a plurality of record, and every record includes difference
Field value corresponding with the field；
The target matrix is carried out at structuring according to the field difference of the target matrix and the source data table
Reason, to obtain with structured objects tables of data；
Data merging is carried out on the basis of the source data table and the same structured objects tables of data.
2. the method according to claim 1, wherein described obtain mesh from target database and source database respectively
Mark tables of data and the step of source data table include:
The table name of the table stored respectively to the target database and the source database traverses, and by the table name
The corresponding and table that is stored respectively in the target database and the source database is as the target matrix and described
Source data table.
3. the method according to claim 1, wherein described according to the target matrix and the source data table
Field difference to the target matrix carry out with structuring handle the step of include:
According to the field difference of the target matrix and the source data table to the target matrix carry out field addition and/
Or field is deleted, to obtain the same structured objects tables of data, wherein field in the same structured objects tables of data with
Field in the source data table is identical.
4. according to the method described in claim 3, it is characterized in that, described according to the target matrix and the source data table
Field difference to the target matrix carry out with structuring handle the step of further include:
Record filling is carried out to the same structured objects tables of data, so that retaining in the same structured objects tables of data
Field value identical with the same structured objects data sheet field in the record of the target matrix, and the same structuring
Field value corresponding to remaining field in the record of target matrix by the corresponding record in the source data table correspondence word
The field value filling of section.
5. according to the method described in claim 4, it is characterized in that, described according to the target matrix and the source data table
Field difference the step of field addition and/or field are deleted carried out to the target matrix include:
The field in the target matrix and the source data table is traversed respectively, with the determination target matrix phase
The field and extra field lacked for the source data table；
Establish an interim table and carry out field setting so that the interim table increased relative to the target matrix it is described
The field that lacks simultaneously deletes the extra field；
It is described to same structured objects tables of data carry out record filling the step of include:
Field value in the record of the target matrix with the interim table same field is filled into the interim table, and
By field value corresponding to field different from the target matrix in the record of the interim table by right in the source data table
The field value for the corresponding field that should be recorded is filled；
The same structured objects tables of data is set by the interim table.
6. the method according to claim 1, wherein described in same structured objects tables of data and the source data
The step of progress data merging, includes: on the basis of table
The identification field institute that mark action is played in the same structured objects tables of data and the record of the source data table is right
The field value answered compares, the record lacked with the determination same structured objects tables of data relative to the source data table
And/or extra record；
The record and and/or the deletion extra record lacked described in addition in the same structured objects tables of data.
7. the method according to claim 1, wherein described in same structured objects tables of data and the source data
The step of progress data merging, includes: on the basis of table
The same structured objects tables of data and the field value in the record of the source data table are compared, played with determination
Field value corresponding to the identification field of mark action is identical but remaining field corresponding at least partly different note of field value
Record；
The corresponding record in the same structured objects tables of data is updated using the record determined in the source data table.
8. the method according to claim 1, wherein described in same structured objects tables of data and the source data
The step of progress data merging, includes: on the basis of table
Record record in the same structured objects tables of data being all substituted in the source data table.
9. the method according to claim 1, wherein described in same structured objects tables of data and the source data
The step of progress data merging, includes: on the basis of table
Protected record or protected field are specified in the same structured objects tables of data；
To the same structured objects tables of data and the source on the basis of skipping the protected record or protected field
Tables of data carries out data merging.
10. the method according to claim 1, wherein described in same structured objects tables of data and the source number
Include: according to the step of data merging is carried out on the basis of table
Record is cyclically read from the source data table using first thread, and is stored to the first buffer area；
Record using the second thread loops is read from first buffer area, and is stored to the second buffer area, wherein described
The record quantity that first buffer area is stored is greater than the record quantity that second buffer area is stored；
Record using third thread loops is read from the same structured objects tables of data, and is stored and arrived third buffer area；
It using the 4th thread loops reads and records from third buffer area, and the 4th buffer area is written, wherein the third caches
The record quantity that area is stored is greater than the record quantity that the 4th buffer area is stored；
The record that the record and the 4th buffer area stored using the 5th thread to second buffer area is stored into
Row compares, and executes corresponding data union operation according to comparison result.
11. a kind of device in merging data library, which is characterized in that described device includes: processor, memory, the processor
The memory is connected, the processor executes following steps by running the program stored in the memory:
Target matrix and source data table are obtained from target database and source database respectively, wherein the target matrix includes
Multiple fields and a plurality of record, the source data table also include multiple fields and a plurality of record, and every record includes difference
Field value corresponding with the field；
The target matrix is carried out at structuring according to the field difference of the target matrix and the source data table
Reason, to obtain with structured objects tables of data；
Data merging is carried out on the basis of the source data table and the same structured objects tables of data.
12. device according to claim 11, which is characterized in that described to be obtained respectively from target database and source database
The step of target matrix and source data table includes:
The table name of the table stored respectively to the target database and the source database traverses, and by the table name
The corresponding and table that is stored respectively in the target database and the source database is as the target matrix and described
Source data table.
13. device according to claim 11, which is characterized in that described according to the target matrix and the source data
The field difference of table carries out the target matrix
According to the field difference of the target matrix and the source data table to the target matrix carry out field addition and/
Or field is deleted, to obtain the same structured objects tables of data, wherein field in the same structured objects tables of data with
Field in the source data table is identical.
14. device according to claim 13, which is characterized in that described according to the target matrix and the source data
The field difference of table carries out the step of handling with structuring to the target matrix further include:
Record filling is carried out to the same structured objects tables of data, so that retaining in the same structured objects tables of data
Field value identical with the same structured objects data sheet field in the record of the target matrix, and the same structuring
Field value corresponding to remaining field in the record of target matrix by the corresponding record in the source data table correspondence word
The field value filling of section.
15. device according to claim 14, which is characterized in that described according to the target matrix and the source data
The field difference of table carries out the step of field addition and/or field are deleted to the target matrix
The field in the target matrix and the source data table is traversed respectively, with the determination target matrix phase
The field and extra field lacked for the source data table；
Establish an interim table and carry out field setting so that the interim table increased relative to the target matrix it is described
The field that lacks simultaneously deletes the extra field；
It is described to same structured objects tables of data carry out record filling the step of include:
Field value in the record of the target matrix with the interim table same field is filled into the interim table, and
By field value corresponding to field different from the target matrix in the record of the interim table by right in the source data table
The field value for the corresponding field that should be recorded is filled；
The same structured objects tables of data is set by the interim table.
16. device according to claim 11, which is characterized in that described in same structured objects tables of data and the source number
Include: according to the step of data merging is carried out on the basis of table
The identification field institute that mark action is played in the same structured objects tables of data and the record of the source data table is right
The field value answered compares, the record lacked with the determination same structured objects tables of data relative to the source data table
And/or extra record；
The record and and/or the deletion extra record lacked described in addition in the same structured objects tables of data.
17. device according to claim 11, which is characterized in that described in same structured objects tables of data and the source number
Include: according to the step of data merging is carried out on the basis of table
The same structured objects tables of data and the field value in the record of the source data table are compared, played with determination
Field value corresponding to the identification field of mark action is identical but remaining field corresponding at least partly different note of field value
Record；
The corresponding record in the same structured objects tables of data is updated using the record determined in the source data table.
18. device according to claim 11, which is characterized in that described in same structured objects tables of data and the source number
Include: according to the step of data merging is carried out on the basis of table
Record record in the same structured objects tables of data being all substituted in the source data table.
19. device according to claim 11, which is characterized in that described in same structured objects tables of data and the source number
Include: according to the step of data merging is carried out on the basis of table
Protected record or protected field are specified in the same structured objects tables of data；
To the same structured objects tables of data and the source on the basis of skipping the protected record or protected field
Tables of data carries out data merging.
20. device according to claim 11, which is characterized in that described in same structured objects tables of data and the source number
Include: according to the step of data merging is carried out on the basis of table
Record is cyclically read from the source data table using first thread, and is stored to the first buffer area；
Record using the second thread loops is read from first buffer area, and is stored to the second buffer area, wherein described
The record quantity that first buffer area is stored is greater than the record quantity that second buffer area is stored；
Record using third thread loops is read from the same structured objects tables of data, and is stored and arrived third buffer area；
It using the 4th thread loops reads and records from third buffer area, and the 4th buffer area is written, wherein the third caches
The record quantity that area is stored is greater than the record quantity that the 4th buffer area is stored；
The record that the record and the 4th buffer area stored using the 5th thread to second buffer area is stored into
Row compares, and executes corresponding data union operation according to comparison result.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2017/119588 WO2019127282A1 (en) | 2017-12-28 | 2017-12-28 | Database merging method and apparatus |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109690521A true CN109690521A (en) | 2019-04-26 |
| CN109690521B CN109690521B (en) | 2023-07-04 |
Family
ID=66184335
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201780037501.9A Active CN109690521B (en) | 2017-12-28 | 2017-12-28 | Database merging method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN109690521B (en) |
| WO (1) | WO2019127282A1 (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110490724A (en) * | 2019-07-03 | 2019-11-22 | 平安证券股份有限公司 | The storage method and device of account data |
| CN110502519A (en) * | 2019-08-26 | 2019-11-26 | 北京启迪区块链科技发展有限公司 | A kind of method, apparatus of data aggregate, equipment and storage medium |
| CN110502563A (en) * | 2019-08-26 | 2019-11-26 | 腾讯科技（深圳）有限公司 | A kind of processing method and processing device of the data of multi-data source, storage medium |
| CN110765162A (en) * | 2019-09-06 | 2020-02-07 | 上海陆家嘴国际金融资产交易市场股份有限公司 | Data comparison method and device, computer equipment and storage medium |
| CN112862579A (en) * | 2021-01-21 | 2021-05-28 | 金蝶软件（中国）有限公司 | Material detail data adjusting method and related equipment thereof |
| CN112948435A (en) * | 2021-02-25 | 2021-06-11 | 苏宁金融科技(南京)有限公司 | Data processing method and device and electronic equipment |
| CN113111101A (en) * | 2021-03-18 | 2021-07-13 | 武汉中地众思科技有限公司 | Database conversion method |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111143350A (en) * | 2019-11-27 | 2020-05-12 | 深圳壹账通智能科技有限公司 | Enterprise data monitoring method and device, computer equipment and storage medium |
| CN112988698B (en) * | 2019-12-02 | 2024-09-17 | 阿里巴巴集团控股有限公司 | Data processing method and device |
| CN113642309B (en) * | 2020-04-27 | 2024-07-26 | 北京国双科技有限公司 | Data comparison method and related device |
| CN111723210A (en) * | 2020-06-29 | 2020-09-29 | 深圳壹账通智能科技有限公司 | Method and device for storing data table, computer equipment and readable storage medium |
| CN111858760B (en) * | 2020-07-13 | 2024-03-22 | 中国工商银行股份有限公司 | Data processing method and device for heterogeneous database |
| CN111985194A (en) * | 2020-09-03 | 2020-11-24 | 深圳壹账通智能科技有限公司 | Data storage method and device, electronic equipment and storage medium |
| CN113362007B (en) * | 2021-06-24 | 2024-05-28 | 北京天健源达科技股份有限公司 | Zero inventory record processing method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102542007A (en) * | 2011-12-13 | 2012-07-04 | 中国电子科技集团公司第十五研究所 | Method and system for synchronization of relational databases |
| CN102722540A (en) * | 2012-05-23 | 2012-10-10 | 中兴通讯股份有限公司 | Method and device for processing data in real-time memory database system |
| CN103778136A (en) * | 2012-10-19 | 2014-05-07 | 阿里巴巴集团控股有限公司 | Cross-room database synchronization method and system |
| CN106325933A (en) * | 2016-08-24 | 2017-01-11 | 明算科技（北京）股份有限公司 | Method and device for synchronizing batch data |
| US20170193043A1 (en) * | 2015-12-31 | 2017-07-06 | International Business Machines Corporation | Parallel execution of merge operations |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8510270B2 (en) * | 2010-07-27 | 2013-08-13 | Oracle International Corporation | MYSQL database heterogeneous log based replication |
-
2017
- 2017-12-28 CN CN201780037501.9A patent/CN109690521B/en active Active
- 2017-12-28 WO PCT/CN2017/119588 patent/WO2019127282A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102542007A (en) * | 2011-12-13 | 2012-07-04 | 中国电子科技集团公司第十五研究所 | Method and system for synchronization of relational databases |
| CN102722540A (en) * | 2012-05-23 | 2012-10-10 | 中兴通讯股份有限公司 | Method and device for processing data in real-time memory database system |
| CN103778136A (en) * | 2012-10-19 | 2014-05-07 | 阿里巴巴集团控股有限公司 | Cross-room database synchronization method and system |
| US20170193043A1 (en) * | 2015-12-31 | 2017-07-06 | International Business Machines Corporation | Parallel execution of merge operations |
| CN106325933A (en) * | 2016-08-24 | 2017-01-11 | 明算科技（北京）股份有限公司 | Method and device for synchronizing batch data |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110490724A (en) * | 2019-07-03 | 2019-11-22 | 平安证券股份有限公司 | The storage method and device of account data |
| CN110502519A (en) * | 2019-08-26 | 2019-11-26 | 北京启迪区块链科技发展有限公司 | A kind of method, apparatus of data aggregate, equipment and storage medium |
| CN110502563A (en) * | 2019-08-26 | 2019-11-26 | 腾讯科技（深圳）有限公司 | A kind of processing method and processing device of the data of multi-data source, storage medium |
| CN110502519B (en) * | 2019-08-26 | 2022-04-29 | 北京启迪区块链科技发展有限公司 | Data aggregation method, device, equipment and storage medium |
| CN110502563B (en) * | 2019-08-26 | 2023-09-29 | 腾讯科技（深圳）有限公司 | Data processing method and device of multiple data sources and storage medium |
| CN110765162A (en) * | 2019-09-06 | 2020-02-07 | 上海陆家嘴国际金融资产交易市场股份有限公司 | Data comparison method and device, computer equipment and storage medium |
| CN112862579A (en) * | 2021-01-21 | 2021-05-28 | 金蝶软件（中国）有限公司 | Material detail data adjusting method and related equipment thereof |
| CN112948435A (en) * | 2021-02-25 | 2021-06-11 | 苏宁金融科技(南京)有限公司 | Data processing method and device and electronic equipment |
| CN113111101A (en) * | 2021-03-18 | 2021-07-13 | 武汉中地众思科技有限公司 | Database conversion method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109690521B (en) | 2023-07-04 |
| WO2019127282A1 (en) | 2019-07-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109690521A (en) | A kind of method and device of database combining | |
| US20160350302A1 (en) | Dynamically splitting a range of a node in a distributed hash table | |
| US7162693B2 (en) | Process for managing data in which existing data item is moved to neighbor page before insertion or after deletion of another data item | |
| US10606865B2 (en) | Database scale-out | |
| CN108959510B (en) | Partition level connection method and device for distributed database | |
| CN101566986A (en) | Method and device for processing data in online business processing | |
| CN103597440A (en) | Method for creating clone file, and file system adopting the same | |
| CN104508666A (en) | Cataloging backup data | |
| TW201941189A (en) | Recording device, reading device, recording method, recording program storage medium, reading method, reading program storage medium, and magnetic tape | |
| CN109597707A (en) | Clone volume data copying method, device and computer readable storage medium | |
| CN102804137B (en) | Controlling access to software component state | |
| CN107632779A (en) | Data processing method and device, server | |
| CN109857519A (en) | The processing method and relevant apparatus of virtual disk | |
| JPS62226375A (en) | Picture information retrieving system | |
| CN118193032B (en) | Method, apparatus, device, medium and program product for eliminating invalid dependency library | |
| WO2024213024A1 (en) | Data access method and data processing system | |
| JPH0225946A (en) | File controller | |
| JPH04115332A (en) | File managing device | |
| JP2009026292A (en) | Data distribution method, data distribution program, and parallel database system | |
| JPH0785079A (en) | Device for managing information file | |
| JP2023068981A (en) | Information processing device, information processing method, and information processing program | |
| CN105260231A (en) | Method of reducing physical disk I/O reading and writing | |
| US8010741B1 (en) | Methods and apparatus for controlling migration of content | |
| CN109582638A (en) | Snapshot volume data copying method, device and computer readable storage medium | |
| JP2658097B2 (en) | Secondary file creation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |