CN102999537B - System and method for data migration - Google Patents
System and method for data migration Download PDFInfo
- Publication number
- CN102999537B CN102999537B CN201110278784.0A CN201110278784A CN102999537B CN 102999537 B CN102999537 B CN 102999537B CN 201110278784 A CN201110278784 A CN 201110278784A CN 102999537 B CN102999537 B CN 102999537B
- Authority
- CN
- China
- Prior art keywords
- data
- plug
- write
- reading
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a system and a method for data migration. The system specifically comprises a configuration reading module, a reading module, a data exchange buffer module and a writing module, wherein the configuration reading module is used for reading a configuration file of a data migration task, and the configuration file comprises data reading information and data writing information; the reading module is used for reading the data according to the data reading information in the configuration file and converting the read data into an intermediate format to be written into the data exchange buffer module; the data exchange buffer module is used for temporarily storing the written intermediate format data; and the writing module is used for reading the intermediate format data from the data exchange buffer module, writing the information according to the data in the configuration file and converting the intermediate format data into a target format to be written into a target address. By the system, the data migration performance can be improved, and the maintenance cost and the development cost which are spent on data migration are lowered.
Description
Technical field
The application is related to computer system data processing technology field, more particularly to a kind of data mover system and side
Method.
Background technology
At present, the increase of the development with network technology and quantity of information, network defines enormous amount distribution different
Structure information source.Inside each enterprises and institutions due to the difference in geographical position or the increase of new demand also form one distributed, different
Structure and enormous amount data acquisition system.One of the data source appearance of isomery is due to multiple business datas the reason critically important
Exist while storehouse.
Heterogeneous data source is various informative, to distinguish from structure, and it not only can include data base's (as relational database)
Middle structural data, but also non-structured data in file system can be included, as text, electronics postal
Part, PowerPoint and various document etc..These data sources are often distributed on geographical position, and data form and code be not same,
These have all pushed directly on the generation of the demand of the data sharing to heterogeneous data source and transparent access.Therefore, how by data
Being migrated between these heterogeneous data sources, thus maintaining a consistent, complete global schema, being had great importance.
In the prior art, conventional data migration method mainly adopts derivation/lead-in mode.Assume be intended to by data from
Oracle database migrates to hdfs (distributed file system, hadoop distributed file system) system, then
Source data is exported as file initially with the data export (swf) command of oracle or instrument by prior art, then utilizes instrument by literary composition
Part form, coding copy in hdfs system after being converted.Prior art has the drawback that
1st, the derivation of data and transcoding process are required for disk dump, because disk dump needs substantial amounts of disk i/o (defeated
Enter/export, input/output), and disk is slow devices, the process CIMS that this results in Data Migration is long, Data Migration
Degraded performance；In addition, in long Data Migration flow process, user not only needs oneself linking upstream and downstream process, to each mistake
Journey is monitored, but also disk space to be monitored, which greatly enhances O＆M cost；
Although 2, data exporting instrument ripe at present is relatively more, general one or several all can only be supported
Individual certain types of data base；Once having a lot of heterogeneous data sources, the such as data base/literary composition such as mysql, oracle, rac, hive
Part system, and need often between these heterogeneous data sources, to import and export data, then need exploitation/maintenance/study to make
With a collection of such instrument, and the increase with data source types, the utility number of needs can linearly increase
Trend, and the exploitation of utility can spend substantial amounts of development cost.
Content of the invention
Technical problems to be solved in this application are to provide a kind of data mover system and method, to improve Data Migration
Performance, reduce the O＆M cost that spent of Data Migration and development cost.
In order to solve the above problems, this application discloses a kind of data mover system, comprising:
Configuration read module, for reading the configuration file of data migration task, includes data in described configuration file
Reading information data writes information；
Read module, for according to the data read information in described configuration file, carrying out the reading of data, and will read
After the data taking is converted to intermediate form, write data exchange buffer module；
Data exchange buffer module, for the data of the intermediate form of interim storage write；And
Writing module, for reading the data of described intermediate form, and in foundation from described data exchange buffer module
Data write information in described configuration file is converted to the data of described intermediate form after object format, write target ground
Location.
On the other hand, disclosed herein as well is a kind of data migration method, comprising:
Read the configuration file of data migration task, in described configuration file, include the write of data read information data
Information；
According to the data read information in described configuration file, carry out the reading of data；
The data of reading is converted to intermediate form, and writes data exchange buffer module, wherein, described data exchange is delayed
Die block is used for the data of the intermediate form of interim storage write；
The data of described intermediate form is read from described data exchange buffer module；
According to the data write information in described configuration file, the data of described intermediate form is converted to object format, and
Write destination address.
Preferably, methods described also includes:
According to the data read information in described configuration file, load the corresponding plug-in unit data exchange buffering that reads and insert
Part, and provide corresponding running environment to described reading plug-in unit, by described reading plug-in unit according to the data in described configuration file
Reading information, carries out the reading of data, and writes data exchange cushioning insert after the data of reading is converted to intermediate form,
Read the data of the intermediate form of plug-in unit write by described data exchange cushioning insert interim storage；
According to the data write information in described configuration file, load corresponding write plug-in unit, and to said write plug-in unit
Corresponding running environment is provided, is read the number of described intermediate form by said write plug-in unit from described data exchange cushioning insert
According to, and after the data of described intermediate form being converted to object format according to the data write information in described configuration file,
Write destination address.
Preferably, methods described also includes:
Before the reading carrying out data, a Data Migration Task-decomposing is become multiple reading subtasks；
Described according to the data read information in described configuration file, the step carrying out the reading of data is, in multithreading
Under the reading running environment in pond, carry out the reading of the plurality of reading subtask
Preferably, methods described also includes:
Before said write module is by the data write destination address of object format, a Data Migration Task-decomposing is become
Multiple write subtasks；
The step of said write destination address is, under the write running environment of described multithread pool, will with the plurality of
The data write destination address of the corresponding object format in write subtask.
Preferably, described data exchange buffer module is the core buffer based on double buffering queue.
Preferably, methods described also includes:
In the running of described read module and/or data exchange buffer module and/or writing module, monitoring is corresponding
Operational factor, described operational factor includes reading and/or interim storage and/or the write line number of data and/or byte number.
Preferably, described data read information includes one or more of following information: the ip address of source database, end
Slogan, library name, login username, login password, data base's table name, filtercondition；
Described data write information includes one or more of following information: data write path, datum target form.
Preferably, described configuration file is extensible markup language document, the name of described extensible markup language document
Rule is source table name _ purpose table name _ source database type _ purpose type of database .xml.
Preferably, methods described also includes:
Deposit the plug-in description file of plug-in unit；
According to the loading plug-in unit information needed of storage in described plug-in description file, corresponding plug-in unit is registered, and
Store corresponding plug-in registration information；
The described step loading corresponding reading plug-in unit data exchange buffering plug-in unit, comprising:
Plug-in registration information by storage in the data read information in described configuration file, with described plug-in registration module
Mated, if the match is successful, loaded plug-in unit corresponding with described plug-in registration information, described loading process includes: according to
Generate corresponding plug-in unit example according to described plug-in description file, and carry out the initialization of plug-in unit according to described plug-in unit example；
The described step loading corresponding write plug-in unit, comprising:
Plug-in registration information by storage in the data write information in described configuration file, with described plug-in registration module
Mated, if the match is successful, loaded plug-in unit corresponding with described plug-in registration information, described loading process includes: according to
Generate corresponding plug-in unit example according to described plug-in description file, and carry out the initialization of plug-in unit according to described plug-in unit example.
Compared with prior art, the application has the advantage that
First, the application carries out Data Migration according to configuration file, and, in data migration process, is handed over using data
Change the data that buffer module interim storage reads from source address, because the data in data exchange buffer module is removed immediately simultaneously
Write to destination address, therefore whole data migration process is without disk transfer.Accordingly, with respect to prior art, number can be shortened
According to the flow process of migration, improve the performance of Data Migration；In addition, in Data Migration flow process, user need not monitor disk space, therefore
O＆M cost can also be reduced.
Secondly, regardless of the source format of data, the data of reading can be converted to prespecified, write by the application
The intermediate form that module is capable of identify that, and carry out interim storage such that it is able to realize the universal buffering management to data, to write
Enter module and according to data write information, the data of this intermediate form can be converted to object format, and write destination address；Institute
State the functional independence that intermediate form makes read module and writing module, so, the exploitation of read module and writing module can
Individually carry out, can either quickly support new heterogeneous data source, development difficulty and development cost can be reduced again.
Furthermore, the application can also carry out the reading of the plurality of reading subtask using thread pool, and/or, multiple write
Enter the write of subtask, and be independent of each other between any two read/write subtask, therefore, it is possible to realize reading parallel and/
Or it is written in parallel to such that it is able to greatly shorten Data Migration flow process, improve Data Migration performance further simultaneously.
Further, with respect to prior art, in long Data Migration flow process, user not only needs oneself linking upstream and downstream
Process, is monitored to each process, but also disk space to be monitored；The application can in short Data Migration flow process automatically
Assume the operational factor of Data Migration for user, for example, currently have read how many byte numbers, interim storage how many byte numbers,
It is written with how many byte numbers, described present so that user is very clear to Data Migration flow process.
Finally, the application can adopt the mechanical development data mover system of framework+plug-in unit, as the framework of system, inserts
Part formula Data Migration platform is equivalent to a data transfer platform, and plug-in unit then be access different types of data source (data base/
File system) provide and realize, support with the access function of each data source of Informal development of plug-in unit；So, user can be very square
Just past plug-in type Data Migration platform increases new plug-in unit, make plug-in type Data Migration platform quick support new data base or
Person's file system, the motility of strengthening system and autgmentability；Meanwhile, with respect to prior art, in order to access a new data
Source, needs substantial amounts of manpower development and test, could develop a set of utility, and the plug-in unit of the application only need less
Workload and development cost, therefore, it is possible to substantially reduce development cost.
Brief description
Fig. 1 is a kind of structure chart of the application data mover system embodiment 1；
Fig. 2 is the schematic diagram of Data Migration in a kind of enterprise of the application；
Fig. 3 is a kind of structure chart of the application data mover system embodiment 2；
Fig. 4 is a kind of flow chart of the application data migration method embodiment 1；
Fig. 5 is a kind of flow chart of the application data migration method embodiment 2.
Specific embodiment
Understandable for enabling the above-mentioned purpose of the application, feature and advantage to become apparent from, below in conjunction with the accompanying drawings and specifically real
Mode of applying is described in further detail to the application.
One of core idea of the embodiment of the present application is, carries out Data Migration according to configuration file, and, moves in data
During shifting, the data that read from source address using data exchange buffer module interim storage, due to data exchange buffer module
In data removed immediately and write to destination address, therefore whole data migration process is without disk transfer.Accordingly, with respect to
Prior art, can shorten the flow process of Data Migration, improve the performance of Data Migration；In addition, in Data Migration flow process, user
Disk space need not be monitored, therefore O＆M cost can also be reduced.
With reference to Fig. 1, it illustrates a kind of structure chart of the application data mover system embodiment 1, specifically may include that
Configuration read module 101, for reading the configuration file of data migration task, can include in described configuration file
There is data read information data write information；
Read module 102, for according to the data read information in described configuration file, carrying out the reading of data, and
The data of reading is converted to after intermediate form, writes data exchange buffer module；
Data exchange buffer module 103, for the data of the intermediate form of interim storage write；And
Writing module 104, for reading the data of described intermediate form from described data exchange buffer module, and foundation
Data write information in described configuration file, the data of described intermediate form is converted to after object format, write target ground
Location.
The application goes between arbitrary data storehouse, between random file system or arbitrary data storehouse and file system
Data Migration between system.
In practice, the application can apply integrated etc. in enterprise warehouse field, electronic commerce data warehouse, company information
Field.With reference to Fig. 2, it illustrates the schematic diagram of Data Migration in a kind of enterprise of the application, because Data Warehouse for Enterprises is used for propping up
The support operation of enterprise and decision-making so that in enterprise each department data, and the operation system of each department is due to building
Time is different, and policymaker is different, often employs different system databases, and such as financial system adopts db2 data base, manpower
Resource system adopts oracle database, and marketing system adopts ms sql server data base, crm (customer relation management system
System, customer relationship management) adopt http (Hyper text transfer, hypertext transfer
Protocol) the data base of interface, logistics system adopts mysql data base, etc..Therefore, at this moment can be incited somebody to action using the application
Data imports to Data Warehouse for Enterprises from enterprise operation system.In addition, the data of Data Warehouse for Enterprises is through etl, and (data carries
Take, change and load, extraction-transformation-loading) after process, define the number of results of analytical type
According in addition it is also necessary to these result datas are exported to enterprise operation system, the mstr using oracel data base
(microstrategy) reporting system or the self-built reporting system using mysql, or in visualization engine, form the figure such as chart
The shape form of expression, uses for enterprise management decision-making person.
Data Migration can act as a data migration task each time, so, the data migration task tool of the application
Body can include entering between heterogeneous database or between heterogeneous file system or between heterogeneous database and file system
The task of row Data Migration.The application is not any limitation as to specific data migration task.
With regard to configuration file
The embodiment of the present application to describe the information of each data migration task using configuration file.In actual applications can be by
One special configuration module, generates the configuration file of data migration task according to user configuring.In the embodiment of the present application, preferably
, described configuration file can be xml (extensible markup language document, extensible markup language) lattice
Formula, the naming rule of described extensible markup language document can be: source table name _ purpose table name _ source database type _ purpose number
According to storehouse type .xml.
Below by mysql- > Data Migration of hdfs (" -> " represent migratory direction) as a example, illustrating should in configuration file
The information of description data migration task.
Firstly, since data will be read from mysql data base, therefore it should be understood that the ip address of source database, port, user
The data read information such as name, password, and these information are left in the xml document of each data migration task；
Secondly as will write data into hdfs system, therefore it should also be understood that data write path, datum target form
Deng data write information, and these information are left in the xml document of each data migration task.
Therefore, in a preferred embodiment of the present application, described data read information specifically can include following information
One or more of: the ip address of source database, port numbers, library name, login username, login password, data base's table name, mistake
Filter condition；Described filtercondition can be to filter required record in where condition, or the specified row dragging required for table etc.
Deng the application is not any limitation as to specific filtercondition.
Described data write information specifically can include one or more of following information: data write path, data
Object format.
User generated data migrates the configuration file of task for convenience, can be generated basic by order line configuration wizard
Configuration file.If necessary, if also needing deep customization, can be changed again on the basis of the xml document generating.
In a preferred embodiment of the present application, divide table more situation for mysql point of storehouse, it is possible to use variable reaches
Support the purpose of multiple data migration task to same masterplate.Specifically can be illustrated by following example:
Assume to need this table of bmw_users in synchronous uic, and the data volume ratio of this table be larger, be therefore dispersed in 16 storehouses,
In 1024 tables:
uic_main_0000bmw_users_[0000-0063]
uic_main_0001bmw_users_[0064-0127]
uic_main_0002bmw_users_[0128-0191]
uic_main_0003bmw_users_[0192-0255]
uic_main_0004bmw_users_[0256-0319]
uic_main_0005bmw_users_[0320-0383]
uic_main_0006bmw_users_[0384-0447]
uic_main_0007bmw_users_[0448-0511]
uic_main_0008bmw_users_[0512-0575]
uic_main_0009bmw_users_[0576-0639]
uic_main_0010bmw_users_[0640-0703]
uic_main_0011bmw_users_[0704-0767]
uic_main_0012bmw_users_[0768-0831]
uic_main_0013bmw_users_[0832-0895]
uic_main_0014bmw_users_[0896-0959]
uic_main_0015bmw_users_[0960-1023]
For this kind of table, this preferred embodiment generally by the parameter of change, protect as variable by such as library name and/or table name etc.
Exist in a general configuration file, then operationally, by defining different parameter values thus reaching an xml document
The purpose that multilist uses.
Example in correspondence, during generating configuration file, can incite somebody to action using dtable_name as the variable of table name
, as the variable of library name, then, operationally for this variable assignments, a kind of assignment example is as follows for ddbname:
/ home/taobao/datax/bin/datax.py-p "-dtable_name=bmw_users_ [0512-
0575]-ddbname=uic_main_0008 " job.xml
Above-mentioned example located this data base's data table of uic_main_0008bmw_users_ [0512-0575] and makees
Source address for data.
With regard to intermediate form
Regardless of the source format of data, the data of reading can be converted to prespecified, writing module by the application
The intermediate form being capable of identify that, and write data exchange buffer module such that it is able to realize the universal buffering management to data, with
Just the data of this intermediate form can be converted to object format according to data write information by writing module, and writes target ground
Location.Described intermediate form makes the function opposite independent of read module and writing module so that read module and writing module
Exploitation can individually be carried out, and this can either quickly support new heterogeneous data source, can reduce development difficulty and development cost again.
In implementing, can with prespecified array formats, or in java map class form intermediate form.Wherein,
Map class data storage in the form of button/numerical value pair, and array is closely similar, indexes present in array, themselves
It is object.
The interface of map class specifically may include that
Map--- realizes map
The inner classes of map.entry--map, the button/numerical value pair in description map.
Sortedmap--- extends map, makes button keep ascending order arrangement
Except of course that outside map class form in array formats, java, described intermediate form can be with its other form, this Shen
Please specific intermediate form is not any limitation as.
With regard to data exchange buffer module 103
In a preferred embodiment of the present application, described data exchange buffer module 103 can be based on double buffering team
The core buffer of row.
System queue is to read thread to access data with write thread from same queue, is necessarily required to exclusive reference,
It is mutually synchronized in wait and wastes valuable time, make queue handling capacity impacted.And double buffering team uses two queues, will read
Write and separate, a queue is specifically used to read, and another is specifically used to write, when reading queue empty or write queue is full, by two queues
Exchange.Herein for the read-write order ensureing queue, just allow two teams when reading queue is sky and write queue is not sky
Row exchange.
In another preferred embodiment of the application, described system can also include:
Read decomposing module, for carrying out in described read module, before the reading of data, a data migration task being divided
Solution becomes multiple reading subtasks；
Described read module, appoints specifically under the reading running environment of multithread pool, carrying out the plurality of son that reads
The reading of business.
For example, certain oracle tables of data is imported to hdfs file system by user, then described reading decomposing module is permissible
This table is pressed row cutting for n interval it is assumed that n=100, then queue can be put in described 100 reading subtasks.
And for example, if the data of hdfs file system is directed into certain oracle tables of data by user, described reading is divided
Solution module can be decomposed to the file of hdfs file system, and the application is not any limitation as to specific catabolic process.
Further, under the reading running environment of multithread pool, carry out the reading of the plurality of reading subtask, main have
Have the following advantages:
1. the recycling of thread, thread completes one and reads after subtask asks, and can again be lost and returning to this thread
Pond, continues waiting for processing next reading subtask；
2. the establishment of thread, if current reading subtask is more than current process thread, new reading subtask obtains
After processing, thread pool can actively create the new reading subtask of new thread process；
3. the destruction of thread, if current reading subtask is fewer, most of thread is in idle condition and obtains not
To reading subtask, thread pool can actively close the thread that subtask is read in some long-time no process, reduces resource occupation.
In implementing, the number of threads in described multithread pool can be arranged in configured letters by user, also can have
There is default value, for example, can arrange number of threads in upper example is 40.
Due to carrying out the reading of the plurality of reading subtask using thread pool, any two reads between subtask mutually not
Impact, therefore, it is possible to realize the parallel reading reading subtask such that it is able to greatly shorten Data Migration flow process, further simultaneously
Improve Data Migration performance.
In the same manner, in the still another preferable embodiment of the application, described system can also include:
Write decomposing module, before in said write module by the data write destination address of object format, by one
Data migration task resolves into multiple write subtasks；
Said write module, specifically under the write running environment of described multithread pool, will be with the plurality of write
The data write destination address of the corresponding object format in subtask.
For the write carrying out the plurality of write subtask using thread pool, due to ablation process and above-mentioned reading process
Similar, and therefore not to repeat here, cross-referenced.
Due to carrying out the write of the plurality of write subtask using thread pool, any two reads between subtask mutually not
Impact, therefore, it is possible to realize write subtask be written in parallel to such that it is able to greatly shorten Data Migration flow process, simultaneously further
Improve Data Migration performance.
In the still another preferable embodiment of the application, described system can also include:
Monitoring counting module, for the fortune in described read module and/or data exchange buffer module and/or writing module
During row, monitor corresponding operational factor, described operational factor specifically can include reading and/or interim storage and/or write
Enter line number and/or the byte number of data.
With respect to prior art, in long Data Migration flow process, user not only needs oneself linking upstream and downstream process, right
Each process is monitored, but also disk space to be monitored；The application can be user in short Data Migration flow process automatically
Assume the operational factor of Data Migration, for example, currently have read how many byte numbers, interim storage how many byte numbers, it is written with
How many byte numbers, described present so that user is very clear to Data Migration flow process.
With reference to Fig. 3, it illustrates a kind of structure chart of the application data mover system embodiment 2, specifically may include that
Plug-in type Data Migration platform 301, may further include:
Plug-in unit loading module 311, for reading the configuration file of data migration task, and according in described configuration file
Data read information data writes information, loads corresponding reading plug-in unit, write plug-in unit data exchange buffering plug-in unit；And
Enforcement engine module 312, for providing running environment for plug-in unit；
Plug-in unit 302, may further include:
Read plug-in unit 321, for according to the data read information in described configuration file, carrying out the reading of data, and
The data of reading is converted to after intermediate form, writes data exchange cushioning insert 322；
Data exchange cushioning insert 322, reads the data of the intermediate form of plug-in unit write for interim storage；And
Write plug-in unit 323, for reading the data of described intermediate form from described data exchange cushioning insert 322, and
According to the data write information in described configuration file, the data of described intermediate form is converted to after object format, write
Destination address.
The present embodiment 2 is with the difference of embodiment 1, using the mechanical development data mover system of framework+plug-in unit, as
The framework of system, plug-in type Data Migration platform 301 is equivalent to a data transfer platform, and plug-in unit 302 is then to access difference
The data source (data base/file system) of type provides to be realized.
The present embodiment, with the access function of each data source of Informal development of plug-in unit, has the advantage that
1st, user easily can increase new plug-in unit toward plug-in type Data Migration platform 301, so that plug-in type data is moved
Move platform 301 and support new data base or file system；In order to access a new data source, prior art needs substantial amounts of
Manpower development and test, could develop a set of utility, and the plug-in unit of the application only needs less workload and exploitation
Cost；
2nd, plug-in type Data Migration platform 301 passes through to encapsulate the control logic of plug-in unit, is possible to execute data migration task
Plug-in unit combine, its cooperation relation can be set up in managed plug-in unit, and then systemic-function is realized by plug-in unit.
With regard to plug-in management
Plug-in management is the Core Feature of plug-in type Data Migration platform 301, and it is mainly used in the description letter according to plug-in unit
Breath carries out unified management to plug-in unit.The main contents of management can include the loading of plug-in unit, plug-in component operation environment provides etc..
In implementing, user can be according to the interface of plug-in type Data Migration platform 301 definition, and self-developing is inserted
Part.For example, plug-in unit can be implemented as a jar (java archive file, java archive) bag, and this jar bag is put into by user
Under plugins (plug-in unit) catalogue of plug-in type Data Migration platform 301 definition, registered in plugins.xml file simultaneously,
Can be with using this plug-in unit.
In order to the load function of plug-in unit is described, in a preferred embodiment of the present application, described plug-in type Data Migration
Platform 301 can also include:
Plug-in unit storage module, for depositing the plug-in description file of plug-in unit；
Plug-in registration module, for the loading plug-in unit information needed according to storage in described plug-in description file, to corresponding
Plug-in unit registered, and store corresponding plug-in registration information；
Now, described plug-in unit loading module 311, can specifically for by the data read information in described configuration file,
Data write information, is mated with the plug-in registration information of storage in described plug-in registration module, if the match is successful, is filled
Carry plug-in unit corresponding with described plug-in registration information.Described loading process specifically may include that according to described plug-in description file
Generate corresponding plug-in unit example, and carry out the initialization of plug-in unit according to described plug-in unit example.
For example, essential information field plugin can be included in described plug-in description file, rely on plugin information field
Required, function definition information field define and functional realiey information field implement etc., wherein, described basic letter
The information of breath field plugin storage can include plug-in unit identifier.
In implementing, described plug-in unit identifier can be represented with data source types and corresponding function, for example, inserts
Part identifier a_0, a represent oracle database, and 0 represents reading data, and and for example, plug-in unit identifier f_1, f represent idfs file
System, 0 represents write data, etc..
In practice, the initialization of described plug-in unit may further include: reads in the data in described plug-in description file
Information, other parts in corresponding plug-in unit and plug-in type Data Migration platform 301 are linked together etc..The application is to specific
Plug-in initialization and plug-in unit loading pattern are not any limitation as.
In order to the function of enforcement engine module 312 is described, in a preferred embodiment of the present application, the work(of described plug-in unit
Can also include:
Read and decompose plug-in unit, for before described reading plug-in unit 321 carries out the reading of data, by a data migration task
Resolve into multiple reading subtasks；
Now, described enforcement engine module 312, may further include:
Concurrently reading environment provides submodule, for providing the reading of multithread pool to run ring for the described plug-in unit 321 that reads
Border, by described reading plug-in unit 321 under the reading running environment of described multithread pool, carries out the reading of the plurality of reading subtask
Take.
In the same manner, in another preferred embodiment of the present application, described plug-in unit 302 can also include:
Plug-in unit is decomposed in write, before in said write plug-in unit 323 by the data write destination address of object format, by one
Individual Data Migration Task-decomposing becomes multiple write subtasks；
Now, described enforcement engine module 312, may further include:
Concurrently write environment provides submodule, and the write for providing multithread pool for said write plug-in unit 323 runs ring
Border, by said write plug-in unit 323 under the write running environment of described multithread pool, will be corresponding to the plurality of write subtask
Object format Data Concurrent write destination address.
In the still another preferable embodiment of the application, described plug-in type Data Migration platform 301 can also have monitoring
Tally function, correspondingly, it can also include:
Monitoring counting module, for inserting in described reading plug-in unit 321 and/or data exchange cushioning insert 322 and/or write
In the running of part 323, monitor corresponding operational factor, described operational factor specifically can include reading and/or depositing temporarily
The line number of storage and/or write data and/or byte number.
It should be noted that in implementing, the data of reading can also be converted to centre by reading in plug-in unit 321
The function of form is integrated in the first special plug-in unit it is also possible to will write in plug-in unit 323 according to the number in described configuration file
According to write information, the function that the data of described intermediate form is converted to object format is integrated in the second special plug-in unit；Always
It, those skilled in the art can be according to the different plug-in unit of different function distribution exploitations, the function distribution to plug-in unit for the application
It is not any limitation as with exploitation.
Furthermore it is also possible to arrange abnormal log module in described plug-in type Data Migration platform 301, different for platform
Often process and log recording.Wherein, described abnormality processing can include the definition of exception object it is also possible to include when abnormal appearance
When, exception object is encapsulated according to practical situation and produces；What described log recording can preserve each data migration task executes note
Record, for example, it is possible under the log file being saved under platform master catalogue, journal file can be configured with the xml of data migration task
Filename+". "+timestamp+" .log " names for mode.
For system embodiment 2, due to itself and system embodiment 1 basic simlarity, so fairly simple, the phase of description
In place of pass, the part referring to system embodiment 1 illustrates.
With reference to Fig. 4, the flow chart that it illustrates the application a kind of data migration method embodiment 1, specifically may include that
Step 401, the configuration file of reading data migration task, can include digital independent letter in described configuration file
Breath data write information；
Step 402, according to the data read information in described configuration file, carry out the reading of data；
Step 403, the data of reading is converted to intermediate form, and writes data exchange buffer module, wherein, described number
Can be used for the data of the intermediate form of interim storage write according to exchange buffering module；
Step 404, read the data of described intermediate form from described data exchange buffer module；
Step 405, according to the data write information in described configuration file, the data of described intermediate form is converted to mesh
Style formula, and write destination address.
In the embodiment of the present application it is preferred that described data read information can include one of following information or
Multiple: the ip address of source database, port numbers, library name, login username, login password, data base's table name, filtercondition；
Described data write information can include one or more of following information: data write path, datum target
Form.
In the embodiment of the present application it is preferred that described configuration file can be extensible markup language document, described can
The naming rule of extended markup language file can be source table name _ purpose table name _ source database type _ purpose type of database
.xml.
In a preferred embodiment of the present application, methods described can also include:
Before the reading carrying out data, a Data Migration Task-decomposing is become multiple reading subtasks；
It is now, described that according to the data read information in described configuration file, the step carrying out the reading of data can be,
Under the reading running environment of multithread pool, carry out the reading of the plurality of reading subtask
In another preferred embodiment of the present application, methods described can also include:
Before said write module is by the data write destination address of object format, a Data Migration Task-decomposing is become
Multiple write subtasks；
Correspondingly, the step of said write destination address is, under the write running environment of described multithread pool, will be with institute
State the data write destination address of the corresponding object format in multiple write subtasks.
In the embodiment of the present application it is preferred that described intermediate form can include array formats, or map in java
Class form.
In the embodiment of the present application it is preferred that described data exchange buffer module is the internal memory based on double buffering queue
Relief area.
In the still another preferable embodiment of the application, methods described can also include:
In the running of described read module and/or data exchange buffer module and/or writing module, monitoring is corresponding
Operational factor, described operational factor can include reading and/or interim storage and/or the write line number of data and/or byte
Number.
With reference to Fig. 5, the flow chart that it illustrates the application a kind of data migration method embodiment 2, specifically may include that
Step 501, according to the data read information in described configuration file, load the corresponding plug-in unit data that reads and exchange
Cushioning insert, and provide corresponding running environment to described reading plug-in unit, by described reading plug-in unit according in described configuration file
Data read information, carry out the reading of data, and after the data of reading is converted to intermediate form write data exchange delay
Rush plug-in unit, read the data of the intermediate form of plug-in unit write by described data exchange cushioning insert interim storage；
Step 502, according to the data write information in described configuration file, load and corresponding write plug-in unit, and to described
Write plug-in unit provides corresponding running environment, reads described centre by said write plug-in unit from described data exchange cushioning insert
The data of form, and according to the data write information in described configuration file, the data of described intermediate form is being converted to target
After form, write destination address.
The present embodiment is with the difference of embodiment 1, using plug-in unit as access different types of data source (data base/literary composition
Part system) provide and realize, namely the access function with each data source of Informal development of plug-in unit；Therefore, the present embodiment passes through envelope
Plug the control logic of part, the plug-in unit being possible to execute data migration task is combined, and can build in managed plug-in unit
Found its cooperation relation, and then systemic-function is realized by plug-in unit.
In a preferred embodiment of the present application, methods described can also include:
Deposit the plug-in description file of plug-in unit；
According to the loading plug-in unit information needed of storage in described plug-in description file, corresponding plug-in unit is registered, and
Store corresponding plug-in registration information；
Now, the described step loading corresponding reading plug-in unit data exchange buffering plug-in unit, may further include:
Plug-in registration information by storage in the data read information in described configuration file, with described plug-in registration module
Mated, if the match is successful, loaded plug-in unit corresponding with described plug-in registration information, described loading process can wrap
Include: generate corresponding plug-in unit example according to described plug-in description file, and carry out the initialization of plug-in unit according to described plug-in unit example；
The described step loading corresponding write plug-in unit, may further include:
Plug-in registration information by storage in the data write information in described configuration file, with described plug-in registration module
Mated, if the match is successful, loaded plug-in unit corresponding with described plug-in registration information, described loading process can wrap
Include: generate corresponding plug-in unit example according to described plug-in description file, and carry out the initialization of plug-in unit according to described plug-in unit example.
In order to illustrate how to provide corresponding running environment to described reading plug-in unit, in a preferred embodiment of the present application
In, described plug-in unit can also include:
Read and decompose plug-in unit, for, before described reading plug-in unit carries out the reading of data, a data migration task being divided
Solution becomes multiple reading subtasks；
Now, the described step providing corresponding running environment to described reading plug-in unit, may include that
There is provided the reading running environment of multithread pool for the described plug-in unit that reads, by described reading plug-in unit in described multithread pool
Reading running environment under, carry out the reading of the plurality of reading subtask.
In the same manner, in another preferred embodiment of the present application, described plug-in unit can also include:
Plug-in unit is decomposed in write, before in said write plug-in unit by the data write destination address of object format, by one
Data migration task resolves into multiple write subtasks；
Now, the described step providing corresponding running environment to said write plug-in unit, may further include:
The write running environment of multithread pool is provided for said write plug-in unit, described multithread pool is inserted in by said write
Under write running environment, the Data Concurrent of object format corresponding with the plurality of write subtask is write destination address.
For embodiment of the method 2, due to itself and embodiment of the method 1 basic simlarity, so fairly simple, the phase of description
In place of pass, the part referring to embodiment of the method 1 illustrates.
Each embodiment in this specification is all described by the way of going forward one by one, what each embodiment stressed be with
The difference of other embodiment, between each embodiment identical similar partly mutually referring to.
Above to a kind of data mover system provided herein and method, it is described in detail, used herein
Specific case is set forth to the principle of the application and embodiment, and the explanation of above example is only intended to help understand
The present processes and its core concept；Simultaneously for one of ordinary skill in the art, according to the thought of the application, in tool
All will change on body embodiment and range of application, in sum, this specification content should not be construed as to the application
Restriction.
Claims (10)
1. a kind of data mover system is it is characterised in that include:
Configuration read module, for reading the configuration file of data migration task, includes digital independent in described configuration file
Information data writes information；
Read module, for according to the data read information in described configuration file, carrying out the reading of data, and will read
After data is converted to intermediate form, write data exchange buffer module；
Data exchange buffer module, for the data of the intermediate form of interim storage write；And
Writing module, for reading the data of described intermediate form, and according to described from described data exchange buffer module
Data write information in configuration file is converted to the data of described intermediate form after object format, writes destination address；
Described system also includes:
Preserving module, for by the running parameter included by described data read information and/or described data write information, as
Variable save is in a general configuration file.
2. a kind of data migration method is it is characterised in that include:
Read the configuration file of data migration task, in described configuration file, include data read information data write letter
Breath；
According to the data read information in described configuration file, carry out the reading of data；
The data of reading is converted to intermediate form, and writes data exchange buffer module, wherein, described data exchange buffers mould
Block is used for the data of the intermediate form of interim storage write；
The data of described intermediate form is read from described data exchange buffer module；
According to the data write information in described configuration file, the data of described intermediate form is converted to object format, and writes
Destination address；
Methods described also includes:
By the running parameter included by described data read information and/or described data write information, as variable save one
In individual general configuration file.
3. method as claimed in claim 2 is it is characterised in that also include:
According to the data read information in described configuration file, load corresponding reading plug-in unit data exchange buffering plug-in unit, and
There is provided corresponding running environment to described reading plug-in unit, believed according to the digital independent in described configuration file by the described plug-in unit that reads
Breath, carries out the reading of data, and writes data exchange cushioning insert after the data of reading is converted to intermediate form, by described
Data exchange cushioning insert interim storage reads the data of the intermediate form of plug-in unit write；
According to the data write information in described configuration file, load corresponding write plug-in unit, and provide to said write plug-in unit
Corresponding running environment, is read the data of described intermediate form from described data exchange cushioning insert by said write plug-in unit,
And after the data of described intermediate form being converted to object format according to the data write information in described configuration file, write
Destination address.
4. method as claimed in claim 2 is it is characterised in that also include:
Before the reading carrying out data, a Data Migration Task-decomposing is become multiple reading subtasks；
Described according to the data read information in described configuration file, the step carrying out the reading of data is, in multithread pool
Read under running environment, carry out the reading of the plurality of reading subtask.
5. method as claimed in claim 2 is it is characterised in that also include:
Before the data write destination address by object format, a Data Migration Task-decomposing is become multiple write subtasks；
The step of said write destination address is, under the write running environment of multithread pool, will appoint with the plurality of write
Be engaged in corresponding object format data write destination address.
6. method as claimed in claim 2 is it is characterised in that described data exchange buffer module is based on double buffering queue
Core buffer.
7. method as claimed in claim 2 is it is characterised in that also include:
In the running of read module and/or data exchange buffer module and/or writing module, monitoring is corresponding to run ginseng
Number, described operational factor includes reading and/or interim storage and/or the line number and/or the byte number that write data；
Wherein, described read module, for according to the data read information in described configuration file, carrying out the reading of data, and
After the data of reading is converted to intermediate form, write data exchange buffer module；
Described data exchange buffer module, for the data of the intermediate form of interim storage write；
Said write module, for reading the data of described intermediate form, and in foundation from described data exchange buffer module
Data write information in described configuration file is converted to the data of described intermediate form after object format, write target ground
Location.
8. method as claimed in claim 2 or claim 3 is it is characterised in that described data read information includes in following information
Plant or multiple: the ip address of source database, port numbers, library name, login username, login password, data base's table name, filtering rod
Part；
Described data write information includes one or more of following information: data write path, datum target form.
9. method as claimed in claim 2 or claim 3 it is characterised in that described configuration file be extensible markup language document, institute
The naming rule stating extensible markup language document is source table name _ purpose table name _ source database type _ purpose type of database
.xml.
10. method as claimed in claim 3 is it is characterised in that also include:
Deposit the plug-in description file of plug-in unit；
According to the loading plug-in unit information needed of storage in described plug-in description file, corresponding plug-in unit is registered, and stores
Corresponding plug-in registration information；
The described step loading corresponding reading plug-in unit data exchange buffering plug-in unit, comprising:
By the data read information in described configuration file, mated with the plug-in registration information of storage, if the match is successful,
Then load plug-in unit corresponding with described plug-in registration information, described loading process includes: generates according to described plug-in description file
Corresponding plug-in unit example, and carry out the initialization of plug-in unit according to described plug-in unit example；
The described step loading corresponding write plug-in unit, comprising:
By the data write information in described configuration file, mated with the plug-in registration information of storage, if the match is successful,
Then load plug-in unit corresponding with described plug-in registration information, described loading process includes: generates according to described plug-in description file
Corresponding plug-in unit example, and carry out the initialization of plug-in unit according to described plug-in unit example.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110278784.0A CN102999537B (en) | 2011-09-19 | 2011-09-19 | System and method for data migration |
| HK13105933.4A HK1178291A1 (en) | 2011-09-19 | 2013-05-21 | System and method for data migration |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110278784.0A CN102999537B (en) | 2011-09-19 | 2011-09-19 | System and method for data migration |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102999537A CN102999537A (en) | 2013-03-27 |
| CN102999537B true CN102999537B (en) | 2017-01-18 |
Family
ID=47928110
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110278784.0A Active CN102999537B (en) | 2011-09-19 | 2011-09-19 | System and method for data migration |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN102999537B (en) |
| HK (1) | HK1178291A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108664496A (en) * | 2017-03-29 | 2018-10-16 | 腾讯科技（深圳）有限公司 | Data migration method and device |
Families Citing this family (71)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103309945A (en) * | 2013-05-15 | 2013-09-18 | 上海证券交易所 | Device for importing data to database |
| WO2014205847A1 (en) * | 2013-06-29 | 2014-12-31 | 华为技术有限公司 | Zoning balance subtask delivering method, apparatus and system |
| CN103473362B (en) * | 2013-09-26 | 2016-06-29 | 浙江省电力锅炉压力容器检验所有限公司 | The method of Data Migration, system and receiving port module |
| CN104598453B (en) * | 2013-10-31 | 2018-02-27 | 中国银联股份有限公司 | Data migration method based on data buffering |
| CN104699723B (en) * | 2013-12-10 | 2018-10-19 | 北京神州泰岳软件股份有限公司 | Data synchronous system and method between data exchange adapter, heterogeneous system |
| CN103944964A (en) * | 2014-03-27 | 2014-07-23 | 上海云数信息科技有限公司 | Distributed system and method carrying out expansion step by step through same |
| CN105320673A (en) * | 2014-06-30 | 2016-02-10 | 苏宁云商集团股份有限公司 | Method and device for transmitting data among multiple data sources |
| CN105335412B (en) * | 2014-07-31 | 2019-06-11 | 阿里巴巴集团控股有限公司 | For data conversion, the method and apparatus of Data Migration |
| CN104360905B (en) * | 2014-10-29 | 2019-01-29 | 中国建设银行股份有限公司 | A kind of self-adaptation control method and device applied to IT system |
| CN104462269A (en) * | 2014-11-24 | 2015-03-25 | 中国联合网络通信集团有限公司 | Isomerous database data exchange method and system |
| CN104536994A (en) * | 2014-12-11 | 2015-04-22 | 北京京东尚科信息技术有限公司 | Universal data migration method and device |
| CN104462562B (en) * | 2014-12-29 | 2018-05-18 | 浪潮软件集团有限公司 | Data migration system and method based on data warehouse automation |
| CN104539431B (en) * | 2014-12-31 | 2018-08-17 | 上海格尔软件股份有限公司 | A kind of method of online user's system switching user's smooth migration |
| US10757186B2 (en) | 2015-03-02 | 2020-08-25 | Microsoft Technology Licensing, Llc | Uploading user and system data from a source location to a destination location |
| CN106156165A (en) | 2015-04-16 | 2016-11-23 | 阿里巴巴集团控股有限公司 | Method of data synchronization between heterogeneous data source and device |
| CN106209933A (en) * | 2015-04-30 | 2016-12-07 | 中国移动通信集团四川有限公司 | source host, migration framework and data migration method |
| CN106469152A (en) * | 2015-08-14 | 2017-03-01 | 阿里巴巴集团控股有限公司 | A kind of document handling method based on ETL and system |
| CN106487841A (en) * | 2015-08-27 | 2017-03-08 | 阿里巴巴集团控股有限公司 | A kind of data migration method and equipment |
| CN105205117B (en) * | 2015-09-09 | 2019-03-05 | 郑州悉知信息科技股份有限公司 | A kind of tables of data moving method and device |
| CN106557486A (en) | 2015-09-25 | 2017-04-05 | 阿里巴巴集团控股有限公司 | A kind of storage method and device of data |
| CN105335477B (en) * | 2015-10-10 | 2019-09-20 | 北京京东尚科信息技术有限公司 | A kind of operating method and device of database |
| CN105373599B (en) * | 2015-10-28 | 2017-05-31 | 北京汇商融通信息技术有限公司 | Data mover system based on various data storing platforms |
| CN105279280A (en) * | 2015-11-16 | 2016-01-27 | 天津南大通用数据技术股份有限公司 | Method and tool for quickly migrating oracle data to MPP database |
| CN105653630B (en) * | 2015-12-25 | 2019-12-24 | 北京奇虎科技有限公司 | Data migration method and device for distributed database |
| CN106933859B (en) * | 2015-12-30 | 2020-10-20 | 中国移动通信集团公司 | Medical data migration method and device |
| CN105718507A (en) * | 2016-01-06 | 2016-06-29 | 杭州数梦工场科技有限公司 | Data migration method and device |
| CN107122361B (en) * | 2016-02-24 | 2021-07-06 | 阿里巴巴集团控股有限公司 | Data migration system and method |
| CN105871815A (en) * | 2016-03-22 | 2016-08-17 | 乐视网信息技术（北京）股份有限公司 | Data transmitting method and device |
| CN105843955A (en) * | 2016-04-13 | 2016-08-10 | 曙光信息产业(北京)有限公司 | Data migration system |
| CN105930389A (en) * | 2016-04-14 | 2016-09-07 | 北京京东尚科信息技术有限公司 | Method and system for transferring data |
| CN106055670A (en) * | 2016-06-06 | 2016-10-26 | 中国工商银行股份有限公司 | Inter-system data migration method and device |
| CN107784026B (en) * | 2016-08-31 | 2021-08-20 | 杭州海康威视数字技术股份有限公司 | ETL data processing method and device |
| CN106446111B (en) * | 2016-09-14 | 2020-01-14 | Oppo广东移动通信有限公司 | Data migration method and terminal |
| CN107992498B (en) * | 2016-10-27 | 2021-10-22 | 腾讯科技（深圳）有限公司 | Method and system for importing data into data warehouse |
| CN106776833A (en) * | 2016-11-28 | 2017-05-31 | 中通服公众信息产业股份有限公司 | Data of information system exchange method and system of the client to client |
| CN106548037A (en) * | 2016-12-02 | 2017-03-29 | 深圳市前海安测信息技术有限公司 | Electronic health record data mover system and method based on mobile terminal |
| CN106649788B (en) * | 2016-12-28 | 2020-12-08 | 深圳启润德科技有限公司 | Database data transmission method and device |
| CN106874389B (en) * | 2017-01-11 | 2023-04-07 | 腾讯科技（深圳）有限公司 | Data migration method and device |
| CN110661851A (en) * | 2017-03-02 | 2020-01-07 | 杭州数梦工场科技有限公司 | Data exchange method and device |
| CN107341063A (en) * | 2017-07-13 | 2017-11-10 | 郑州云海信息技术有限公司 | A kind of data transmission method and system |
| CN107506451B (en) * | 2017-08-28 | 2020-11-03 | 泰康保险集团股份有限公司 | Abnormal information monitoring method and device for data interaction |
| CN108121757A (en) * | 2017-11-10 | 2018-06-05 | 广州优视网络科技有限公司 | A kind of method of data synchronization, device, system, computing device and storage medium |
| CN108228755A (en) * | 2017-12-21 | 2018-06-29 | 江苏瑞中数据股份有限公司 | The data of MySQL database based on daily record analytic technique to Hadoop platform synchronize clone method |
| CN107977462A (en) * | 2017-12-21 | 2018-05-01 | 泰康保险集团股份有限公司 | Data migration method and device |
| CN110109891B (en) * | 2018-01-18 | 2023-04-21 | 伊姆西Ip控股有限责任公司 | Method, apparatus and storage medium for data migration |
| CN108279972A (en) * | 2018-01-26 | 2018-07-13 | 郑州云海信息技术有限公司 | A kind of LUA virtual machines task executing method and relevant apparatus |
| CN109189756A (en) * | 2018-06-29 | 2019-01-11 | 平安科技（深圳）有限公司 | Electronic device, the method for Data Migration and storage medium |
| CN109165260A (en) * | 2018-09-25 | 2019-01-08 | 安徽信息工程学院 | Method of data transfer based on ORACLE data basd link |
| CN109657167B (en) * | 2018-11-29 | 2023-11-21 | 彩讯科技股份有限公司 | Data acquisition method, device, server and storage medium |
| CN109582722A (en) * | 2018-11-30 | 2019-04-05 | 珠海市新德汇信息技术有限公司 | Public security resource data service system |
| CN109495581A (en) * | 2018-12-13 | 2019-03-19 | 爱普（福建）科技有限公司 | A kind of communication means and device of polyisocyanate structure control station |
| CN109802884B (en) * | 2018-12-29 | 2021-09-07 | 论客科技（广州）有限公司 | Campus mail migration method and device |
| CN110825453B (en) * | 2019-10-25 | 2023-12-05 | 中盈优创资讯科技有限公司 | Data processing method and device based on big data platform |
| CN110879801A (en) * | 2019-11-21 | 2020-03-13 | 中国银行股份有限公司 | Data reading method and device |
| CN110895490A (en) * | 2019-11-29 | 2020-03-20 | 深圳乐信软件技术有限公司 | Data batch processing system, method, equipment and storage medium |
| CN111078384A (en) * | 2019-12-26 | 2020-04-28 | 深圳前海环融联易信息科技服务有限公司 | Method and device for migrating core data, computer equipment and storage medium |
| CN111414409B (en) * | 2020-03-17 | 2024-03-15 | 网易（杭州）网络有限公司 | Method and device for exchanging data between game engines, storage medium and electronic equipment |
| CN111522792A (en) * | 2020-04-20 | 2020-08-11 | 中国银行股份有限公司 | File migration method and device |
| CN111737235B (en) * | 2020-08-12 | 2020-11-13 | 国网浙江省电力有限公司营销服务中心 | Heterogeneous data migration method for power industry |
| CN112035459A (en) * | 2020-09-01 | 2020-12-04 | 中国银行股份有限公司 | Data format conversion method and device |
| CN112822494A (en) * | 2020-12-30 | 2021-05-18 | 稿定（厦门）科技有限公司 | Double-buffer coding system and control method thereof |
| CN112699098A (en) * | 2020-12-31 | 2021-04-23 | 中国农业银行股份有限公司 | Index data migration method, device and equipment |
| CN112800026B (en) * | 2021-01-18 | 2024-02-06 | 中国银联股份有限公司 | Data transfer node, method, system and computer readable storage medium |
| CN113312147B (en) * | 2021-06-11 | 2022-12-30 | 中国科学技术大学 | Method and system for migrating object storage across cluster mass data |
| CN113254522A (en) * | 2021-06-18 | 2021-08-13 | 深圳追一科技有限公司 | Database access method, device, equipment and readable storage medium |
| CN113434598B (en) * | 2021-06-28 | 2024-03-22 | 青岛海尔科技有限公司 | Method, device and electronic device for realizing data double writing |
| CN113626407A (en) * | 2021-07-30 | 2021-11-09 | 浪潮云信息技术股份公司 | Grammar conversion migration method |
| CN113656474A (en) * | 2021-08-05 | 2021-11-16 | 京东科技控股股份有限公司 | Service data access method and device, electronic equipment and storage medium |
| CN113609226B (en) * | 2021-08-09 | 2024-05-14 | 深圳平安智慧医健科技有限公司 | Data export method and device, computer equipment and storage medium |
| CN115525631B (en) * | 2022-10-31 | 2023-09-05 | 华润数字科技有限公司 | Database data migration method, device, equipment and storage medium |
| CN117407362B (en) * | 2023-12-15 | 2024-04-16 | 中电云计算技术有限公司 | Method and device for file migration among heterogeneous file systems |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090240704A1 (en) * | 2002-03-18 | 2009-09-24 | International Business Machines Corporation | Method, System, and Program Product for Migrating Data from One Data Base Management System to Another Data Base Management System |
| CN102004745A (en) * | 2009-09-02 | 2011-04-06 | 中国银联股份有限公司 | Data transfer system and method |
| CN101551808B (en) * | 2009-05-13 | 2011-04-13 | 山东中创软件商用中间件股份有限公司 | Technology for supporting multi-process embedded tree-based databases |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1904885A (en) * | 2005-07-29 | 2007-01-31 | 北京航空航天大学 | Database adapter |
| CN102004743B (en) * | 2009-09-02 | 2013-08-14 | 中国银联股份有限公司 | System and method for copying data among heterogeneous databases |
| CN102142027A (en) * | 2011-02-23 | 2011-08-03 | 南京智尚丰软件有限公司 | Adaptive method for data integration |
-
2011
- 2011-09-19 CN CN201110278784.0A patent/CN102999537B/en active Active
-
2013
- 2013-05-21 HK HK13105933.4A patent/HK1178291A1/en unknown
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090240704A1 (en) * | 2002-03-18 | 2009-09-24 | International Business Machines Corporation | Method, System, and Program Product for Migrating Data from One Data Base Management System to Another Data Base Management System |
| CN101551808B (en) * | 2009-05-13 | 2011-04-13 | 山东中创软件商用中间件股份有限公司 | Technology for supporting multi-process embedded tree-based databases |
| CN102004745A (en) * | 2009-09-02 | 2011-04-06 | 中国银联股份有限公司 | Data transfer system and method |
Non-Patent Citations (2)
| Title |
|---|
| ASP.NET在异构数据源转换中的应用;陈静，等;《重庆大学学报（自然科学版）》;20060228;第29卷(第2期);第77页 * |
| 基于XML的异构数据库数据转换工具的设计与实现;杜邵敏;《中国优秀硕士学位论文全文数据库信息科技辑》;20100615(第6期);第29-40、48页 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108664496A (en) * | 2017-03-29 | 2018-10-16 | 腾讯科技（深圳）有限公司 | Data migration method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102999537A (en) | 2013-03-27 |
| HK1178291A1 (en) | 2013-09-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102999537B (en) | System and method for data migration | |
| CN106611046B (en) | Spatial data storage processing middleware system based on big data technology | |
| Butts | network: a Package for Managing Relational Data in R | |
| Yuan et al. | A data placement strategy in scientific cloud workflows | |
| CN103246749B (en) | The matrix database system and its querying method that Based on Distributed calculates | |
| US20090012983A1 (en) | System and method for federated member-based data integration and reporting | |
| US20110087708A1 (en) | Business object based operational reporting and analysis | |
| CN107103064B (en) | Data statistical method and device | |
| CN107766402A (en) | A kind of building dictionary cloud source of houses big data platform | |
| CN101901265B (en) | Objectification management system of virtual test data | |
| KR20060045622A (en) | Extraction, transformation and loading designer module of a computerized financial system | |
| CN111213137A (en) | Alignment of source infrastructure data with BIS conceptual patterns | |
| CN102377824A (en) | Cloud computing-based spatial information service system | |
| Sivaraman et al. | High performance and fault tolerant distributed file system for big data storage and processing using hadoop | |
| US10521442B1 (en) | Hierarchical value-based governance architecture for enterprise data assets | |
| CN103092906A (en) | Multi-client generic persistence for extension nodes | |
| CN110297869B (en) | AI data warehouse platform and operation method | |
| CN103744647A (en) | Java workflow development system and method based on workflow GPD | |
| WO2013175422A1 (en) | Methodology supported business intelligence (bi) software and system | |
| CN108241724A (en) | A kind of metadata management method and device | |
| CN112163017B (en) | Knowledge mining system and method | |
| CN103473332A (en) | Data archive repository with virtual test architecture | |
| Kafando et al. | Spatial data lake for smart cities: From design to implementation | |
| CN105550351B (en) | The extemporaneous inquiry system of passenger's run-length data and method | |
| US20140143248A1 (en) | Integration to central analytics systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 1178291 Country of ref document: HK |
|
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: GR Ref document number: 1178291 Country of ref document: HK |