CN106909595B - Data migration method and device - Google Patents
Data migration method and device Download PDFInfo
- Publication number
- CN106909595B CN106909595B CN201610445610.1A CN201610445610A CN106909595B CN 106909595 B CN106909595 B CN 106909595B CN 201610445610 A CN201610445610 A CN 201610445610A CN 106909595 B CN106909595 B CN 106909595B
- Authority
- CN
- China
- Prior art keywords
- file
- migrated
- database
- data record
- identification information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to the technical field of databases, in particular to a data migration method and device, which are used for solving the problem that the storage and access of binary files influence the data transmission speed and stability of a database in the prior art. The embodiment of the application provides a data migration method, which comprises the following steps: determining a first file to be migrated based on a data record to be migrated in a first database; storing the first file to be migrated in a set file system, and storing the identification information of the first file to be migrated in a second database; the identification information of the first file to be migrated is information for identifying a storage location of the first file to be migrated in the file system.
Description
Technical Field
The present application relates to the field of database technologies, and in particular, to a data migration method and apparatus.
Background
Computer files are basically divided into two categories: binary files and computer programs such as ASCII (also called plain text files), graphic files, and word processing programs belong to the binary files. These files contain special formats and computer code. ASCII is a simple text file that can be read with any word processing program.
In some cases, for example, when a service is split, a current database cannot meet the storage requirement of a binary file (the binary file generally occupies a large storage space), etc., we may need to migrate data in the current database (hereinafter referred to as an old database) to another database (hereinafter referred to as a new database), and migrate an online service from the old database to the new database after completing the data migration. In the process of data migration, the old library is still used for providing data service for the online service.
During the data migration process, new data needs to be stored. In order not to affect the normal operation of the online service, a synchronous double-write mechanism is generally adopted to perform the migration of new data. Specifically, new data is written into the queue to be processed of the old bank first, then the data in the queue to be processed is synchronously written into the old bank and the new bank, and the data in the old bank before the time point when the synchronous double-write is started is migrated to the new bank.
At present, the storage and access to the binary file become a bottleneck affecting the data transmission speed and stability of the database. When data migration is performed, if the existing method is still used to perform migration of the binary file, the data migration efficiency and the data access performance after migration will be seriously affected.
Disclosure of Invention
The embodiment of the application provides a data migration method and device, which are used for solving the problem that the storage and access of binary files influence the data transmission speed and stability of a database in the prior art.
An embodiment of the present application provides a data migration method, including:
determining a first file to be migrated based on a data record to be migrated in a first database;
storing the first file to be migrated in a set file system, and storing the identification information of the first file to be migrated in a second database;
the identification information of the first file to be migrated is information for identifying a storage location of the first file to be migrated in the file system.
Optionally, the file system is a distributed file system.
Optionally, after storing the first file to be migrated in a set file system and storing the identification information of the first file to be migrated in a second database, the method further includes:
when an access request of a client for the first file to be migrated is received, the first file to be migrated stored in the file system is acquired based on the identification information of the first file to be migrated in the second database, and the acquired first file to be migrated is returned to the client.
Optionally, determining the first file to be migrated based on the data record to be migrated in the first database includes:
extracting a binary file from the data record to be migrated;
and determining at least one of the extracted binary file and a text file converted based on the binary file as the first file to be migrated.
Optionally, if the first file to be migrated includes the binary file, after storing the binary file in a set file system and storing the identification information of the binary file in a second database, the method further includes:
when a downloading request of a client for the binary file is received, acquiring the binary file from the file system based on the identification information of the binary file in the second database, and caching the acquired binary file in a Content Delivery Network (CDN);
and returning the access address of the binary file in the CDN to the client.
Optionally, if the data record to be migrated is a new data record located in the queue to be stored of the first database, the method further includes:
and storing the text file converted based on the binary file in the first database.
Optionally, the method further comprises:
extracting a second file to be migrated from the data record to be migrated, wherein the second file to be migrated is a non-binary file;
if the data record to be migrated is a new data record in the queue to be stored of the first database, storing the second file to be migrated in the first database and the second database;
and if the data record to be migrated is a historical data record in the stored list of the first database, storing the second file to be migrated in the second database.
Optionally, the method further comprises:
writing the identification information of each historical data record in the stored list of the first database into a preset data table; the identification information of each historical data record is used for identifying the storage position of the historical data record in the first database;
and respectively extracting each unprocessed historical data record from the first database as the data record to be migrated based on the identification information of each historical data record in the preset data table, and recording the processing state of the historical data record in the preset data table.
An embodiment of the present application provides a data migration apparatus, including:
the determining module is used for determining a first file to be migrated based on the data record to be migrated in the first database;
the storage module is used for storing the first file to be migrated in a set file system and storing the identification information of the first file to be migrated in a second database;
the identification information of the first file to be migrated is information for identifying a storage location of the first file to be migrated in the file system.
When the library migration is needed, the first file to be migrated is determined from the data records to be migrated in the first database, the first file to be migrated is stored in the set file system, and the identification information of the first file to be migrated is stored in the second database. In addition, the file system can be a distributed file system, and the distributed file system can provide high-reliability and high-concurrency storage access for the outside, so that the data access performance after data migration can be improved. Therefore, the storage and access capacity of the first file to be migrated, such as a binary file, can be improved.
In addition, in a preferred embodiment of the present application, format conversion is performed when the binary file is stored, that is, text content therein is extracted and stored, and after the conversion is performed, after a page access request of a client for the binary file is received, page rendering feedback can be quickly performed based on the text content of the binary file.
Drawings
FIG. 1 is a schematic diagram illustrating data migration according to an embodiment of the present application;
FIG. 2 is a flowchart of a data migration method according to an embodiment of the present application;
fig. 3 is a flowchart of a data migration method according to a second embodiment of the present application;
fig. 4 is a flowchart of a data migration method according to a third embodiment of the present application;
fig. 5 is a schematic structural diagram of a data migration apparatus according to an embodiment of the present application.
Detailed Description
Fig. 1 is a schematic diagram illustrating data migration according to an embodiment of the present application. In the embodiment of the application, for new data received after the determination of the migration is performed (after the new data is received, the new data is written into a queue to be processed of an old database at first), synchronous double writing (that is, writing into the new database and the old database respectively) is performed after different processes are performed on binary files and non-binary files therein respectively. When a binary file is written into a new database, a first file to be migrated (including the binary file and/or a text file converted based on the binary file) is stored in the distributed file system, and identification information identifying the storage location of the first file to be migrated is stored in the new database. The historical data stored in the old database is also divided into binary files and non-binary files, and the binary files and the non-binary files are respectively subjected to different processing and then stored in the new database.
The embodiments of the present application will be described in further detail with reference to the drawings attached hereto.
Example one
As shown in fig. 2, a flowchart of a data migration method provided in an embodiment of the present application includes the following steps:
s201: and determining a first file to be migrated based on the data record to be migrated in the first database.
Here, a plurality of data records to be migrated may exist in the first database, and for each of the data records to be migrated, a first file to be migrated is determined.
In the embodiment of the present application, it is considered that the binary file occupies a larger storage space, and the non-binary file in the data record to be migrated, that is, the text file, occupies a smaller storage space, so that the processing of step S202 may be performed only on the binary file therein.
Specifically, a binary file may be extracted from the data record to be migrated; and determining at least one of the extracted binary file and the text file converted based on the binary file as the first file to be migrated.
S202: storing the first file to be migrated in a set file system, and storing the identification information of the first file to be migrated in a second database; the identification information of the first file to be migrated is information for identifying a storage location of the first file to be migrated in the file system.
The binary file in the embodiment of the application comprises a graphic file, a word processing program and the like, and is a non-text file. Because the binary file occupies a large storage space, the binary file can be extracted from the data record to be migrated, only the corresponding identification information is stored in the database, and the binary file is stored in the file system.
Preferably, the file system may refer to a distributed file system. A Distributed File System (Distributed File System) is a File System network composed of a plurality of nodes. The distributed file system has the characteristics of high performance, high reliability and strong expandability, can support access with high concurrency, large storage capacity and large throughput by adding the server, and can effectively avoid single-point faults.
S203: when an access request of a client for the first file to be migrated is received, the first file to be migrated stored in the file system is acquired based on the identification information of the first file to be migrated in the second database, and the acquired first file to be migrated is returned to the client.
In specific implementation, if the client needs to access the first file to be migrated, the corresponding first file to be migrated is acquired from the distributed file system based on the identification information of the first file to be migrated, and is returned to the client.
Example two
The following takes the first file to be migrated as a binary file, and the second file to be migrated as a non-binary file, for example, as further described.
As shown in fig. 3, a flowchart of a data migration method provided in the second embodiment of the present application includes the following steps:
s301: the binary file and the non-binary file in the data record to be migrated in the first database are distinguished, and the following S302 is performed for the binary file, and the following S303 is performed for the non-binary file.
Here, the data record to be migrated may be new data (newly received, not yet put in storage) written in the queue to be processed of the first database (old database) after it is determined that the migration is necessary, or may be history data already stored in the first database.
For the historical data, the identification information (Identity, ID) of each historical data record in the stored list of the first database may be written into a preset data table (the data table is a created temporary table and may be deleted after the migration of the historical data is completed); the identification information of each historical data record is used for identifying the storage position of the historical data record in the first database (the identification information may be a storage address or an identification recorded in the first database and corresponding to the historical data record); and respectively extracting each unprocessed historical data record from the first database as the data record to be migrated based on the identification information of each historical data record in the preset data table, and recording the processing state of the historical data record in the preset data table. Here, a timing task may be set up, and based on the identification information of each history data record in the preset data table, each unprocessed history data record in the first database (which is not written into the second database) is periodically taken out for processing (i.e., written into the second database). Through the record processing state of the preset data table, the record of the historical data can be processed without repetition, and idempotent is supported.
For each new data record or history data record, the binary file (such as a PDF-formatted binary file) and the non-binary file are distinguished and processed separately.
S302: converting the binary file to obtain a text file; storing the binary file and the converted text file in a distributed file system, and storing identification information for identifying the storage position of the binary file in the distributed file system and identification information for identifying the storage position of the text file in the distributed file system in a second database; and if the data to be migrated is recorded as new data, storing the converted text file in the first database.
In S302, the binary file and the text file converted based on the binary file are stored in the distributed file system, and identification information (the identification information may be a storage address or an identification recorded in the distributed file system and corresponding to the binary file) identifying a storage location of the binary file in the distributed file system and identification information identifying the storage location of the text file are stored in the second database.
In this way, after the migration is completed (the database providing data services for the user is migrated from the first database to the second database), when a download request for the binary file by the client is received, the corresponding binary file can be obtained in the distributed file system and fed back to the client based on the identification information of the binary file stored in the second database. In addition, in order to further increase the download speed of the binary file, after the corresponding binary file is obtained from the distributed file system based on the identification information of the binary file, the obtained binary file may be cached in a Content Delivery Network (CDN), and an access address of the binary file in the CDN is returned to the client.
Here, for the binary file downloading, after the CDN cache is adopted, the access address of the binary file in the CDN is returned to the client. The CDN system can redirect the access request of the client to the service node closest to the client in real time according to the network flow, the connection and load conditions of each node, the distance to the client, the response time and other comprehensive information. Therefore, the client can acquire the access content nearby, the network congestion condition is avoided, and the response speed of the user for accessing the website is improved.
In addition, after the migration is completed, if a page access request of the client for the binary file is received, the corresponding text file can be obtained in the distributed file system based on the identification information of the text file, and the access page is rendered and fed back based on the text file.
In addition, if the data record to be migrated is new data, because the binary file occupies a large storage space, the embodiment of the present application does not store the binary file in the first database, that is, discards the binary file in the first database, but stores the text content extracted from the binary file in the first database in order not to affect the normal operation of the online service (that is, performs synchronous double-write). In this way, before the migration is completed, after a page access request of the client for the binary file is received, rendering feedback can be performed on an access page based on the text file in the first database.
In a specific implementation, when the binary file is a file in PDF format, a Pdfbox tool may be used to extract text content from the PDF file.
According to the method and the device, format conversion is performed when the binary file is stored, namely text content in the binary file is extracted to be stored, after the conversion is performed, the content of the binary file can be rendered and fed back quickly after an access request of a client for the binary file is received, and compared with the mode that format conversion is not performed in the prior art, the response speed is improved.
S303: if the data record to be migrated is a new data record in a queue to be stored of the first database, storing the non-binary file in the first database and the second database; and if the data record to be migrated is a historical data record in the stored list of the first database, storing the non-binary file in the second database.
Here, the non-binary file in the data record to be migrated is stored in the second database after performing model conversion processing (for example, splitting a data table), and the non-binary file is also stored in the first database when the data record to be migrated is new data.
S304: a database servicing the online service is migrated from the first database to the second database.
Specifically, a database interface for providing service for online services is switched from an access interface of the first database to an access interface of the second database.
Here, the database providing the data service to the client switches to the second database, that is, the data service interface using the second data provides the data service, but the interface of the client is not changed.
When a library needs to be migrated, the binary file in the data record to be migrated and the converted text file are stored in the distributed file system, and then the identification information for identifying the storage position of the binary file in the distributed file system and the identification information for identifying the storage position of the text file in the distributed file system are stored in the new database. Therefore, the identification information of the binary file and the converted text file is only stored in the new database, so that the efficiency of migrating data into the new database can be greatly improved. In addition, the distributed file system can provide high-reliability and high-concurrency storage access for the outside, so that the data access performance after data migration can be improved. Therefore, the storage and access capacity of the binary file can be improved. According to the method and the device, format conversion is performed when the binary file is stored, namely text content in the binary file is extracted to be stored, after the conversion is performed, after a page access request of a client for the binary file is received, page rendering feedback can be performed on the content of the binary file quickly, and compared with the mode that format conversion is not performed in the prior art, the response speed is improved.
Example two
Referring to the data migration diagram shown in fig. 1, a specific implementation process is provided in embodiment two of the present application:
the following S401a to S404a and S405 to S407 are performed for new data received after the determination of the migration, and the following S401b to S404b and S405 to S407 are performed for history data stored in the first database before the determination of the migration.
S401 a: and writing each received new data record into a pending queue of the first database.
S402 a: and sequentially taking out each new data record from the queue to be processed of the first database, distinguishing a binary file and a non-binary file in each new data record, executing the following S403a for the binary file, and executing the following S404a for the non-binary file.
S403 a: a text file is obtained based on the binary file conversion, the text file and the binary file are stored in a distributed file system, and identification information which respectively corresponds to the text file and the binary file and identifies the storage positions in the distributed file system is stored in a second database; and synchronously storing the text file in a first database.
S404 a: and after model conversion is carried out on the non-binary file, the non-binary file is synchronously stored in a first database and a second database.
S401 b: and writing the identification information of each historical data record in the first database into a preset data table.
S402 b: extracting each unprocessed historical data record from the first database respectively based on the identification information of each historical data record in the preset data table, distinguishing a binary file from a non-binary file in each extracted historical data record, executing the following S403b for the binary file, and executing the following S404b for the non-binary file. And recording the processing state of the historical data record in a preset data table.
S403 b: extracting a text in a binary file, storing the text and the binary file in a distributed file system, and storing identification information for identifying the storage positions of the text and the binary file in the distributed file system in a second database.
S404 b: and after model conversion is carried out on the non-binary file, the non-binary file is stored in a second database.
S405: and after the data migration is finished, migrating the database for providing service for the online service from the first database to the second database.
S406: when a page access request of a client aiming at a specified binary file is received, a text file stored in a distributed file system is obtained based on identification information of a text file corresponding to the specified binary file in a second database, and rendering feedback is carried out on an access page based on the text file.
S407: when a downloading request of a client for a specified binary file is received, the specified binary file stored in the distributed file system is obtained based on the identification information of the specified binary file in the second database, the specified binary file is cached in the CDN, and the corresponding access address of the CDN is returned to the client.
Based on the same inventive concept, the embodiment of the present application further provides a data migration apparatus corresponding to the data migration method, and as the principle of the apparatus for solving the problem is similar to the data migration method in the embodiment of the present application, the implementation of the apparatus may refer to the implementation of the method, and repeated details are not repeated.
As shown in fig. 5, a structure diagram of a data migration apparatus provided in an embodiment of the present application includes:
a determining module 51, configured to determine a first file to be migrated based on a data record to be migrated in a first database;
the storage module 52 is configured to store the first file to be migrated in a set file system, and store the identification information of the first file to be migrated in a second database;
the identification information of the first file to be migrated is information for identifying a storage location of the first file to be migrated in the file system.
Optionally, the apparatus further comprises:
a first obtaining module 53, configured to, when receiving an access request of a client for the first file to be migrated, obtain, based on the identification information of the first file to be migrated in the second database, the first file to be migrated stored in the file system, and return the first file to be migrated to the client.
Optionally, the determining module 51 is specifically configured to:
extracting a binary file from a data record to be migrated in a first database; and determining at least one of the extracted binary file and a text file converted based on the binary file as the first file to be migrated.
Optionally, the apparatus further comprises:
a second obtaining module 54, configured to, when a download request of a client for the binary file is received, obtain the binary file from the file system based on the identification information of the binary file in the second database, and cache the obtained binary file in a content delivery network CDN; and returning the access address of the binary file in the CDN to the client.
Optionally, if the data record to be migrated is a new data record located in the queue to be stored in the first database, the storage module 52 is further configured to:
and storing the text file converted based on the binary file in the first database.
Optionally, the determining module 51 is further configured to:
extracting a second file to be migrated from a data record to be migrated in a first database, wherein the second file to be migrated is a non-binary file;
the storage module 52 is further configured to: if the data record to be migrated is a new data record in the queue to be stored of the first database, storing the second file to be migrated in the first database and the second database; and if the data record to be migrated is a historical data record in the stored list of the first database, storing the second file to be migrated in the second database.
Optionally, the apparatus further comprises:
a recording module 55, configured to write the identification information of each historical data record in the stored list of the first database into a preset data table; the identification information of each historical data record is used for identifying the storage position of the historical data record in the first database; and respectively extracting each unprocessed historical data record from the first database as the data record to be migrated based on the identification information of each historical data record in the preset data table, and recording the processing state of the historical data record in the preset data table.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (15)
1. A method for data migration, the method comprising:
determining a first file to be migrated based on a data record to be migrated in a first database; the first file to be migrated is historical data written into the first database or new data to be written into the first database after the database migration is determined;
storing the first file to be migrated in a set file system, and storing the identification information of the first file to be migrated in a second database;
the identification information of the first file to be migrated is used for identifying the storage position of the first file to be migrated in the file system;
after the data migration is completed, the online business is migrated from the first database to the second database.
2. The method of claim 1, wherein the file system is a distributed file system.
3. The method of claim 1, wherein after storing the first file to be migrated in a set file system and storing identification information of the first file to be migrated in a second database, further comprising:
when an access request of a client for the first file to be migrated is received, the first file to be migrated stored in the file system is acquired based on the identification information of the first file to be migrated in the second database, and the acquired first file to be migrated is returned to the client.
4. The method of claim 1, wherein determining the first file to be migrated based on the data records to be migrated in the first database comprises:
extracting a binary file from the data record to be migrated;
and determining at least one of the extracted binary file and a text file converted based on the binary file as the first file to be migrated.
5. The method of claim 4, wherein if the first file to be migrated comprises the binary file, after storing the binary file in a set file system and storing identification information of the binary file in a second database, further comprising:
when a downloading request of a client for the binary file is received, acquiring the binary file from the file system based on the identification information of the binary file in the second database, and caching the acquired binary file in a Content Delivery Network (CDN);
and returning the access address of the binary file in the CDN to the client.
6. The method of claim 4, wherein if the data record to be migrated is a new data record located in a queue to be stored of the first database, the method further comprises:
and storing the text file converted based on the binary file in the first database.
7. The method of claim 4, wherein the method further comprises:
extracting a second file to be migrated from the data record to be migrated, wherein the second file to be migrated is a non-binary file;
if the data record to be migrated is a new data record in the queue to be stored of the first database, storing the second file to be migrated in the first database and the second database;
and if the data record to be migrated is a historical data record in the stored list of the first database, storing the second file to be migrated in the second database.
8. The method of claim 1, wherein the method further comprises:
writing the identification information of each historical data record in the stored list of the first database into a preset data table; the identification information of each historical data record is used for identifying the storage position of the historical data record in the first database;
and respectively extracting each unprocessed historical data record from the first database as the data record to be migrated based on the identification information of each historical data record in the preset data table, and recording the processing state of the historical data record in the preset data table.
9. A data migration apparatus, comprising:
the determining module is used for determining a first file to be migrated based on the data record to be migrated in the first database; the first file to be migrated is historical data written into the first database or new data to be written into the first database after the database migration is determined;
the storage module is used for storing the first file to be migrated in a set file system and storing the identification information of the first file to be migrated in a second database;
after the data migration is completed, migrating the online service from the first database to a second database;
the identification information of the first file to be migrated is information for identifying a storage location of the first file to be migrated in the file system.
10. The apparatus of claim 9, wherein the apparatus further comprises:
and the first obtaining module is used for obtaining the first file to be migrated stored in the file system based on the identification information of the first file to be migrated in the second database when receiving an access request of a client for the first file to be migrated, and returning the first file to be migrated to the client.
11. The apparatus of claim 9, wherein the determination module is specifically configured to:
extracting a binary file from the data record to be migrated; and determining at least one of the extracted binary file and a text file converted based on the binary file as the first file to be migrated.
12. The apparatus of claim 11, wherein the apparatus further comprises:
the second obtaining module is used for obtaining the binary file from the file system based on the identification information of the binary file in the second database when a downloading request of a client for the binary file is received, and caching the obtained binary file in a Content Delivery Network (CDN); and returning the access address of the binary file in the CDN to the client.
13. The apparatus of claim 11, wherein if the data record to be migrated is a new data record located in a queue to be stored of the first database, the storage module is further configured to:
and storing the text file converted based on the binary file in the first database.
14. The apparatus of claim 11, wherein the determination module is further configured to:
extracting a second file to be migrated from the data record to be migrated, wherein the second file to be migrated is a non-binary file;
the storage module is further configured to: if the data record to be migrated is a new data record in the queue to be stored of the first database, storing the second file to be migrated in the first database and the second database; and if the data record to be migrated is a historical data record in the stored list of the first database, storing the second file to be migrated in the second database.
15. The apparatus of claim 9, wherein the apparatus further comprises:
the recording module is used for writing the identification information of each historical data record in the stored list of the first database into a preset data table; the identification information of each historical data record is used for identifying the storage position of the historical data record in the first database; and respectively extracting each unprocessed historical data record from the first database as the data record to be migrated based on the identification information of each historical data record in the preset data table, and recording the processing state of the historical data record in the preset data table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610445610.1A CN106909595B (en) | 2016-06-20 | 2016-06-20 | Data migration method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610445610.1A CN106909595B (en) | 2016-06-20 | 2016-06-20 | Data migration method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106909595A CN106909595A (en) | 2017-06-30 |
| CN106909595B true CN106909595B (en) | 2020-12-29 |
Family
ID=59206721
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610445610.1A Active CN106909595B (en) | 2016-06-20 | 2016-06-20 | Data migration method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106909595B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107704633A (en) * | 2017-11-01 | 2018-02-16 | 郑州云海信息技术有限公司 | A kind of method and system of file migration |
| CN110019133B (en) * | 2017-12-21 | 2021-07-13 | 北京京东尚科信息技术有限公司 | Data online migration method and device |
| CN107920136B (en) * | 2017-12-29 | 2019-09-17 | Oppo广东移动通信有限公司 | Data synchronization control method, device and server |
| CN108921065A (en) * | 2018-06-21 | 2018-11-30 | 北京陌上花科技有限公司 | The method and apparatus for establishing property data base |
| CN111475467A (en) * | 2020-03-27 | 2020-07-31 | 平安科技（深圳）有限公司 | File management method, cloud file management system and terminal |
| CN112527775A (en) * | 2020-12-18 | 2021-03-19 | 福建天晴数码有限公司 | Database expansion method and device based on double writing |
| CN112631991A (en) * | 2020-12-26 | 2021-04-09 | 中国农业银行股份有限公司 | File migration method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7620665B1 (en) * | 2000-11-21 | 2009-11-17 | International Business Machines Corporation | Method and system for a generic metadata-based mechanism to migrate relational data between databases |
| CN104598459A (en) * | 2013-10-30 | 2015-05-06 | 阿里巴巴集团控股有限公司 | Database processing method and system and data access method and system |
| CN105354250A (en) * | 2015-10-16 | 2016-02-24 | 浪潮(北京)电子信息产业有限公司 | Data storage method and device for cloud storage |
| CN105530273A (en) * | 2014-09-29 | 2016-04-27 | 国际商业机器公司 | Method and device for data migration |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7523122B2 (en) * | 2006-09-25 | 2009-04-21 | Microsoft Corporation | Automatic discovery of application settings' storage locations |
| US20110060719A1 (en) * | 2009-09-05 | 2011-03-10 | Vivek Kapoor | Method for Transforming Setup Data in Business Applications |
| CN103631907B (en) * | 2013-11-26 | 2016-09-07 | 中国科学院信息工程研究所 | A kind of method and system that relational data is migrated to HBase |
| CN104572842B (en) * | 2014-12-04 | 2018-12-28 | 长沙数析信息科技有限公司 | A kind of striping in distributed file system rolls up growth data moving method |
-
2016
- 2016-06-20 CN CN201610445610.1A patent/CN106909595B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7620665B1 (en) * | 2000-11-21 | 2009-11-17 | International Business Machines Corporation | Method and system for a generic metadata-based mechanism to migrate relational data between databases |
| CN104598459A (en) * | 2013-10-30 | 2015-05-06 | 阿里巴巴集团控股有限公司 | Database processing method and system and data access method and system |
| CN105530273A (en) * | 2014-09-29 | 2016-04-27 | 国际商业机器公司 | Method and device for data migration |
| CN105354250A (en) * | 2015-10-16 | 2016-02-24 | 浪潮(北京)电子信息产业有限公司 | Data storage method and device for cloud storage |
Non-Patent Citations (3)
| Title |
|---|
| 于海涛.基于Java的异构数据库间数据迁移.《中国优秀硕士学位论文全文数据库 信息科技辑》.中国学术期刊(光盘版)电子杂志社,2008,(第05期),第20-21，39页. * |
| 基于Java的异构数据库间数据迁移;于海涛;《中国优秀硕士学位论文全文数据库 信息科技辑》;中国学术期刊(光盘版)电子杂志社;20080515(第05期);第21，38-39页 * |
| 非结构化到结构化数据转换的研究与实现;万里鹏;《中国优秀硕士学位论文全文数据库 信息科技辑》;中国学术期刊（光盘版）电子杂志社;20131115(第11期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106909595A (en) | 2017-06-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106909595B (en) | Data migration method and device | |
| CN104133867A (en) | DOT in-fragment secondary index method and DOT in-fragment secondary index system | |
| CN109033360B (en) | Data query method, device, server and storage medium | |
| CN106649828B (en) | Data query method and system | |
| US11599591B2 (en) | System and method for updating a search index | |
| US11507277B2 (en) | Key value store using progress verification | |
| CN104657435A (en) | Storage management method for application data and network management system | |
| CN109992469B (en) | Method and device for merging logs | |
| CN106021566A (en) | Method, device and system for improving concurrent processing capacity of single database | |
| US11157456B2 (en) | Replication of data in a distributed file system using an arbiter | |
| CN115114232A (en) | Method, device and medium for enumerating historical version objects | |
| US11080239B2 (en) | Key value store using generation markers | |
| CN112860412B (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN110019169B (en) | Data processing method and device | |
| CN107665224B (en) | Method, system and device for scanning HDFS cold data | |
| CN112965939A (en) | File merging method, device and equipment | |
| US10083121B2 (en) | Storage system and storage method | |
| US9817853B1 (en) | Dynamic tier-maps for large online databases | |
| CN107340974B (en) | Virtual disk migration method and virtual disk migration device | |
| CN109359085A (en) | File migration method, apparatus, computer equipment and storage medium | |
| CN111061719B (en) | Data collection method, device, equipment and storage medium | |
| CN113282347A (en) | Plug-in operation method, device, equipment and storage medium | |
| CN112528119A (en) | Distributed webpage information crawling system based on Pulsar | |
| CN107679093B (en) | Data query method and device | |
| CN111651531A (en) | Data import method, device, equipment and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200924 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant after: Innovative advanced technology Co.,Ltd. Address before: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant before: Advanced innovation technology Co.,Ltd. Effective date of registration: 20200924 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant after: Advanced innovation technology Co.,Ltd. Address before: Cayman Islands Grand Cayman capital building, a four storey No. 847 mailbox Applicant before: Alibaba Group Holding Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |