CN110568995A - Data migration method and system - Google Patents
Data migration method and system Download PDFInfo
- Publication number
- CN110568995A CN110568995A CN201910818684.9A CN201910818684A CN110568995A CN 110568995 A CN110568995 A CN 110568995A CN 201910818684 A CN201910818684 A CN 201910818684A CN 110568995 A CN110568995 A CN 110568995A
- Authority
- CN
- China
- Prior art keywords
- data
- database server
- storage area
- temporary storage
- storage unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data migration method and a system, wherein the method comprises the following steps: when data of the database server are migrated, an external access interface of the database server is closed, the data of the database server are migrated from a data storage unit of an old architecture to a data storage unit of a new architecture, and a temporary storage area is arranged to receive externally imported bottom data; and after the data migration of the database server is finished, migrating the data in the temporary storage area to a data storage unit of the new architecture. By adopting the technical scheme of the invention, the real-time migration and the lossless migration of the data in the database server can be realized.
Description
Technical Field
The present invention relates to the field of storage, and in particular, to a data migration method and system.
background
In energy internet application, the existing energy management system checks the data requirements of the whole system transmission and storage network through web and app, and also collects and stores system energy data and state data in real time. After data storage for several years, the data volume is increased rapidly, and a data storage architecture is required to be designed completely. In order to ensure that the data access performance is not reduced, the original architecture data needs to be migrated to a new architecture, and in the migration process, the target needs to support real-time migration and lossless migration, namely, the data needs to be ensured to be still collected and stored in the migration process, and after the migration is completed, the data stored in the migration process needs to be stored in the new architecture. How to implement real-time migration and lossless migration of data in a database server is a technical problem which needs to be solved currently.
Disclosure of Invention
The present invention provides a data migration method and system, aiming at the technical problem that real-time migration and lossless migration need to be supported during data migration in a database server in the prior art.
In an embodiment of the present invention, a data migration method is provided, which includes:
When data of the database server are migrated, an external access interface of the database server is closed, the data of the database server are migrated from a data storage unit of an old architecture to a data storage unit of a new architecture, and a temporary storage area is arranged to receive externally imported bottom data;
And after the data migration of the database server is finished, migrating the data in the temporary storage area to a data storage unit of the new architecture.
in this embodiment of the present invention, the data migration method further includes:
And before the data of the database server is migrated, closing an external access interface of the database server.
In this embodiment of the present invention, the data migration method further includes:
And after the data in the temporary storage area is migrated to the data storage unit of the new framework, closing the temporary storage area, receiving externally imported bottom data by adopting the data storage unit of the new framework, and opening an external access interface of the database server.
In this embodiment of the present invention, the data migration method further includes:
and before storing the externally imported bottom layer data into the temporary storage area, filtering the externally imported bottom layer data.
In the embodiment of the invention, in the process of migrating the data in the temporary storage area to the data storage unit of the new architecture, the temporary storage area is adopted to receive externally imported bottom layer data, and the speed of migrating the data from the temporary storage area to the data storage unit of the new architecture is higher than the speed of importing the bottom layer data from the temporary storage area.
In an embodiment of the present invention, a data migration system is further provided, which includes:
the logic processing unit is used for closing an external access interface of the database server and setting a temporary storage area to receive externally imported bottom data when the data of the database server are migrated;
And the data migration unit is used for migrating the data of the database server from the data storage unit of the old architecture to the data storage unit of the new architecture, and is also used for migrating the data of the temporary storage area to the data storage unit of the new architecture after the data migration of the database server is finished.
in the embodiment of the present invention, before the data migration unit migrates data of the database server, the logic processing unit is further configured to close an external access interface of the database server.
In the embodiment of the present invention, after the data migration unit migrates the data in the temporary storage area to the data storage unit of the new architecture, the logic processing unit closes the temporary storage area, receives externally imported bottom data by using the data storage unit of the new architecture, and opens the external access interface of the database server.
In the embodiment of the invention, before storing the externally imported bottom layer data in the temporary storage area, the logic processing unit performs filtering processing on the externally imported bottom layer data.
in the embodiment of the present invention, the speed of migrating data from the temporary storage area to the data storage unit of the new architecture by the data migration unit is greater than the speed of importing the bottom layer data from the outside by the temporary storage area.
Compared with the prior art, in the data migration method and the data migration system, when the data of the database server is migrated, closing an external access interface of the database server, migrating the data of the database server from the data storage unit of the old architecture to the data storage unit of the new architecture, setting a temporary storage area to receive externally imported bottom data, after the data migration of the database server is completed, migrating the data in the temporary storage area to the data storage unit of the new architecture, after migrating the data in the temporary storage area to the data storage unit of the new architecture, closing the temporary storage area, adopting the data storage unit of the new architecture to receive externally imported bottom layer data, and an external access interface of the database server is opened, so that the real-time migration and the lossless migration of the data in the database server can be realized.
Drawings
Fig. 1 is a schematic structural diagram of a data migration system according to an embodiment of the present invention.
Fig. 2 is a flowchart illustrating a data migration method according to an embodiment of the present invention.
Detailed Description
The invention provides a data migration method and a system, when migrating data of a database server, an external access interface of the database server is closed, the data of the database server is migrated from a data storage unit of an old architecture to a data storage unit of a new architecture, and a temporary storage area is arranged to receive externally imported bottom data; after the data migration of the database server is completed, migrating the data in the temporary storage area to a data storage unit of a new architecture; and after the data in the temporary storage area is migrated to the data storage unit of the new framework, closing the temporary storage area, receiving externally imported bottom data by adopting the data storage unit of the new framework, and opening an external access interface of the database server.
as shown in fig. 1, in the embodiment of the present invention, the data migration system includes a data parsing unit 1, a logical processing unit 2, a data migration unit 3, and a data storage area 4. The data analysis unit 1, the logic processing unit 2, and the data migration unit 3 may be components of a database server, or may be independent of the database server. The data storage area 4 is a storage space of the database server, is used for storing externally imported bottom layer data, and is provided with a plurality of data storage units. In general, the data storage unit adopts a set data storage architecture, and in the case of a rapid increase in data volume, the storage architecture of the data storage unit may need to be redesigned, and a new data storage architecture is adopted, so as to support the data storage requirement of a larger capacity.
The data analysis unit 1 is configured to analyze externally-imported bottom layer data, and import the analyzed data into the data storage area 4 of the database server through the logic processing unit 2. Since encoding and encryption are required for the externally-introduced underlying data during data transmission, data analysis operations such as decoding and decryption need to be performed by the data analysis unit 1 when the underlying data is introduced into the database. The data parsing unit 1 is further configured to parse the data into corresponding data structures, so as to store the data structures in the data storage units of the corresponding data structures.
And the logic processing unit 2 is configured to import the bottom layer data analyzed by the data analysis unit 1 into a data storage unit of a database storage space, and is further configured to respond to a request of an APP or a web page end corresponding to the database server to access and query the database server. When migrating the data of the database server, the logical processing unit 2 is configured to close an external access interface of the database server, and set a temporary storage area 5 to receive externally imported bottom layer data.
It should be noted that data in the database server needs to be migrated, and data in the data storage unit of the original architecture needs to be migrated to the data storage unit of the new architecture. At this time, because data migration is in progress, the logic processing unit 2 cannot respond to the request of the APP or the network page side to access and query the data storage unit in the database server, so before migrating the data of the database server, the logic processing unit 2 announces information that cannot be accessed by system upgrade to the interface page of the access database server, and prompts the user. At this time, since data migration is being performed, external underlying data cannot be directly imported into the data storage unit of the new architecture, and it is necessary to set the temporary storage area 5 to receive the externally imported underlying data. The temporary storage area 5 may be provided in the data storage area or in an external storage device.
The data migration unit 3 is configured to migrate data of the database server from the data storage unit of the old architecture to the data storage unit of the new architecture, and is further configured to migrate data in the temporary storage area 5 to the data storage unit of the new architecture after the data migration of the database server is completed.
After the data migration unit 3 migrates the data in the temporary storage area 5 to the data storage unit of the new architecture, the logic processing unit 2 closes the temporary storage area 5, receives externally imported bottom data by using the data storage unit of the new architecture, and opens an external access interface of the database server, so that a user can access the database server through an APP or a web page. The speed of the data migration unit 3 migrating data from the temporary storage area 5 to the data storage unit of the new architecture is greater than the speed of the temporary storage area 5 importing bottom data from the outside. In this embodiment of the present invention, the data migration unit 3 migrates data from the temporary storage area 5 to a data storage unit of a new architecture by using multiple threads, and the logic processing unit 2 stores externally-imported bottom data in the temporary storage area by using a single thread, so as to ensure that the speed of migrating data from the temporary storage area 5 to the data storage unit of the new architecture by the data migration unit 3 is greater than the speed of importing bottom data from the temporary storage area 5.
as shown in FIG. 2, the process of performing data migration on the database server by the data migration system includes steps S1-S4, which are described below.
Step S1: before data migration, the logic processing unit 2 actively closes the access interfaces of the web and the app of the database server, and actively pushes information that the existing system is upgraded and cannot be logged in to the interface pages of the web and the app of the database server.
Step S2: when data of the database server is migrated, the data of the database server is migrated from the data storage unit of the old architecture to the data storage unit of the new architecture, and a temporary storage area 5 is arranged to receive externally imported bottom data. It should be noted that, in order to ensure that the temporary storage area 5 stores as much data as possible and stores important data, the logic processing unit 2 performs filtering processing on the data in the time period, retains data such as statistical energy data, modified engineering information, account information, and the like, and filters out data such as real-time energy data, status data, and the like;
Step S3: and after the data migration of the database server is finished, migrating the data in the temporary storage area 5 to a data storage unit with a new architecture. It should be noted that, in the process of migrating the data in the temporary storage area 5 to the data storage unit of the new architecture, the temporary storage area 5 is adopted to receive externally imported bottom layer data, and the speed of migrating the data from the temporary storage area 5 to the data storage unit of the new architecture is greater than the speed of importing the bottom layer data from the outside by the temporary storage area 5, and the data is fetched from and transferred to the temporary storage area 5 by multiple threads, and the bottom layer data is stored in the temporary storage area 5 by a single thread;
Step S4: after the data in the temporary storage area is migrated to the data storage unit of the new architecture, the temporary storage area is closed, the data storage unit of the new architecture is adopted to receive externally imported bottom data, an external access interface of the database server is opened, information is actively pushed to web and app, and the prompt that the system is upgraded and can be normally logged in for use is given.
In summary, in the data migration method and system of the present invention, when migrating data of a database server, an external access interface of the database server is closed, the data of the database server is migrated from a data storage unit of an old architecture to a data storage unit of a new architecture, a temporary storage area is set to receive externally imported bottom data, after the data migration of the database server is completed, the data of the temporary storage area is migrated to the data storage unit of the new architecture, and after the data of the temporary storage area is migrated to the data storage unit of the new architecture, the temporary storage area is closed, the data storage unit of the new architecture is used to receive externally imported bottom data, and an external access interface of the database server is opened, so that real-time migration and lossless migration of data inside the database server can be achieved.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (10)
1. A method of data migration, comprising:
When data of a database server are migrated, a temporary storage area is arranged to receive externally imported bottom data, and the data of the database server are migrated from a data storage unit of an old architecture to a data storage unit of a new architecture;
and after the data migration of the database server is finished, migrating the data in the temporary storage area to a data storage unit of the new architecture.
2. the data migration method of claim 1, further comprising:
And before the data of the database server is migrated, closing an external access interface of the database server.
3. the data migration method of claim 2, further comprising:
and after the data in the temporary storage area is migrated to the data storage unit of the new framework, closing the temporary storage area, receiving externally imported bottom data by adopting the data storage unit of the new framework, and opening an external access interface of the database server.
4. The data migration method of claim 1, further comprising:
and before storing the externally imported bottom layer data into the temporary storage area, filtering the externally imported bottom layer data.
5. The data migration method according to claim 1, wherein in the process of migrating the data in the temporary storage area to the data storage unit of the new architecture, the temporary storage area is used to receive externally imported bottom layer data, and the speed of migrating the data from the temporary storage area to the data storage unit of the new architecture is faster than the speed of importing the bottom layer data from the temporary storage area externally.
6. A data migration system is characterized by comprising a logic processing unit and a data migration unit,
the logic processing unit is used for closing an external access interface of the database server and setting a temporary storage area to receive externally imported bottom data when the data of the database server are migrated;
The data migration unit is used for migrating the data of the database server from the data storage unit of the old architecture to the data storage unit of the new architecture, and is also used for migrating the data of the temporary storage area to the data storage unit of the new architecture after the data migration of the database server is completed.
7. The data migration system of claim 6, wherein the logical processing unit is further configured to shut down an external access interface of the database server before the data migration unit migrates the data of the database server.
8. the data migration system according to claim 6, wherein after the data migration unit migrates the data in the temporary storage area to the data storage unit of the new architecture, the logic processing unit closes the temporary storage area, receives externally imported bottom data by using the data storage unit of the new architecture, and opens an external access interface of the database server.
9. The data migration system of claim 6, wherein the logical processing unit filters the externally imported underlying data prior to saving the externally imported underlying data to the temporary storage area.
10. The data migration system of claim 6, wherein the speed at which the data migration unit migrates data from the temporary storage area to the data storage unit of the new architecture is greater than the speed at which the temporary storage area imports underlying data from outside.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910818684.9A CN110568995A (en) | 2019-08-30 | 2019-08-30 | Data migration method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910818684.9A CN110568995A (en) | 2019-08-30 | 2019-08-30 | Data migration method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110568995A true CN110568995A (en) | 2019-12-13 |
Family
ID=68777150
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910818684.9A Pending CN110568995A (en) | 2019-08-30 | 2019-08-30 | Data migration method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110568995A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112631991A (en) * | 2020-12-26 | 2021-04-09 | 中国农业银行股份有限公司 | File migration method and device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1783876A (en) * | 2004-12-03 | 2006-06-07 | 华为技术有限公司 | System and method for realizing packet data polymerized protocol function |
| CN101957729A (en) * | 2010-09-27 | 2011-01-26 | 中兴通讯股份有限公司 | Logical block transformation method and method and device compatible with reading and writing of user based on same |
| CN102103629A (en) * | 2010-12-14 | 2011-06-22 | 西北工业大学 | Online data migration method |
| CN103678163A (en) * | 2012-09-18 | 2014-03-26 | 腾讯科技（深圳）有限公司 | Method, device and system for switching of data stream |
| CN105718570A (en) * | 2016-01-20 | 2016-06-29 | 北京京东尚科信息技术有限公司 | Data migration method and device used for database |
| CN106649419A (en) * | 2015-12-31 | 2017-05-10 | 国网河北省电力公司 | Electricity information acquisition system-based cross-platform database data migration method |
| CN107193489A (en) * | 2017-05-15 | 2017-09-22 | 郑州云海信息技术有限公司 | A kind of storage level data migration method and device based on storage virtual gateway |
-
2019
- 2019-08-30 CN CN201910818684.9A patent/CN110568995A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1783876A (en) * | 2004-12-03 | 2006-06-07 | 华为技术有限公司 | System and method for realizing packet data polymerized protocol function |
| CN101957729A (en) * | 2010-09-27 | 2011-01-26 | 中兴通讯股份有限公司 | Logical block transformation method and method and device compatible with reading and writing of user based on same |
| CN102103629A (en) * | 2010-12-14 | 2011-06-22 | 西北工业大学 | Online data migration method |
| CN103678163A (en) * | 2012-09-18 | 2014-03-26 | 腾讯科技（深圳）有限公司 | Method, device and system for switching of data stream |
| CN106649419A (en) * | 2015-12-31 | 2017-05-10 | 国网河北省电力公司 | Electricity information acquisition system-based cross-platform database data migration method |
| CN105718570A (en) * | 2016-01-20 | 2016-06-29 | 北京京东尚科信息技术有限公司 | Data migration method and device used for database |
| CN107193489A (en) * | 2017-05-15 | 2017-09-22 | 郑州云海信息技术有限公司 | A kind of storage level data migration method and device based on storage virtual gateway |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112631991A (en) * | 2020-12-26 | 2021-04-09 | 中国农业银行股份有限公司 | File migration method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10783198B2 (en) | Row-based data filtering at a database level | |
| CN106021445B (en) | It is a kind of to load data cached method and device | |
| WO2019148713A1 (en) | Sql statement processing method and apparatus, computer device, and storage medium | |
| CN107122360A (en) | Data mover system and method | |
| US20090204570A1 (en) | Method and apparatus for performing multi-stage table updates | |
| CN107004016B (en) | Efficient data manipulation support | |
| CN107122361A (en) | Data mover system and method | |
| CN111291023A (en) | Data migration method, system, device and medium | |
| CN107179878A (en) | The method and apparatus of data storage based on optimizing application | |
| CN107766431B (en) | Parameterization removing function method and system based on grammar parsing | |
| CN109871364B (en) | Data fusion and association processing method | |
| Shi et al. | A case study of tuning MapReduce for efficient Bioinformatics in the cloud | |
| CN103136215A (en) | Data read-write method and device of storage system | |
| CN106021566A (en) | Method, device and system for improving concurrent processing capacity of single database | |
| CN108255966A (en) | A kind of data migration method and storage medium | |
| CN110287152A (en) | A kind of method and relevant apparatus of data management | |
| US7958083B2 (en) | Interacting methods of data summarization | |
| US8600990B2 (en) | Interacting methods of data extraction | |
| CN107145394A (en) | A kind of equally loaded processing method and processing device for data skew | |
| US20190384754A1 (en) | In-place updates with concurrent reads in a decomposed state | |
| CN106570151A (en) | Data collection processing method and system for mass files | |
| CN108959149A (en) | A kind of multi-core processor interaction bus design method based on shared drive | |
| CN110568995A (en) | Data migration method and system | |
| CN101425034A (en) | Log recording method and log system | |
| US20180081942A1 (en) | Managing Data Obsolescence in Relational Databases |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191213 |
|
| RJ01 | Rejection of invention patent application after publication |