CN110633265A - Method for realizing ES (ES) cross-multi-version data migration - Google Patents
Method for realizing ES (ES) cross-multi-version data migration Download PDFInfo
- Publication number
- CN110633265A CN110633265A CN201910780461.8A CN201910780461A CN110633265A CN 110633265 A CN110633265 A CN 110633265A CN 201910780461 A CN201910780461 A CN 201910780461A CN 110633265 A CN110633265 A CN 110633265A
- Authority
- CN
- China
- Prior art keywords
- data
- directory
- machine
- cluster
- backup
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000013508 migration Methods 0.000 title claims abstract description 45
- 230000005012 migration Effects 0.000 title claims abstract description 45
- 238000011084 recovery Methods 0.000 claims abstract description 18
- 238000004590 computer program Methods 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims description 3
- 238000009434 installation Methods 0.000 claims 1
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
An implementation method for ES cross-multi-version data migration comprises three different implementation methods, specifically backup and recovery of ES snapshot data, copying ES physical directories/files, and querying and exporting data. The method for realizing the ES cross-multi-version data migration comprises three different implementation methods of ES snapshot data backup and recovery, ES physical directory/file copying and data query and export, a user can select different data according to different operating environments or data information, the use method is flexible, and convenience is brought to the user during data migration.
Description
Technical Field
The invention relates to the technical field of internet, in particular to a method for realizing ES (electronic storage) cross-multi-version data migration.
Background
The ElasticSearch is a Lucene-based search server. It provides a distributed multi-user capable full-text search engine based on RESTful web interface. The Elasticsearch was developed in Java and published as open source under the Apache licensing terms, and is currently a popular enterprise-level search engine. The design is used in cloud computing, can achieve real-time search, and is stable, reliable, quick, and convenient to install and use.
At present, many users do not know the data migration method when migrating ES data, and cannot select a suitable data migration method according to actual situations, so that data migration is not possible or data migration efficiency is not high.
In order to solve the above problem, the present application provides a method for implementing ES data migration across multiple versions.
Disclosure of Invention
Objects of the invention
In order to solve the technical problems in the background art, the invention provides a method for realizing ES (electronic storage) cross-multi-version data migration.
(II) technical scheme
In order to solve the above problems, the present invention provides a method for implementing ES data migration across multiple versions, which includes three different implementation methods, specifically, ES snapshot data backup and recovery, ES physical directory/file copy, and ES physical directory/file query and export.
Preferably, the ES snapshot data backup and restoration is used to implement data migration, and the specific steps are as follows:
(1) remote Mount the directory of the new cluster machine to the old cluster machine;
(2) using sshfs to Mount a directory or Mount a directory with Mount nfs;
(3) creating an ES warehouse;
(4) creating a snapshot backup;
(5) checking the state of the backup;
(6) after the backup is finished, the mounting is cancelled;
(7) creating a warehouse in the new cluster, and performing backup data recovery by using RESTful API;
(8) and checking the recovery state until the recovery is completed, and finishing the data migration.
Preferably, the specific method for mounting the directory by using sshfs is as follows: installing sshfs on each machine; creating a Mount shared directory on each machine; mounting a shared directory on each machine of the old cluster (respectively mounting the/opt/data 07 directories to/opt/backup _ es of the new machine); testing whether a user running the ES has write permission to the shared directory; the rights to share the directory are given to the running users of the ES on the old machine.
Preferably, the specific method for mounting the directory by using Mount nfs is as follows: adding a shared folder and an IP address which can be accessed by a client on the machine of the new cluster; viewing a shared folder; restarting and starting NFS service of the new cluster machine; creating a shared directory on each machine of the old cluster; mount mounting is carried out on the old cluster machine; the rights to share the directory are given to the running users of the ES on the old machine.
Preferably, the data migration is realized by using the copied ES physical directory/file, and the specific steps are as follows:
(1) finding a storage directory of the ES file in a magnetic disk;
(2) finding and copying a folder according to the cluster name;
(3) writing migration codes;
(4) and warehousing the data into the new ES cluster.
Preferably, the data migration is realized by using the query and the derived data, and the specific steps are as follows:
(1) querying an API of the ES according to the ES;
(2) writing a program according to the API of the ES;
(3) exporting the ES data from the csv file through a computer program; querying the ES data through a computer program;
(4) and warehousing into a new ES cluster.
Preferably, a pyes module of python needs to be installed when data migration is realized by using the query and exporting data.
The technical scheme of the invention has the following beneficial technical effects:
the method for realizing the ES cross-multi-version data migration comprises three different implementation methods of ES snapshot data backup and recovery, ES physical directory/file copying and data query and export, a user can select different data according to different operating environments or data information, the use method is flexible, and convenience is brought to the user during data migration.
Drawings
Fig. 1 is a flowchart of a first method in the implementation method of ES cross multi-version data migration according to the present invention.
FIG. 2 is a flowchart of a second method of the method for implementing ES data migration across multiple versions according to the present invention.
FIG. 3 is a flowchart of a third method in the implementation method of ES cross-multi-version data migration proposed by the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings in conjunction with the following detailed description. It should be understood that the description is intended to be exemplary only, and is not intended to limit the scope of the present invention. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present invention.
The invention provides a method for realizing ES (ES) cross-multi-version data migration, which comprises three different realization methods, specifically backup and recovery of ES snapshot data, copying of ES physical directories/files, and query and export of data.
As shown in fig. 1, a method for implementing ES cross-multi-version data migration, which implements data migration by using ES snapshot data backup and recovery, includes the following specific steps:
(1) remote Mount the directory of the new cluster machine to the old cluster machine;
(2) using sshfs to Mount a directory or Mount a directory with Mount nfs;
(3) creating an ES warehouse;
(4) creating a snapshot backup;
(5) checking the state of the backup;
(6) after the backup is finished, the mounting is cancelled;
(7) creating a warehouse in the new cluster, and performing backup data recovery by using RESTful API;
(8) and checking the recovery state until the recovery is completed, and finishing the data migration.
In an alternative embodiment, the specific method for mounting the directory using sshfs is as follows: installing sshfs on each machine; creating a Mount shared directory on each machine; mounting a shared directory on each machine of the old cluster (respectively mounting the/opt/data 07 directories to/opt/backup _ es of the new machine); testing whether a user running the ES has write permission to the shared directory; the rights to share the directory are given to the running users of the ES on the old machine.
In an alternative embodiment, the specific method for mounting the directory using Mount nfs is as follows: adding a shared folder and an IP address which can be accessed by a client on the machine of the new cluster; viewing a shared folder; restarting and starting NFS service of the new cluster machine; creating a shared directory on each machine of the old cluster; mount mounting is carried out on the old cluster machine; the rights to share the directory are given to the running users of the ES on the old machine.
As shown in fig. 2, a method for implementing ES cross-multi-version data migration, which implements data migration by copying ES physical directories/files, includes the following specific steps:
(1) finding a storage directory of the ES file in a magnetic disk;
(2) finding and copying a folder according to the cluster name;
(3) writing migration codes;
(4) and warehousing the data into the new ES cluster.
As shown in fig. 3, a method for implementing ES data migration across multiple versions adopts query and export data to implement data migration, and includes the following specific steps:
(1) querying an API of the ES according to the ES;
(2) writing a program according to the API of the ES;
(3) exporting the ES data from the csv file through a computer program; querying the ES data through a computer program;
(4) and warehousing into a new ES cluster.
In an alternative embodiment, a pyes module of python needs to be installed when data migration is implemented by using the query and exporting data.
The method for realizing the ES cross-multi-version data migration comprises three different implementation methods of ES snapshot data backup and recovery, ES physical directory/file copying and data query and export, a user can select different data according to different operating environments or data information, the use method is flexible, and convenience is brought to the user during data migration.
It is to be understood that the above-described embodiments of the present invention are merely illustrative of or explaining the principles of the invention and are not to be construed as limiting the invention. Therefore, any modification, equivalent replacement, improvement and the like made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. Further, it is intended that the appended claims cover all such variations and modifications as fall within the scope and boundaries of the appended claims or the equivalents of such scope and boundaries.
Claims (7)
1. An implementation method for ES (ES) cross-multi-version data migration is characterized by comprising three different implementation methods, specifically backup and recovery of ES snapshot data, copying of ES physical directories/files, and query and export of data.
2. The method for implementing ES cross multi-version data migration according to claim 1, wherein the specific steps of implementing the data migration for ES snapshot data backup and recovery are as follows:
(1) remote Mount the directory of the new cluster machine to the old cluster machine;
(2) using sshfs to Mount a directory or Mount a directory with Mount nfs;
(3) creating an ES warehouse;
(4) creating a snapshot backup;
(5) checking the state of the backup;
(6) after the backup is finished, the mounting is cancelled;
(7) creating a warehouse in the new cluster, and performing backup data recovery by using RESTful API;
(8) and checking the recovery state until the recovery is completed, and finishing the data migration.
3. The method of claim 2, wherein the specific method for mounting the directory using sshfs is as follows: installing sshfs on each machine; creating a Mount shared directory on each machine; mounting a shared directory on each machine of the old cluster (respectively mounting the/opt/data 07 directories to/opt/backup _ es of the new machine); testing whether a user running the ES has write permission to the shared directory; the rights to share the directory are given to the running users of the ES on the old machine.
4. The method according to claim 2, wherein the specific method for mounting the directory using Mount nfs is as follows: adding a shared folder and an IP address which can be accessed by a client on the machine of the new cluster; viewing a shared folder; restarting and starting NFS service of the new cluster machine; creating a shared directory on each machine of the old cluster; mount mounting is carried out on the old cluster machine; the rights to share the directory are given to the running users of the ES on the old machine.
5. The method for implementing ES data migration across multiple versions according to claim 1, wherein the specific steps of implementing the data migration by copying ES physical directories/files are as follows:
(1) finding a storage directory of the ES file in a magnetic disk;
(2) finding and copying a folder according to the cluster name;
(3) writing migration codes;
(4) and warehousing the data into the new ES cluster.
6. The method of claim 1, wherein the steps of querying and exporting data to implement data migration are as follows:
(1) querying an API of the ES according to the ES;
(2) writing a program according to the API of the ES;
(3) exporting the ES data from the csv file through a computer program; querying the ES data through a computer program;
(4) and warehousing into a new ES cluster.
7. The method as claimed in claim 6, wherein the querying and exporting data requires installation of a pyes module of python when implementing data migration.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910780461.8A CN110633265A (en) | 2019-08-22 | 2019-08-22 | Method for realizing ES (ES) cross-multi-version data migration |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910780461.8A CN110633265A (en) | 2019-08-22 | 2019-08-22 | Method for realizing ES (ES) cross-multi-version data migration |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110633265A true CN110633265A (en) | 2019-12-31 |
Family
ID=68970600
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910780461.8A Pending CN110633265A (en) | 2019-08-22 | 2019-08-22 | Method for realizing ES (ES) cross-multi-version data migration |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110633265A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113282537A (en) * | 2021-06-15 | 2021-08-20 | 成都深思科技有限公司 | ES data migration system and migration method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104468734A (en) * | 2014-11-18 | 2015-03-25 | 国云科技股份有限公司 | Virtual cluster expanding method based on cloning |
| US9201887B1 (en) * | 2012-03-30 | 2015-12-01 | Emc Corporation | Cluster file server proxy server for backup and recovery |
| CN108255994A (en) * | 2017-12-29 | 2018-07-06 | 上海上讯信息技术股份有限公司 | A kind of database version management method based on database snapshot |
| CN109299225A (en) * | 2018-10-24 | 2019-02-01 | 平安科技（深圳）有限公司 | Log searching method, system, terminal and computer readable storage medium |
-
2019
- 2019-08-22 CN CN201910780461.8A patent/CN110633265A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9201887B1 (en) * | 2012-03-30 | 2015-12-01 | Emc Corporation | Cluster file server proxy server for backup and recovery |
| CN104468734A (en) * | 2014-11-18 | 2015-03-25 | 国云科技股份有限公司 | Virtual cluster expanding method based on cloning |
| CN108255994A (en) * | 2017-12-29 | 2018-07-06 | 上海上讯信息技术股份有限公司 | A kind of database version management method based on database snapshot |
| CN109299225A (en) * | 2018-10-24 | 2019-02-01 | 平安科技（深圳）有限公司 | Log searching method, system, terminal and computer readable storage medium |
Non-Patent Citations (1)
| Title |
|---|
| WYL9527: "大数据量Elastic数据迁移方法及注意事项", pages 1, Retrieved from the Internet <URL:https://blog.csdn.net/wyl9527/article/details/78787701?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522168914114616800222898765%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=168914114616800222898765&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_ecpm_v1~rank_v31_ecpm-1-78787701-null-null.142^v88^koosearch_v1, 239^v2^insert_chatgpt&utm_term=Elastic%E6%95%B0%E6%8D%AE%E8%BF%81%E7%A7%BB%E6%96%B9%E6%B3%95&spm=1018.2226.3001.4449> * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113282537A (en) * | 2021-06-15 | 2021-08-20 | 成都深思科技有限公司 | ES data migration system and migration method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11204844B2 (en) | File level access to block level incremental backups of a virtual disk | |
| US11392542B2 (en) | Image level copy or restore, such as image level restore without knowledge of data object metadata | |
| US10831609B2 (en) | Data storage system with LUN snapshot shipping using volume-to-object translation | |
| US10089183B2 (en) | Method and apparatus for reconstructing and checking the consistency of deduplication metadata of a deduplication file system | |
| EP3477481B1 (en) | Data block name based efficient restore of multiple files from deduplicated storage | |
| US8306950B2 (en) | Managing data access requests after persistent snapshots | |
| CA2729075C (en) | Using a snapshot as a data source | |
| JP4160933B2 (en) | Fast restore of file system usage on very large file systems | |
| US10536522B2 (en) | Data storage system with LUN archiving to cloud using volume-to-object translation | |
| US10496601B1 (en) | Efficient file system parsing using snap based replication | |
| US10102083B1 (en) | Method and system for managing metadata records of backups | |
| US8977662B1 (en) | Storing data objects from a flat namespace in a hierarchical directory structured file system | |
| US9020994B1 (en) | Client-based migrating of data from content-addressed storage to file-based storage | |
| CN103309772A (en) | Database file backup method and system based on cloud storage | |
| US9122689B1 (en) | Recovering performance of a file system post-migration | |
| EP4016312A1 (en) | Data operations using a cache table in a file system | |
| US10223206B1 (en) | Method and system to detect and delete uncommitted save sets of a backup | |
| Henry | Howard chu on lightning memory-mapped database | |
| US20220067065A1 (en) | Providing instant and distributed access to a source blob via copy-on-read blobs and link blobs | |
| US20200042229A1 (en) | Tape drive data reclamation | |
| CN110633265A (en) | Method for realizing ES (ES) cross-multi-version data migration | |
| US9965357B1 (en) | Virtual backup and restore of virtual machines | |
| CN113821490A (en) | Data synchronization method and device | |
| US10445183B1 (en) | Method and system to reclaim disk space by deleting save sets of a backup | |
| US7987335B1 (en) | Techniques for virtualizing data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191231 |