CN111221792A - Ceph-based rbd file transmission method and system - Google Patents
Ceph-based rbd file transmission method and system Download PDFInfo
- Publication number
- CN111221792A CN111221792A CN201911382609.9A CN201911382609A CN111221792A CN 111221792 A CN111221792 A CN 111221792A CN 201911382609 A CN201911382609 A CN 201911382609A CN 111221792 A CN111221792 A CN 111221792A
- Authority
- CN
- China
- Prior art keywords
- rbd
- file
- client
- distributed storage
- storage platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 37
- 238000000034 method Methods 0.000 title claims abstract description 19
- 230000006835 compression Effects 0.000 claims description 14
- 238000007906 compression Methods 0.000 claims description 14
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1464—Management of the backup or restore process for networked environments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1737—Details of further file system functions for reducing power consumption or coping with limited storage space, e.g. in mobile devices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/141—Setup of application sessions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/04—Protocols for data compression, e.g. ROHC
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Quality & Reliability (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a method and a system for rbd file transmission based on ceph, wherein the method for rbd file transmission comprises the following steps: a local rbd client accesses an rbd file of the ceph distributed storage platform; establishing an rbd file of a ceph distributed storage platform by a remote rbd client; establishing tcp long connection between a local rbd client and a remote rbd client; compressing and transmitting a local rbd client rbd file to a remote rbd client; the remote rbd client decompresses the received rbd file data stream and writes the data stream into the newly created rbd file in the step S2; and the local rbd client reads all the rbd files of the ceph distributed storage platform, sends an ending mark to the different-place rbd client, and stops the rbd files after the different-place rbd client receives the ending mark. The invention simplifies the rbd file transmission operation, saves the storage space, accelerates the transmission time and reduces the rbd file transmission cost.
Description
Technical Field
The invention relates to the technical field of file transmission of distributed storage, in particular to a ceph-based rbd file fast transmission method and a system thereof.
Background
ceph is the currently more popular virtualized storage architecture. In the industry, the best scenario used by ceph distributed storage clusters is block storage and file storage. Whereas the block storage file of ceph (rbd file) now supports only the RAW format (non-sparse file format). Therefore, when storing or transmitting the RAW format file, more network resources and storage resources need to be consumed.
In the prior art, the problem of cross-cluster file transmission is as follows:
1) the method comprises the following steps that (1) the rbd file is exported to a local file, then the file is transmitted, and finally the file is imported to a different place, so that the operation is complicated, and meanwhile, the rbd file is required to be stored in a sufficient space in the local place and the different place;
2) the rbd file transmission is a file stream, namely in an actual scene, the file capacity and the transmission flow are 1:1, so that the transmission time and the cost are greatly improved.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a ceph-based rbd file transmission method and a system thereof, which simplify the rbd file transmission operation, save the storage space, accelerate the transmission time and reduce the rbd file transmission cost.
In order to solve the technical problems, the invention provides the following technical scheme: a method for transmitting rbd files based on ceph comprises the following steps:
s1, accessing and opening the rbd file of the ceph distributed storage platform by the local rbd client;
s2, creating an rbd file of the ceph distributed storage platform by the remote rbd client, and storing the received data stream;
s3, establishing tcp long connection between the local rbd client and the remote rbd client;
s4, reading the rbd file of the ceph distributed storage platform by the local rbd client, compressing the rbd file and transmitting the compressed rbd file to the remote rbd client;
the rbd file is compressed once and then transmitted at the same time, and a data stream of the rbd file is generated in the process of transmitting the rbd file to a remote rbd client; a local rbd client sends data flow to a foreign rbd client through a tcp session;
s5, decompressing the received rbd file data stream by the allopatric rbd client, and writing the data stream into the newly-created rbd file in the step S2;
and S6, repeating the steps S4-S5 until the local rbd client reads all the rbd files of the ceph distributed storage platform and completely sends the rbd files to the allopatric rbd client, then the local rbd client sends an end mark to the allopatric rbd client, and the allopatric rbd client stops writing the rbd files of the ceph distributed storage platform after receiving the end mark.
Further, step S1 is preceded by step S0, where the local rbd client and the offsite rbd client are both connected to the ceph distributed storage platform.
Further, step S6 is followed by step S7, both the local rbd client and the remote rbd client close the connection with the ceph distributed storage platform, and the local rbd client and the remote rbd client disconnect the tcp long connection.
Further, in the step S4, the local rbd client reads the rbd file of the ceph distributed storage platform, performs lz4 compression on the rbd file, checks the rbd file by using xxHash-32, and then transmits the rbd file to the remote rbd client;
in the step S5, the allopatric rbd client decompresses the received rbd file data stream by lz4 and verifies the data stream by using xxHash-32, and then writes the data stream into the new rbd file in the step S2.
The invention also aims to provide a ceph-based rbd file rapid transmission system, which comprises a local rbd client, a remote rbd client and a ceph distributed storage platform, wherein the local rbd client and the remote rbd client are both connected with the ceph distributed storage platform;
the local rbd client comprises an access module, a first connection module, a compression transmission module and an end mark generation module;
the remote rbd client comprises a new rbd file generation module, a second connection module, a receiving and decompressing module and an end mark receiving module;
the first connection module is used for connecting the ceph distributed storage platform, connecting the second connection module and establishing tcp long connection; the second connection module is also used for connecting the ceph distributed storage platform;
the access module is used for accessing and opening the rbd file of the ceph distributed storage platform and reading the rbd file of the ceph distributed storage platform;
the new rbd file generation module is used for creating an rbd file of the ceph distributed storage platform;
the compression transmission module is used for compressing the rbd file of the ceph distributed storage platform and transmitting the rbd file to the allopatric rbd client;
the receiving and decompressing module is used for receiving and decompressing the rbd file transmitted by the compression transmission module and writing the rbd file into a newly-built rbd file of the new rbd file generating module;
the end mark generating module is used for generating an end mark and sending the end mark to the end mark receiving module; after the ending mark receiving module receives an ending mark, the remote rbd client is instructed to stop receiving the rbd file of the ceph distributed storage platform; and the end mark generation module stops transmitting the rbd file of the ceph distributed storage platform after generating an end mark.
After the technical scheme is adopted, the invention at least has the following beneficial effects: the invention does not occupy the disk space of the local rbd client and the allopatric rbd client during the rbd transmission; the ceph block file can be automatically backed up, so that intermediate steps are reduced; the invention improves the transmission efficiency of the rbd data stream and reduces the transmission cost.
Drawings
FIG. 1 is a flowchart illustrating the steps of a ceph-based rbd file transmission method according to the present invention;
FIG. 2 is a block diagram of a ceph-based rbd file transfer system according to the present invention.
Detailed Description
It should be noted that, in the present application, the embodiments and features of the embodiments may be combined with each other without conflict, and the present application is further described in detail with reference to the drawings and specific embodiments.
Example 1
The invention aims to realize the conversion of the rbd file into a compressed data stream and realize the rapid transmission of data; and realizing a rbd file backup mode of a cross ceph cluster.
Therefore, the embodiment proposes a method for fast transmitting rbd files based on ceph, as shown in fig. 1, the steps include:
and S10, the local rbd client and the allopatric rbd client are all connected with the ceph distributed storage platform.
S11, accessing and opening the rbd file of the ceph distributed storage platform by the local rbd client;
s12, creating an rbd file of the ceph distributed storage platform by the remote rbd client, and storing the received data stream;
s13, establishing tcp long connection between the local rbd client and the remote rbd client;
s14, reading the rbd file of the ceph distributed storage platform by the local rbd client, compressing the rbd file and transmitting the compressed rbd file to the remote rbd client;
in the embodiment, an lz4 compression technology can be preferentially adopted to compress the rbd file, and then the rbd file is verified by using xxHash-32 and then transmitted to a remote rbd client;
the rbd file is compressed once and then transmitted at the same time, and a data stream of the rbd file is generated in the process of transmitting the rbd file to a remote rbd client; each rbd file can be divided into a plurality of small blocks to be transmitted to a remote rbd client side in a shunting manner, and compression and transmission are carried out simultaneously; during transmission, the Rbd file is divided into a plurality of 1M small blocks, and compression and transmission are realized at the same time.
Because the local rbd client and the allopatric rbd client establish tcp long connection, the local rbd client sends data stream to the allopatric rbd client through tcp session;
s15, decompressing the received rbd file data stream by the allopatric rbd client, and writing the data stream into the newly-created rbd file in the step S12; in step S14, if the rbd file data stream is compressed by lz4 and verified by xxHash-32, then the remote rbd client also needs to decompress the rbd file data stream by lz4 and verify by xxHash-32;
s16, repeating the steps S14-S15 until the local rbd client reads all the rbd files of the ceph distributed storage platform and completely sends the rbd files to the different-place rbd client, then the local rbd client sends an end mark to the different-place rbd client, and the different-place rbd client stops writing the rbd files of the ceph distributed storage platform after receiving the end mark;
and S17, the local rbd client and the remote rbd client close the connection with the ceph distributed storage platform, and the local rbd client and the remote rbd client disconnect tcp long connection.
Example 2
The embodiment discloses a ceph-based rbd file fast transmission system based on the method of embodiment 1, and as shown in fig. 2, the system comprises a local rbd client, an allopatric rbd client and a ceph distributed storage platform, wherein the local rbd client and the allopatric rbd client are both connected with the ceph distributed storage platform;
the local rbd client comprises an access module, a first connection module, a compression transmission module and an end mark generation module;
the remote rbd client comprises a new rbd file generation module, a second connection module, a receiving and decompressing module and an end mark receiving module;
the first connection module is used for connecting the ceph distributed storage platform, connecting the second connection module and establishing tcp long connection; the second connection module is also used for connecting the ceph distributed storage platform;
the access module is used for accessing and opening the rbd file of the ceph distributed storage platform and reading the rbd file of the ceph distributed storage platform;
the new rbd file generation module is used for creating an rbd file of the ceph distributed storage platform;
the compression transmission module is used for compressing the rbd file of the ceph distributed storage platform and transmitting the rbd file to the allopatric rbd client;
the receiving and decompressing module is used for receiving and decompressing the rbd file transmitted by the compression transmission module and writing the rbd file into a newly-built rbd file of the new rbd file generating module;
the end mark generating module is used for generating an end mark and sending the end mark to the end mark receiving module; after the ending mark receiving module receives an ending mark, the remote rbd client is instructed to stop receiving the rbd file of the ceph distributed storage platform; and the end mark generation module stops transmitting the rbd file of the ceph distributed storage platform after generating an end mark.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various equivalent changes, modifications, substitutions and alterations can be made herein without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims (5)
1. A method for quickly transmitting rbd files based on ceph is characterized by comprising the following steps:
s1, accessing and opening the rbd file of the ceph distributed storage platform by the local rbd client;
s2, creating an rbd file of the ceph distributed storage platform by the remote rbd client, and storing the received data stream;
s3, establishing tcp long connection between the local rbd client and the remote rbd client;
s4, reading the rbd file of the ceph distributed storage platform by the local rbd client, compressing the rbd file and transmitting the compressed rbd file to the remote rbd client;
the rbd file is compressed once and then transmitted at the same time, and a data stream of the rbd file is generated in the process of transmitting the rbd file to a remote rbd client; a local rbd client sends data flow to a foreign rbd client through a tcp session;
s5, decompressing the received rbd file data stream by the allopatric rbd client, and writing the data stream into the newly-created rbd file in the step S2;
and S6, repeating the steps S4-S5 until the local rbd client reads all the rbd files of the ceph distributed storage platform and completely sends the rbd files to the allopatric rbd client, then the local rbd client sends an end mark to the allopatric rbd client, and the allopatric rbd client stops writing the rbd files of the ceph distributed storage platform after receiving the end mark.
2. The method according to claim 1, wherein the step S1 is preceded by a step S0, in which the local rbd client and the remote rbd client are both connected to a ceph distributed storage platform.
3. The method according to claim 2, wherein after step S6, the method further comprises step S7, wherein the local rbd client and the remote rbd client both close the connection with the ceph distributed storage platform, and wherein the tcp long connection is disconnected between the local rbd client and the remote rbd client.
4. The method according to claim 1, wherein in step S4, the local rbd client reads the rbd file of the ceph distributed storage platform, performs lz4 compression on the rbd file, checks the rbd file with xxHash-32, and then transmits the rbd file to the remote rbd client;
in the step S5, the allopatric rbd client decompresses the received rbd file data stream by lz4 and verifies the data stream by using xxHash-32, and then writes the data stream into the new rbd file in the step S2.
5. A ceph-based rbd file rapid transmission system is characterized by comprising a local rbd client, a remote rbd client and a ceph distributed storage platform, wherein the local rbd client and the remote rbd client are connected with the ceph distributed storage platform;
the local rbd client comprises an access module, a first connection module, a compression transmission module and an end mark generation module;
the remote rbd client comprises a new rbd file generation module, a second connection module, a receiving and decompressing module and an end mark receiving module;
the first connection module is used for connecting the ceph distributed storage platform, connecting the second connection module and establishing tcp long connection; the second connection module is also used for connecting the ceph distributed storage platform;
the access module is used for accessing and opening the rbd file of the ceph distributed storage platform and reading the rbd file of the ceph distributed storage platform;
the new rbd file generation module is used for creating an rbd file of the ceph distributed storage platform;
the compression transmission module is used for compressing the rbd file of the ceph distributed storage platform and transmitting the rbd file to the allopatric rbd client;
the receiving and decompressing module is used for receiving and decompressing the rbd file transmitted by the compression transmission module and writing the rbd file into a newly-built rbd file of the new rbd file generating module;
the end mark generating module is used for generating an end mark and sending the end mark to the end mark receiving module; after the ending mark receiving module receives an ending mark, the remote rbd client is instructed to stop receiving the rbd file of the ceph distributed storage platform; and the end mark generation module stops transmitting the rbd file of the ceph distributed storage platform after generating an end mark.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911382609.9A CN111221792B (en) | 2019-12-27 | 2019-12-27 | Rbd file transmission method and system based on ceph |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911382609.9A CN111221792B (en) | 2019-12-27 | 2019-12-27 | Rbd file transmission method and system based on ceph |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111221792A true CN111221792A (en) | 2020-06-02 |
| CN111221792B CN111221792B (en) | 2024-01-19 |
Family
ID=70827920
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911382609.9A Active CN111221792B (en) | 2019-12-27 | 2019-12-27 | Rbd file transmission method and system based on ceph |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111221792B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113254091A (en) * | 2021-06-09 | 2021-08-13 | 广东睿江云计算股份有限公司 | Ceph block device-based diskless system starting method |
| CN113419673A (en) * | 2021-06-05 | 2021-09-21 | 济南浪潮数据技术有限公司 | Method, system, medium and device for RBD access storage cluster |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107948334A (en) * | 2018-01-09 | 2018-04-20 | 无锡华云数据技术服务有限公司 | Data processing method based on distributed memory system |
| CN108536729A (en) * | 2018-02-24 | 2018-09-14 | 国家计算机网络与信息安全管理中心 | Across the subregion image file synchronous method of one kind and device |
-
2019
- 2019-12-27 CN CN201911382609.9A patent/CN111221792B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107948334A (en) * | 2018-01-09 | 2018-04-20 | 无锡华云数据技术服务有限公司 | Data processing method based on distributed memory system |
| CN108536729A (en) * | 2018-02-24 | 2018-09-14 | 国家计算机网络与信息安全管理中心 | Across the subregion image file synchronous method of one kind and device |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113419673A (en) * | 2021-06-05 | 2021-09-21 | 济南浪潮数据技术有限公司 | Method, system, medium and device for RBD access storage cluster |
| CN113419673B (en) * | 2021-06-05 | 2022-05-27 | 济南浪潮数据技术有限公司 | Method, system, medium and device for RBD access storage cluster |
| CN113254091A (en) * | 2021-06-09 | 2021-08-13 | 广东睿江云计算股份有限公司 | Ceph block device-based diskless system starting method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111221792B (en) | 2024-01-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101039212B (en) | Fast data storage method | |
| US20210318836A1 (en) | Data compression method and apparatus | |
| CN111221792A (en) | Ceph-based rbd file transmission method and system | |
| CN101197773B (en) | Packet compressed transmitting and received de-compression method and system based on multilink | |
| KR20110025359A (en) | Block unit data compression and decompression method and apparatus thereof | |
| US20220147255A1 (en) | Method and apparatus for compressing data of storage system, device, and readable storage medium | |
| CN102088332B (en) | Method and device for expansion 1553B bus to perform file transfer from BC to RT | |
| CN102694730B (en) | Method and device for parallel processing | |
| CN102739650B (en) | File transmission system combining broadcast and television net and internet | |
| CN103503381B (en) | The method of data transmission of device redirection, Apparatus and system | |
| CN100401671C (en) | Method for transmitting files | |
| CN114024952B (en) | File compression transmission method based on DL/T698.45-2007 protocol | |
| KR101682508B1 (en) | Routing apparatus and network apparatus | |
| CN112689223A (en) | Voice playing system and method for automobile combination instrument | |
| CN114020666A (en) | DMA transmission system and method applied to NVMe SSD | |
| CN103117955A (en) | Method, device and system of message transmission | |
| CN103906007A (en) | Multimedia-message forwarding method and device | |
| CN104219537A (en) | Video data processing method, device and system | |
| WO2024002077A1 (en) | Data transmission method and multi-core chip | |
| CN104661049A (en) | Transmission method and transmission system for transport stream, streaming media server and terminal | |
| CN114416752B (en) | Data processing method and device of KV SSD | |
| CN116546461A (en) | Train section in-field batch data transmission system and method based on 5G communication technology | |
| CN117411891A (en) | Data transmission method, data transmission device and data transmission system | |
| US20220166855A1 (en) | System for development interface and data transmission method for development interface | |
| CN106790462B (en) | Short sentence transmission method and system, server, sending client and receiving client |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |