CN116112511A - Distributed storage system based on multiple gateways - Google Patents
Distributed storage system based on multiple gateways Download PDFInfo
- Publication number
- CN116112511A CN116112511A CN202211697253.XA CN202211697253A CN116112511A CN 116112511 A CN116112511 A CN 116112511A CN 202211697253 A CN202211697253 A CN 202211697253A CN 116112511 A CN116112511 A CN 116112511A
- Authority
- CN
- China
- Prior art keywords
- gateway
- file
- distributed storage
- storage
- storage system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 210000001503 joint Anatomy 0.000 claims description 8
- 230000002159 abnormal effect Effects 0.000 claims description 2
- 238000000034 method Methods 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 7
- 238000002955 isolation Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
- H04L41/0663—Performing the actions predefined by failover planning, e.g. switching to standby network elements
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention relates to a distributed storage system based on multiple gateways, which aims at the problems of mixed complex demand environment, cloud platform block storage demand and file demand of different service systems at the upper layer of a cloud platform, and designs a distributed storage architecture deployed by multiple blocks and files. Each file gateway group comprises 2 gateway nodes of the main and the standby, so that when the main node fails, the service can be switched to the standby node to operate, and the high availability of the gateway group is realized. Compared with the prior art, the method has the advantages of high stability and safety, high availability of the file gateway group and the like.
Description
Technical Field
The invention relates to the field of servers, in particular to a distributed storage system based on multiple gateways.
Background
The development of information technology brings new challenges to the IT architecture of each enterprise while facilitating the daily life of people. The data is the most core asset of each enterprise, the storage of the data is the most important in the IT construction process of the enterprise, the application scene of an enterprise system is gradually diversified under the background of a cloud data center, unstructured data is rapidly increased, and new requirements are put forward for the storage construction of the enterprise.
Today's enterprise business systems no longer only need a single SAN/NAS storage service, and typically need multiple protocol storage support such as RBD, NFS, CIFS, S, SWIFT, etc., which can meet different types of access requirements for front-end business blocks, files, objects, etc. In addition, with the rapid expansion of services and the rapid increase of data volume, the underlying storage system often needs to meet agile deployment and linear capacity expansion requirements. The traditional storage architecture has strong software and hardware coupling, needs to purchase professional storage equipment, has long deployment period, generally only supports single SAN/NAS storage service, has poor linear expansion capability and obvious upper performance limit, and therefore has obvious limitation in application to the present enterprise architecture.
In summary, there is currently a lack of a storage system to solve the problem of low storage system stability when the conventional storage architecture provides file services for a business system.
Disclosure of Invention
The present invention is directed to a distributed storage system based on multiple gateways, so as to solve or partially solve the problem of low storage system stability when the conventional storage architecture provides file services for a business system.
The aim of the invention can be achieved by the following technical scheme:
the invention provides a distributed storage system based on multiple gateways, which comprises:
a plurality of physical hard disks, the plurality of physical hard disks being partitioned into a plurality of storage pools, the plurality of storage pools including at least one file storage pool;
and the file gateway groups are matched with the file storage pool and are in butt joint with the cloud platform and are used for responding to file requests from the cloud platform, each file gateway group comprises a main gateway and at least one standby gateway, the main gateway provides service when the main gateway is normal, the file gateway groups continuously acquire the state information of the main gateway, and the standby gateway is switched to and provides service when the main gateway is abnormal.
As an optimal technical scheme, the plurality of physical hard disks are deployed on the rack server.
As a preferred technical solution, the plurality of storage pools further includes at least one block storage pool, and the distributed storage cluster further includes a block storage gateway matched with the block storage pool, for responding to the block request of the cloud platform.
As an optimal technical scheme, the block storage gateway is in butt joint with the cloud platform cander through a SCSI protocol.
As a preferable technical scheme, the number of the file gateway groups is the same as the number of the file storage pools.
As a preferred technical solution, each of the file gateway groups includes a primary gateway and a standby gateway.
As an preferable technical scheme, the file gateway group is in butt joint with the cloud platform through NFS and/or CIFS protocols.
As an preferable technical scheme, each file gateway group is respectively in one-to-one butt joint with a service system deployed on the cloud platform.
As a preferable technical scheme, the distributed storage cluster realizes data protection by using erasure codes.
As a preferable technical scheme, the number of the file gateway groups is 1-5.
Compared with the prior art, the invention has the following advantages:
(1) Through configuration file gateway groups, each file gateway group comprises a main gateway and a standby gateway, the file gateway groups adopt main-standby redundancy configuration, a continuous activity detection mechanism exists between the main-standby gateways, and the service can be switched to the standby gateway nodes in time when the main gateway nodes are in failure, so that the problem that the stability of a storage system is lower when the traditional storage architecture provides file service for a service system is solved or partially solved, the high availability of the file gateway groups is realized, and the stability of the system is improved.
(2) By enabling the file gateway group to be in one-to-one butt joint with service systems deployed on the cloud platform, isolation of data traffic among different service systems is achieved, and safety of the system is improved.
(3) The storage cluster can provide various types of storage services, provides the block storage services for the cloud platform through butt joint of the block storage gateway and the cloud platform cander, and provides the file services for the business system through the file gateway.
(4) And the software and hardware decoupling is realized by adopting distributed architecture deployment.
Drawings
FIG. 1 is a schematic diagram of the overall architecture of a distributed storage system based on multiple gateways in embodiment 1;
fig. 2 is a schematic diagram of a file request of a service system 01 in embodiment 1, where (a) is a schematic diagram under normal conditions and (b) is a schematic diagram when a primary gateway fails;
fig. 3 is a schematic diagram of a file request of the service system 02 in embodiment 1, where (a) is a schematic diagram under normal conditions and (b) is a schematic diagram when the primary gateway fails.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
Example 1
As shown in fig. 1-3, the present embodiment designs a distributed storage system based on multiple gateways for front-end service block storage requirements and file storage requirements of multiple systems. Mainly comprises the following concepts:
(1) distributed storage clusters: the storage cluster of the distributed architecture deployed based on the universal rack-mounted server can simultaneously provide various types of storage services to the outside.
(2) OSD: each physical hard disk corresponds to an OSD process, which is a unit for actually storing data for the storage cluster.
(3) And (3) a storage pool: there is a logical resource pool consisting of multiple OSDs.
(4) Block gateway: running the Block component may provide an interface based on the SCSI protocol.
(5) File gateway: the file service is deployed, and the interface of NFS and CIFS protocols can be provided.
(6) File gateway group: the gateway in the gateway group is in redundancy relation.
The distributed storage adopts a distributed architecture, can be deployed by using a rack-mounted server, has small dependence on hardware, realizes data protection by adopting a multi-copy or erasure code mode at the bottom layer, and can ensure the consistency and the safety of data. The method is applied to a cloud platform scene, can realize calculation separation, reduces the load of a front-end platform, and can realize linear capacity expansion along with the expansion of the machine scale. The storage requirements of the traditional blocks and files are met, and the storage interfaces of the objects such as S3, switf and the like are also supported; the method can be applied to traditional databases and file sharing scenes, and also can be applied to clouds, so that various cloud storage services are realized.
As shown in fig. 1, the system deploys a distributed storage cluster by using a rack server, deploys a group of block storage gateways for the cluster, and provides a block storage service by interfacing with a cloud platform cander through a SCSI protocol. Four file gateways are deployed respectively: gateway a, gateway B, gateway C, gateway D. According to the file requirement of the front-end class 2 service system, a gateway group is created by using a gateway A and a gateway B respectively: a file gateway group 01; creating a gateway group using gateway C and gateway D: file gateway group 02. 2 gateways in the gateway group are in a primary-backup relationship, the primary-backup gateways can continuously detect activity, and when the primary gateway fails, the service can be switched to the backup gateway to operate. Creating 3 storage pools: block storage pool, file storage pool 01, file storage pool 02. The block storage pool provides block storage service for the cloud platform through the block gateway; the file storage pool 01 provides file service for the business system 01 through the file gateway group 01; the file storage pool 02 provides file services to the business system 02 through the file gateway group 02. The block storage request of the cloud platform is executed through the block storage gateway to the block storage pool. The types and the number of the gateway nodes can be flexibly deployed according to the service requirements.
As described in fig. 2 (a), the file request of the service system 01 is performed by default through the gateway a of the file gateway group 01 to the file storage pool 01. When gateway a fails, as described in fig. 2 (B), the traffic will switch to gateway B operation.
As described in fig. 3 (a), the file request of the service system 02 is performed by default through the gateway C of the file gateway group 02 to the file storage pool 02. When gateway C fails, as described in fig. 3 (b), the traffic will switch to gateway D operation.
The embodiment provides a storage scheme based on a distributed architecture and adopting multi-gateway hybrid deployment, and through flexible gateway deployment, not only the block storage requirement of a cloud platform is met, but also the front-end service file and object storage requirement is met, and according to the front-end file service isolation requirement, the isolation of the flow and the load of the front-end file service is successfully realized by adopting a mode of newly adding a file storage gateway group.
This embodiment has the following advantages:
1. and the software and hardware decoupling is realized by adopting distributed architecture deployment.
2. The cluster can provide various types of storage services, specifically, the cluster can provide the block storage services for the cloud platform through butt joint of the block storage gateway and the cloud platform cander, and the file service for the business system through the file gateway.
3. The invention configures the file gateway groups, each file gateway group comprises 2 file gateways, the 2 file gateways adopt main-standby redundancy configuration, a continuous activity detection mechanism exists between the main-standby gateways, the service can be ensured to be switched to the standby gateway nodes in time when the main gateway node fails, and the high availability of the file gateway groups is realized.
4. According to the requirements of the class 2 service systems, the invention provides a configuration of 2 file gateway groups, realizes isolation of data traffic and load pressure sharing among different service systems, and improves the stability and safety of the system.
While the invention has been described with reference to certain preferred embodiments, it will be understood by those skilled in the art that various changes and substitutions of equivalents may be made and equivalents will be apparent to those skilled in the art without departing from the scope of the invention. Therefore, the protection scope of the invention is subject to the protection scope of the claims.
Claims (10)
1. A multi-gateway based distributed storage system, comprising:
a plurality of physical hard disks, the plurality of physical hard disks being partitioned into a plurality of storage pools, the plurality of storage pools including at least one file storage pool;
and the file gateway groups are matched with the file storage pool and are in butt joint with the cloud platform and are used for responding to file requests from the cloud platform, each file gateway group comprises a main gateway and at least one standby gateway, the main gateway provides service when the main gateway is normal, the file gateway groups continuously acquire the state information of the main gateway, and the standby gateway is switched to and provides service when the main gateway is abnormal.
2. The multiple gateway-based distributed storage system of claim 1, wherein the plurality of physical hard disks are deployed on a rack server.
3. The multi-gateway based distributed storage system of claim 1, wherein said plurality of storage pools further comprises at least one chunk storage pool, said distributed storage cluster further comprising a chunk storage gateway matching said chunk storage pool for responding to chunk requests of said cloud platform.
4. A distributed storage system according to claim 3, wherein said block storage gateway interfaces with said cloud platform cander via SCSI protocol.
5. The multi-gateway based distributed storage system of claim 1, wherein the number of file gateway groups is the same as the number of file storage pools.
6. The multi-gateway based distributed storage system of claim 1, wherein each of said file gateway groups comprises a primary gateway and a backup gateway.
7. The distributed storage system of claim 1, wherein the file gateway group interfaces with the cloud platform via NFS and/or CIFS protocols.
8. The distributed storage system according to claim 1, wherein each of the file gateway groups interfaces with a service system deployed on the cloud platform one by one.
9. The multi-gateway based distributed storage system of claim 1, wherein the distributed storage clusters implement data protection by using erasure codes.
10. The multiple gateway-based distributed storage system of claim 1, wherein the number of file gateway groups is 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211697253.XA CN116112511A (en) | 2022-12-28 | 2022-12-28 | Distributed storage system based on multiple gateways |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211697253.XA CN116112511A (en) | 2022-12-28 | 2022-12-28 | Distributed storage system based on multiple gateways |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116112511A true CN116112511A (en) | 2023-05-12 |
Family
ID=86257371
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211697253.XA Pending CN116112511A (en) | 2022-12-28 | 2022-12-28 | Distributed storage system based on multiple gateways |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116112511A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117041321A (en) * | 2023-09-28 | 2023-11-10 | 宁波银行股份有限公司 | Service processing system and method |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120079313A1 (en) * | 2010-09-24 | 2012-03-29 | Honeywell International Inc. | Distributed memory array supporting random access and file storage operations |
| CN106034164A (en) * | 2016-05-16 | 2016-10-19 | 深圳元核云技术有限公司 | Cloud storage gateway file sharing service method and system |
| WO2019026222A1 (en) * | 2017-08-03 | 2019-02-07 | 株式会社日立製作所 | Storage system and data transfer control method |
| CN111158587A (en) * | 2019-12-10 | 2020-05-15 | 南京道熵信息技术有限公司 | Distributed storage system based on storage pool virtualization management and data read-write method |
| CN113282563A (en) * | 2021-05-28 | 2021-08-20 | 深圳市杉岩数据技术有限公司 | File gateway high availability implementation method based on object storage and electronic equipment |
| CN114116285A (en) * | 2021-11-22 | 2022-03-01 | 深圳市杉岩数据技术有限公司 | Processing method, device and medium for file gateway fault switching |
| CN115442367A (en) * | 2022-08-23 | 2022-12-06 | 中国银联股份有限公司 | Data transmission method, virtual private cloud, device, medium and product |
-
2022
- 2022-12-28 CN CN202211697253.XA patent/CN116112511A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120079313A1 (en) * | 2010-09-24 | 2012-03-29 | Honeywell International Inc. | Distributed memory array supporting random access and file storage operations |
| CN106034164A (en) * | 2016-05-16 | 2016-10-19 | 深圳元核云技术有限公司 | Cloud storage gateway file sharing service method and system |
| WO2019026222A1 (en) * | 2017-08-03 | 2019-02-07 | 株式会社日立製作所 | Storage system and data transfer control method |
| CN111158587A (en) * | 2019-12-10 | 2020-05-15 | 南京道熵信息技术有限公司 | Distributed storage system based on storage pool virtualization management and data read-write method |
| CN113282563A (en) * | 2021-05-28 | 2021-08-20 | 深圳市杉岩数据技术有限公司 | File gateway high availability implementation method based on object storage and electronic equipment |
| CN114116285A (en) * | 2021-11-22 | 2022-03-01 | 深圳市杉岩数据技术有限公司 | Processing method, device and medium for file gateway fault switching |
| CN115442367A (en) * | 2022-08-23 | 2022-12-06 | 中国银联股份有限公司 | Data transmission method, virtual private cloud, device, medium and product |
Non-Patent Citations (1)
| Title |
|---|
| AP;: "公共云存储服务性能优化", 电脑知识与技术(经验技巧), no. 11, 5 November 2017 (2017-11-05) * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117041321A (en) * | 2023-09-28 | 2023-11-10 | 宁波银行股份有限公司 | Service processing system and method |
| CN117041321B (en) * | 2023-09-28 | 2023-12-26 | 宁波银行股份有限公司 | Service processing system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7539735B2 (en) | Multi-session no query restore | |
| US6665812B1 (en) | Storage array network backup configuration | |
| US7318095B2 (en) | Data fail-over for a multi-computer system | |
| CN109828868B (en) | Data storage method, device, management equipment and double-active data storage system | |
| US9785691B2 (en) | Method and apparatus for sequencing transactions globally in a distributed database cluster | |
| US8655851B2 (en) | Method and system for performing a clean file lock recovery during a network filesystem server migration or failover | |
| US8938604B2 (en) | Data backup using distributed hash tables | |
| US7370083B2 (en) | System and method for providing virtual network attached storage using excess distributed storage capacity | |
| US8732520B2 (en) | Clustered array controller for global redundancy in a SAN | |
| US9558075B2 (en) | Synthetic full backup generation | |
| US6915448B2 (en) | Storage disk failover and replacement system | |
| US8856091B2 (en) | Method and apparatus for sequencing transactions globally in distributed database cluster | |
| US7941455B2 (en) | Notification for a distributed file system | |
| US20020196744A1 (en) | Path discovery and mapping in a storage area network | |
| US20020188697A1 (en) | A method of allocating storage in a storage area network | |
| KR101670343B1 (en) | Method, device, and system for peer-to-peer data replication and method, device, and system for master node switching | |
| CN102708165A (en) | Method and device for processing files in distributed file system | |
| US20140032960A1 (en) | Information processing system and access control method | |
| CN103838755A (en) | Remote heterogeneous disaster tolerant system of database | |
| CN116112511A (en) | Distributed storage system based on multiple gateways | |
| EP1785834A2 (en) | Storage system and method of controlling a storage system | |
| CN112148219A (en) | Design method and device for ceph type distributed storage cluster | |
| US20050193021A1 (en) | Method and apparatus for unified storage of data for storage area network systems and network attached storage systems | |
| US11496547B2 (en) | Storage system node communication | |
| CN111813332A (en) | High-performance, high-expansion and high-safety intelligent distributed storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |