CN112783669A - Distributed storage management method and system - Google Patents
Distributed storage management method and system Download PDFInfo
- Publication number
- CN112783669A CN112783669A CN202110014809.XA CN202110014809A CN112783669A CN 112783669 A CN112783669 A CN 112783669A CN 202110014809 A CN202110014809 A CN 202110014809A CN 112783669 A CN112783669 A CN 112783669A
- Authority
- CN
- China
- Prior art keywords
- storage
- nodes
- management method
- distributed
- distributed storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title claims abstract description 25
- 230000007246 mechanism Effects 0.000 claims description 4
- 238000000034 method Methods 0.000 abstract description 5
- 230000015556 catabolic process Effects 0.000 abstract description 2
- 230000001360 synchronised effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Abstract
The invention provides a distributed storage management method and a distributed storage management system. The method comprises the following steps: deploying message queues among different storage nodes; and synchronizing the data update of the storage node to other storage nodes in the storage system through the deployed message queue. The distributed storage management method and the distributed storage management system provided by the invention can improve the availability of the system and avoid the problems of single-point failure and continuous breakdown.
Description
Technical Field
The present invention relates to the field of distributed system technologies, and in particular, to a distributed storage management method and system.
Background
With the rapid development of the mobile internet and the explosive growth of business data, businesses of enterprises increasingly depend on accurate data information, and therefore, the position of a storage system in an enterprise information system is increasingly highlighted. At present, the data storage mode of a storage system is mainly divided into: centralized storage and distributed storage. Correspondingly, the storage management software is also divided into centralized storage management software and distributed storage management software, and all necessary interface operations of storage resource management, monitoring, reporting, hierarchical storage, performance management, disaster recovery and service continuity are provided for users. The distributed storage management software runs on a plurality of device nodes of distributed storage, in order to ensure the continuity of services, different High Availability (HA) schemes are adopted in many occasions, and the HA scheme aims to construct reliable, safe, extensible and easily maintained storage management software so as to ensure the stable running of a system and avoid single-point failure.
At present, a high availability scheme of distributed storage management software mainly adopts a method of transferring stateless services, that is, when a certain node fails, services running on the node are transferred to other nodes running normally in a distributed storage cluster. Highly available solutions using the above method typically include two parts: IP address drift and service data synchronous backup. The IP address drifting technology is to transfer the IP address of a failure node to a normally operating node or configure a virtual IP, the virtual IP drifts among a plurality of nodes in distributed storage according to the health state of the node, and the virtual IP always points to the normally operating node, so that the continuous operation of the service is ensured. The service data synchronous backup technology comprises two modes: software synchronization and hardware synchronization. And the software synchronization utilizes the synchronization function of the database to synchronize the service data to the data of different nodes in real time. The hardware backup adopts a storage backup function, and data is stored on different physical disks at random.
The highly available solution for distributed storage management software to employ stateless service migration may include the following disadvantages: node idleness problems and continuous crash problems. The distributed storage management software runs on a plurality of nodes of the distributed storage, and when one node fails, the service running on the node is transferred to other nodes which normally run. Before the failure, the clock of the transferred node is in an idle state, and the service resource of the node is not fully utilized. If the failed node crashes because the running task is too heavy, the standby node will also crash with the same problem after transferring to the standby node.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a distributed storage management method and system, which can improve the availability of the system and avoid the problems of single-point failure and continuous breakdown.
In order to solve the above technical problem, the present invention provides a distributed storage management method, where the method includes: deploying message queues among different storage nodes; and synchronizing the data update of the storage node to other storage nodes in the storage system through the deployed message queue.
In some embodiments, the synchronization of the data updates comprises: real-time synchronization and timing synchronization.
In some embodiments, further comprising: when a part of storage nodes in the storage system have faults, the service node is set as the storage node of which the storage system has no faults.
In some embodiments, further comprising: and distributing the access load to each storage node in the storage system through a load balancing mechanism.
In some embodiments, deploying message queues between different storage nodes includes: message queues are deployed between adjacent storage nodes of a ring storage system.
In some embodiments, deploying message queues between different storage nodes includes: message queues are deployed between adjacent storage nodes of the star storage system.
In some embodiments, deploying message queues between different storage nodes includes: and deploying the message queue between any two storage nodes in the storage system.
In addition, the present invention also provides a distributed storage management system, including: a plurality of different storage nodes, with message queues deployed between the plurality of different storage nodes, the distributed storage management system being configured to perform the distributed storage management method according to the foregoing.
After adopting such design, the invention has at least the following advantages:
1. improve equipment's utilization ratio, avoid the node idle:
the operation tasks of the users are distributed to a plurality of nodes by using a message queue mechanism, so that the utilization rate of the equipment is improved;
2. the availability of the system is improved, and single-point failure and continuous running are avoided:
and averagely transferring the service on the failed node to other running nodes by using a message queue mechanism.
Drawings
The foregoing is only an overview of the technical solutions of the present invention, and in order to make the technical solutions of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and the detailed description.
Fig. 1 is a system architecture diagram of distributed storage management software provided by an embodiment of the present invention.
Detailed Description
The preferred embodiments of the present invention will be described in conjunction with the accompanying drawings, and it will be understood that they are described herein for the purpose of illustration and explanation and not limitation.
Referring to fig. 1, a distributed storage system according to an embodiment of the present invention includes a plurality of different storage nodes. These storage nodes may be, for example, IP a nodes, IP B nodes, or IP C nodes shown in fig. 1. The storage data in the distributed storage system is distributed over these different storage nodes.
In the embodiment of the invention, message queues are deployed among different storage nodes. These message queues may be, for example, Redis message queues. The function of these message queues is to synchronize data updates on the storage nodes to other storage nodes in the storage system.
The deployment positions of the message queues are among different storage nodes. If the storage nodes in the storage system are in a star configuration, the message queues may be deployed between different network nodes in the star overall architecture. If the storage nodes in the storage system are ring structured, the message queues are deployed between adjacent nodes of the ring structure.
There is also a special case where the above-mentioned message queue is deployed between any two storage nodes of the storage system.
An advantage of deploying message queues between different storage nodes is that the overall availability of the storage system can be increased. Because any data update is synchronized to other storage nodes in the system, if part of nodes in the system fail, the system can continue to provide service to the outside by depending on the nodes without failure, and the system does not need to be stopped for maintenance.
In addition, any data updating is synchronous in real time, and any node in the system can independently provide service for users, so that the service load actually borne by each node is greatly reduced, the equipment utilization rate of a single node is improved, and the idle condition of the node is avoided.
Of course, if a certain load balancing strategy is matched, the technical effect of improving the utilization rate of the equipment is more obvious.
In addition, the data synchronization mode through the message queue may be: real-time synchronization, or timing synchronization.
Real-time synchronization refers to synchronization to other storage nodes as soon as a data update occurs locally. The timing synchronization means that even if data update occurs locally, update to other storage nodes is not performed, and all data update occurring in a period of time is performed in batch synchronization to a fixed time point.
The invention also provides a distributed storage management method. Specifically, the method comprises the following steps: deploying message queues among different storage nodes; and synchronizing the data update of the storage node to other storage nodes in the storage system through the deployed message queue.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the present invention in any way, and it will be apparent to those skilled in the art that the above description of the present invention can be applied to various modifications, equivalent variations or modifications without departing from the spirit and scope of the present invention.
Claims (8)
1. A distributed storage management method, comprising:
deploying message queues among different storage nodes;
and synchronizing the data update of the storage node to other storage nodes in the storage system through the deployed message queue.
2. The distributed storage management method of claim 1, wherein synchronizing data updates comprises: real-time synchronization and timing synchronization.
3. The distributed storage management method according to claim 1, further comprising:
when a part of storage nodes in the storage system have faults, the service node is set as the storage node of which the storage system has no faults.
4. The distributed storage management method according to claim 1, further comprising:
and distributing the access load to each storage node in the storage system through a load balancing mechanism.
5. The distributed storage management method of claim 1, wherein deploying message queues among different storage nodes comprises:
message queues are deployed between adjacent storage nodes of a ring storage system.
6. The distributed storage management method of claim 1, wherein deploying message queues among different storage nodes comprises:
message queues are deployed between adjacent storage nodes of the star storage system.
7. The distributed storage management method of claim 1, wherein deploying message queues among different storage nodes comprises:
and deploying the message queue between any two storage nodes in the storage system.
8. A distributed storage management system, comprising:
a plurality of different storage nodes having message queues deployed therebetween, the distributed storage management system being configured to perform the distributed storage management method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110014809.XA CN112783669A (en) | 2021-01-06 | 2021-01-06 | Distributed storage management method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110014809.XA CN112783669A (en) | 2021-01-06 | 2021-01-06 | Distributed storage management method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112783669A true CN112783669A (en) | 2021-05-11 |
Family
ID=75755902
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110014809.XA Pending CN112783669A (en) | 2021-01-06 | 2021-01-06 | Distributed storage management method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112783669A (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103312624A (en) * | 2012-03-09 | 2013-09-18 | 腾讯科技（深圳）有限公司 | Message queue service system and method |
| CN104980519A (en) * | 2015-06-29 | 2015-10-14 | 北京奇虎科技有限公司 | Multi-computer room storage system |
| CN105791015A (en) * | 2016-03-09 | 2016-07-20 | 浪潮通信信息系统有限公司 | High-availability information transmission channel and high-availability information transmission method for aiming at distributed alarm processing system |
| CN107135274A (en) * | 2017-06-20 | 2017-09-05 | 郑州云海信息技术有限公司 | The memory management method and device of a kind of distributed cluster system |
| CN107315641A (en) * | 2017-05-27 | 2017-11-03 | 杭州铭师堂教育科技发展有限公司 | Message queue high-availability system and method based on Qconf |
| CN107479829A (en) * | 2017-08-03 | 2017-12-15 | 杭州铭师堂教育科技发展有限公司 | A kind of Redis cluster mass datas based on message queue quickly clear up system and method |
| CN108845867A (en) * | 2018-07-16 | 2018-11-20 | 郑州云海信息技术有限公司 | A kind of distributed transaction management method, apparatus, system and storage medium |
| CN110990113A (en) * | 2019-10-31 | 2020-04-10 | 北京浪潮数据技术有限公司 | Storage management method, system and related device in storage cluster |
| CN111343251A (en) * | 2020-02-16 | 2020-06-26 | 苏州浪潮智能科技有限公司 | Message queue service deployment method and device |
| CN111488395A (en) * | 2020-04-02 | 2020-08-04 | 中国船舶重工集团公司第七一六研究所 | Double-node high-availability distributed storage system |
-
2021
- 2021-01-06 CN CN202110014809.XA patent/CN112783669A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103312624A (en) * | 2012-03-09 | 2013-09-18 | 腾讯科技（深圳）有限公司 | Message queue service system and method |
| CN104980519A (en) * | 2015-06-29 | 2015-10-14 | 北京奇虎科技有限公司 | Multi-computer room storage system |
| CN105791015A (en) * | 2016-03-09 | 2016-07-20 | 浪潮通信信息系统有限公司 | High-availability information transmission channel and high-availability information transmission method for aiming at distributed alarm processing system |
| CN107315641A (en) * | 2017-05-27 | 2017-11-03 | 杭州铭师堂教育科技发展有限公司 | Message queue high-availability system and method based on Qconf |
| CN107135274A (en) * | 2017-06-20 | 2017-09-05 | 郑州云海信息技术有限公司 | The memory management method and device of a kind of distributed cluster system |
| CN107479829A (en) * | 2017-08-03 | 2017-12-15 | 杭州铭师堂教育科技发展有限公司 | A kind of Redis cluster mass datas based on message queue quickly clear up system and method |
| CN108845867A (en) * | 2018-07-16 | 2018-11-20 | 郑州云海信息技术有限公司 | A kind of distributed transaction management method, apparatus, system and storage medium |
| CN110990113A (en) * | 2019-10-31 | 2020-04-10 | 北京浪潮数据技术有限公司 | Storage management method, system and related device in storage cluster |
| CN111343251A (en) * | 2020-02-16 | 2020-06-26 | 苏州浪潮智能科技有限公司 | Message queue service deployment method and device |
| CN111488395A (en) * | 2020-04-02 | 2020-08-04 | 中国船舶重工集团公司第七一六研究所 | Double-node high-availability distributed storage system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7702947B2 (en) | System and method for enabling site failover in an application server environment | |
| CN108259175B (en) | Distributed password service method and system | |
| EP3210367B1 (en) | System and method for disaster recovery of cloud applications | |
| EP2224341B1 (en) | Node system, server switching method, server device, and data transfer method | |
| US20130212205A1 (en) | True geo-redundant hot-standby server architecture | |
| CN102664747B (en) | Cloud calculating platform system | |
| CN103294701B (en) | A kind of method that distributed file system and data process | |
| CN101207517B (en) | Method for reliability maintenance of distributed enterprise service bus node | |
| CN102394914A (en) | Cluster brain-split processing method and device | |
| CN103312809A (en) | Distributed management method for service in cloud platform | |
| CN105630589A (en) | Distributed process scheduling system and process scheduling and execution method | |
| CN103036719A (en) | Cross-regional service disaster method and device based on main cluster servers | |
| CN111949444A (en) | Data backup and recovery system and method based on distributed service cluster | |
| CN108469996A (en) | A kind of system high availability method based on auto snapshot | |
| CN111800484B (en) | Service anti-destruction replacing method for mobile edge information service system | |
| CN111935244B (en) | Service request processing system and super-integration all-in-one machine | |
| CN112783694B (en) | Long-distance disaster recovery method for high-availability Redis | |
| CN111045602A (en) | Cluster system control method and cluster system | |
| KR101358995B1 (en) | Method and system for managing high availability | |
| CN112052127A (en) | Data synchronization method and device for dual-computer hot standby environment | |
| CN112783669A (en) | Distributed storage management method and system | |
| CN111083074A (en) | High availability method and system for main and standby dual OSPF state machines | |
| CN110677288A (en) | Edge computing system and method generally used for multi-scene deployment | |
| CN104503871A (en) | Implementation method based on full-redundancy model of small computer system | |
| JP2010231257A (en) | High availability system and method for handling failure of high availability system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |