CN112181314A - Distributed storage method and system - Google Patents
Distributed storage method and system Download PDFInfo
- Publication number
- CN112181314A CN112181314A CN202011169756.0A CN202011169756A CN112181314A CN 112181314 A CN112181314 A CN 112181314A CN 202011169756 A CN202011169756 A CN 202011169756A CN 112181314 A CN112181314 A CN 112181314A
- Authority
- CN
- China
- Prior art keywords
- data
- node
- nodes
- data node
- name
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 18
- 230000007246 mechanism Effects 0.000 claims description 22
- 230000009123 feedback regulation Effects 0.000 claims description 13
- 230000005540 biological transmission Effects 0.000 claims description 5
- 230000002159 abnormal effect Effects 0.000 claims description 3
- 238000012790 confirmation Methods 0.000 claims description 3
- 230000000977 initiatory effect Effects 0.000 claims description 3
- 235000012054 meals Nutrition 0.000 claims 1
- 238000013500 data storage Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0625—Power saving in storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the field of cloud storage and distributed storage, and particularly provides a distributed storage method, which comprises the steps of firstly, establishing a data node cluster; then, judging whether the operation is normal or not, if not, recording error data information of the data nodes, and if so, recording data information of the data nodes; and finally, judging and selecting a proper data node by the name node. Compared with the prior art, the distributed storage method and the distributed storage system provided by the invention can improve the robustness and the fault tolerance of the distributed storage system, improve the overall performance of the system and optimize the data storage performance.
Description
Technical Field
The invention relates to the field of cloud storage and distributed storage, and particularly provides a distributed storage method and system.
Background
With the explosive growth of data information, people's daily life and various industries generate massive data to be managed and stored, and with the growth of data volume at every moment, a distributed storage system becomes a necessary choice for data storage. Meanwhile, new higher requirements are put on storage performance including read-write and fault tolerance. Therefore, how to efficiently and stably implement storage of mass data becomes a challenge, and how to improve robustness and fault tolerance of a distributed storage system becomes a problem to be solved urgently.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a distributed storage method with strong practicability.
The invention further aims to provide a distributed storage system which is reasonable in design, safe and applicable.
The technical scheme adopted by the invention for solving the technical problems is as follows:
a distributed storage method comprises the steps of firstly, establishing a data node cluster; then, judging whether the operation is normal or not, if not, recording error data information of the data nodes, and if so, recording data information of the data nodes; and finally, judging and selecting a proper data node by the name node.
Furthermore, the data node cluster comprises a plurality of data nodes, and the running state of each data node in the data node cluster is judged.
Further, whether each data node in the data node cluster is in a normal operation state or not is judged, the state of each data node in the data node cluster is traversed, and if the operation state of the data node is normal, correct data information of the data node is recorded and stored;
and if the running state of the data node is abnormal, recording and storing wrong data information of the data node.
Further, for the traversed data nodes, the client side sends a reading request to the name node, the name node makes corresponding judgment on the running state of the data node through a feedback regulation mechanism, and feedbacks again to perform state judgment and confirmation again, and after the judgment on the name node is completed, a proper data node is selected nearby to complete the storage work of the data information.
Further, the feedback regulation mechanism can record problem information existing in the data nodes, and gather and feed back the problem information to the name node in real time, and the data information judgment is performed twice, and the data node and the name node are subjected to data information judgment once.
A distributed storage system comprises clients, name nodes, data node clusters and a feedback regulation mechanism,
the client is used for initiating a reading request to the name node;
the name node is used for recording relevant data information of all the data node clusters and feeding back available data information of the data node clusters to the client;
the data node cluster comprises a plurality of data nodes, and adjacent data nodes are stored in a mutual transmission mode to realize multi-copy storage and backup of data;
the feedback adjusting mechanism is used for recording problem information existing in the data nodes and summarizing and feeding back the problem information to the name nodes in real time.
Furthermore, when each data node in the data node cluster is started, the data list held by the data node cluster is sent to the name node periodically through the client.
Further, the name node comprehensively judges according to the problem information, and selects an effective data node nearby by using a feedback regulation mechanism to feed back and transmit the data information.
Further, the feedback adjustment mechanism records and stores the data nodes with problems.
Compared with the prior art, the distributed storage method and the system have the following outstanding beneficial effects:
(1) the distributed storage method and the system provided by the invention can improve the robustness and the fault tolerance of the distributed storage system, improve the overall performance of the system and optimize the data storage performance.
(2) The invention improves the accuracy and the high efficiency of data reading and writing, avoids repeated judgment on the data node cluster and improves the judgment efficiency of the system.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a schematic flow diagram of a distributed storage method;
fig. 2 is a schematic structural diagram of a distributed storage system.
Detailed Description
The present invention will be described in further detail with reference to specific embodiments in order to better understand the technical solutions of the present invention. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
A preferred embodiment is given below:
as shown in fig. 1, a distributed storage method in this embodiment includes the following steps:
s01, establishing a data node cluster, wherein the data node cluster comprises 3 data nodes, and judging the running state of the 3 data nodes in the data node cluster.
S02, judging whether the 3 data nodes in the data node cluster are in a normal operation state, traversing the state of each data node in the data node cluster, and if the operation state of the data node is normal, recording and storing correct data information of the data node.
And S03, traversing the state of each data node in the data node cluster, and recording and storing the wrong data information of the data node if the running state of the data node is abnormal.
S04, for the traversed data nodes, the client side sends a reading request to the name node, the name node makes corresponding judgment on the running state of the data nodes through a feedback regulation mechanism, and feedbacks again to make the state judgment and confirmation again;
and S05, after the judgment of the name node is completed, selecting a proper data node nearby to complete the storage work of the data information.
The feedback adjusting mechanism can record problem information existing in the data nodes and summarize and feed back the problem information to the name nodes in real time. The feedback regulation mechanism comprises two data information judgments, and the data information judgments are carried out by the data nodes and the name nodes. The feedback regulation mechanism can also store the data node state and other data information, thereby avoiding repeated judgment and improving the judgment efficiency.
As shown in fig. 2, a distributed storage system includes clients, name nodes, data node clusters, and a feedback adjustment mechanism.
The client is mainly responsible for initiating a reading request to the name node and needs to store data.
The name node is a core module and a main node of the distributed storage system, does not store actual file data, but records relevant data information of all data nodes, and is mainly responsible for feeding back available data information of the data nodes to the client.
The data node cluster of this embodiment includes 3 data nodes, and the data node is auxiliary module and the slave node of distributed storage system to store actual file data, can store through the mode of mutual transmission between the adjacent data node, thereby realize many copies of storage and backup to the data. When the three data nodes are started, the name nodes issued by the three data nodes report the data list held by the three data nodes, and the information is sent to the name nodes at regular intervals.
In view of the fact that the data nodes may not be accessible or are down, in order to ensure stable and efficient transmission of the data information, the feedback adjustment mechanism is added, and for the data nodes with problems, the feedback adjustment mechanism records the problem information of the data nodes and summarizes and feeds the problem information back to the name nodes in real time. And the name node comprehensively judges according to the problem information and quickly selects other effective data nodes nearby by using a feedback regulation mode to feed back and transmit the data information. And the data nodes with problems are recorded and stored through a feedback regulation mechanism, so that repeated judgment work is avoided.
The feedback regulation mechanism comprises two data information judgments, and the data information judgments are carried out on the data nodes and the name nodes. The feedback regulation mechanism can store the data node state and other data information, thereby avoiding repeated judgment and improving the judgment efficiency.
The above embodiments are only specific examples of the present invention, and the scope of the present invention includes but is not limited to the above embodiments, and any suitable changes or substitutions that are consistent with the claims of a distributed storage method and system of the present invention and are made by those of ordinary skill in the art should fall within the scope of the present invention.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (9)
1. A distributed storage method is characterized in that firstly, a data node cluster is established; then, judging whether the operation is normal or not, if not, recording error data information of the data nodes, and if so, recording data information of the data nodes; and finally, judging and selecting a proper data node by the name node.
2. The distributed storage method according to claim 1, wherein the data node cluster includes a plurality of data nodes, and the operation state determination is performed on each data node in the data node cluster.
3. The distributed storage method according to claim 2, wherein whether each data node in the data node cluster is in a normal operation state is determined, the state of each data node in the data node cluster is traversed, and if the operation state of the data node is normal, correct data information of the data node is recorded and stored;
and if the running state of the data node is abnormal, recording and storing wrong data information of the data node.
4. The distributed storage method according to claim 3, wherein for the traversed data nodes, the client initiates a read request to the name node, the name node makes a corresponding judgment on the operating state of the data node through a feedback regulation mechanism, and performs feedback again to perform state judgment and confirmation, and after the judgment on the name node is completed, an appropriate data node is selected nearby to complete the storage work of the data information.
5. The distributed storage method according to claim 4, wherein the feedback adjustment mechanism records problem information existing in the data nodes, and summarizes the problem information in real time and feeds the problem information back to the name node, and the method includes two data information determinations, and the data information determination is performed by both the data node and the name node.
6. A distributed storage system is characterized by comprising clients, name nodes, data node clusters and a feedback regulation mechanism,
the client is used for initiating a reading request to the name node;
the name node is used for recording relevant data information of all the data node clusters and feeding back available data information of the data node clusters to the client;
the data node cluster comprises a plurality of data nodes, and adjacent data nodes are stored in a mutual transmission mode to realize multi-copy storage and backup of data;
the feedback adjusting mechanism is used for recording problem information existing in the data nodes and summarizing and feeding back the problem information to the name nodes in real time.
7. The distributed storage system according to claim 6, wherein each data node in the data node cluster sends its own data list to name node periodically through client at startup.
8. The distributed storage system according to claim 7, wherein the name node selects an effective data node nearby for feedback and transmission of data information by using a feedback adjustment mechanism according to the problem information comprehensive judgment.
9. A meal storage system according to claim 8, wherein the feedback adjustment mechanism records and saves the data node in question.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011169756.0A CN112181314A (en) | 2020-10-28 | 2020-10-28 | Distributed storage method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011169756.0A CN112181314A (en) | 2020-10-28 | 2020-10-28 | Distributed storage method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112181314A true CN112181314A (en) | 2021-01-05 |
Family
ID=73922875
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011169756.0A Pending CN112181314A (en) | 2020-10-28 | 2020-10-28 | Distributed storage method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112181314A (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019119311A1 (en) * | 2017-12-20 | 2019-06-27 | 华为技术有限公司 | Data storage method, device, and system |
| CN111274205A (en) * | 2020-01-07 | 2020-06-12 | 北京松果电子有限公司 | Data block access method and device and storage medium |
-
2020
- 2020-10-28 CN CN202011169756.0A patent/CN112181314A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019119311A1 (en) * | 2017-12-20 | 2019-06-27 | 华为技术有限公司 | Data storage method, device, and system |
| CN111274205A (en) * | 2020-01-07 | 2020-06-12 | 北京松果电子有限公司 | Data block access method and device and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 肖君: "《教育大数据》", 30 January 2020 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP2474919B1 (en) | System and method for data replication between heterogeneous databases | |
| US9785498B2 (en) | Archival storage and retrieval system | |
| US9747301B2 (en) | Distributed file system using consensus nodes | |
| EP1625502B1 (en) | Redundant data assigment in a data storage system | |
| EP1625501B1 (en) | Read, write, and recovery operations for replicated data | |
| CN101655813B (en) | Storage system | |
| CN101334797B (en) | Distributed file systems and its data block consistency managing method | |
| CN100458721C (en) | Method for automatic backup using computer network | |
| US9424272B2 (en) | Distributed file system using consensus nodes | |
| CN111630826A (en) | Consensus system and method | |
| CN101079896B (en) | A method for constructing multi-availability mechanism coexistence framework of concurrent storage system | |
| CN101751415B (en) | Metadata service system, metadata synchronized method and writing server updating method | |
| CN101137984B (en) | Systems, methods, and software for distributed loading of databases | |
| CN113987064A (en) | Data processing method, system and equipment | |
| CN112039970B (en) | Distributed business lock service method, server, system and storage medium | |
| CN113905054B (en) | RDMA (remote direct memory access) -based Kudu cluster data synchronization method, device and system | |
| CN114385755A (en) | Distributed storage system | |
| CN103544081A (en) | Management method and device for double metadata servers | |
| CN112181314A (en) | Distributed storage method and system | |
| CN109815207A (en) | Date storage method and Client Agent | |
| CN113204424A (en) | Method and device for optimizing Raft cluster and storage medium | |
| CN114330698B (en) | Neural model storage system and method of brain-like computer operating system | |
| CN114741367B (en) | Data synchronization method, data synchronization device, electronic equipment, storage medium and program product | |
| CN112667698B (en) | MongoDB data synchronization method based on converged media platform | |
| CN112256202B (en) | Distributed storage system and method for deleting volumes in distributed storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210105 |