CN106776952B - Date storage method in a kind of distributed system - Google Patents
Date storage method in a kind of distributed system Download PDFInfo
- Publication number
- CN106776952B CN106776952B CN201611097372.6A CN201611097372A CN106776952B CN 106776952 B CN106776952 B CN 106776952B CN 201611097372 A CN201611097372 A CN 201611097372A CN 106776952 B CN106776952 B CN 106776952B
- Authority
- CN
- China
- Prior art keywords
- node
- storage
- grouping
- data
- risk
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present invention relates to date storage methods in a kind of distributed system, according to memory capacity, operation load and failure-frequency calculate the storage value-at-risk for determining each node, multiple nodes are grouped, it include a higher node of storage value-at-risk and a lower node of storage value-at-risk in each grouping, so that the average storage value-at-risk relative equilibrium in each grouping, the case where several nodes for avoiding redundant data distribution are the node of Frequent Troubles, redundant data is distributed in the grouping of storage risk balance in this way, it facilitates data maintenance and reduces loss of data risk.
Description
[technical field]
The invention belongs to date storage methods under field of data storage more particularly to distributed system environment.
[background technique]
Under normal circumstances, the highest of handling up that a machine externally provides also can only achieve 200MBps, according to common machine
The completely the same mode of data on device mirror image, that is, several machines, the data for repairing 12TB need time-consuming more than 20 a hours, consider
To normal service pressure, the time of reparation is up to tens hours.
Distributed storage mode is proposed in the prior art, is multiple libraries by data cutting, and replicate several parts of redundancies, it is same
The different redundant distributions in library are stored in the different location of different machines, improve reparation speed.But in the prior art by redundancy
Data distribution is when on different machines, using random fashion, is not selected different machines, causes identical data
The machine of distribution all may be Frequent Troubles machine, therefore causes data maintenance difficulty and improve loss of data risk.
Based on the above issues, a kind of new distributed system storage method is needed now, by node according to storage value-at-risk
It is grouped, so that the average storage value-at-risk relative equilibrium of each grouping, redundant data is distributed in each grouping, it is convenient
Data maintenance and reduce loss of data risk.
[summary of the invention]
In order to solve the above problem in the prior art, the invention proposes data storage sides in a kind of distributed system
Method.
The technical solution adopted by the invention is as follows:
Date storage method in a kind of distributed system, this method comprises the following steps:
(1) m node i in distributed system is loaded according to memory capacity, operation and failure-frequency is according to following public affairs
Formula (a) calculates the storage value-at-risk R for determining each nodei:
Ri=Si×Pi+1/Fi×Qi+Gi×Ti(a)；
Wherein SiIndicate the memory capacity of node i, PiIndicate weight corresponding to memory capacity, FiIndicate the operation of node i
Highest system in history runs load, QiIndicate the corresponding weight of operation load, GiIn the history run for indicating node i
Failure-frequency, TiIndicate weight corresponding to failure-frequency, and Pi, Qi, Gi>1；
(2) m node is lined up into an ordered queue according to the sequence of storage value-at-risk from low to high, it will be in ordered queue
First node of head of the queue and the last one node of tail of the queue, which take out, constitutes the first storage grouping, the orderly team constituted for remaining node
Column continue in the manner described above constitutes subsequent multiple storage groupings, until only 2 or 3 nodes in ordered queue, then
It is grouped using above-mentioned 2 or 3 nodes as a storage, final m node is divided into k storage grouping；
(3) when system receives data storage request, data are cut into k number according to segment, to each data slot
Duplication obtains the grouping of k group data slot, includes data slot replicate data corresponding with its in every group of data slot grouping
Segment；
(4) by a data fragment packet data slot and corresponding replicate data segment be respectively stored into one and deposit
In two nodes in storage grouping, until in the grouping of k group data slot data slot and replicate data segment all store that arrive k a
In storage grouping；
(5) it when a nodes break down in storage grouping, is deposited according to another node in above-mentioned storage grouping
The data slot or replicate data segment of storage, repair the above-mentioned node to break down；
(6) one-to-one port is arranged to each node, when a node fails, the corresponding port of above-mentioned node is automatic
It closes, after the node to break down is successfully repaired, automatically turns on the corresponding port of above-mentioned node.
It include a storage value-at-risk in each grouping the beneficial effect comprise that multiple nodes are grouped
Higher node and a lower node of storage value-at-risk, so that the average storage value-at-risk relative equilibrium in each grouping,
The case where several nodes for avoiding redundant data distribution are the node of Frequent Troubles, is distributed to such storage for redundant data
In the grouping of risk balance, facilitates data maintenance and reduce loss of data risk.
[Detailed description of the invention]
Described herein the drawings are intended to provide a further understanding of the invention, constitutes part of this application, but
It does not constitute improper limitations of the present invention, in the accompanying drawings:
Fig. 1 is the structure chart of distributed system of the present invention.
Fig. 2 is the flow chart of date storage method in distributed system of the present invention.
[specific embodiment]
Come that the present invention will be described in detail below in conjunction with attached drawing and specific embodiment, illustrative examples therein and says
It is bright to be only used to explain the present invention but not as a limitation of the invention.
It is distributed system applied by the present invention referring to attached drawing 1, which includes multiple calculate nodes.
Referring to attached drawing 2, date storage method in a kind of distributed system, this method comprises the following steps:
(1) m node i in distributed system is loaded according to memory capacity, operation and failure-frequency is according to following public affairs
Formula (a) calculates the storage value-at-risk R for determining each nodei:
Ri=Si×Pi+1/Fi×Qi+Gi×Ti(a)；
Wherein SiIndicate the memory capacity of node i, PiIndicate weight corresponding to memory capacity, FiIndicate the operation of node i
Highest system in history runs load, QiIndicate the corresponding weight of operation load, GiIn the history run for indicating node i
Failure-frequency, TiIndicate weight corresponding to failure-frequency, and Pi, Qi, Gi>1；
Wherein, memory capacity, operation load and failure-frequency be influence respectively node storage value-at-risk it is different because
Element, wherein memory capacity indicates the storage capacity of a node, and memory capacity is bigger, then above-mentioned node is because data store pressure
Caused by failure risk it is lower, otherwise it is higher；The operation load of one node is bigger, then above-mentioned node is because of operation excess load institute
Caused by failure risk it is bigger, otherwise it is lower；The failure-frequency that one node occurs within the history run period is higher, then shows
A possibility that above-mentioned node breaks down in the period afterwards is higher, otherwise lower.
In one embodiment, memory capacity, operation load and failure-frequency are recorded in a table, each node
Memory capacity can refer to its hard disc of computer memory capacity, and the memory capacity of each node is recorded in table, monitor each section
Occupied system resource is run multiple times in point predetermined time cycle, and occupied most system resource conducts will be run
Highest system runs load record in above-mentioned table, monitors the failure frequency in each node predetermined time cycle, makees
It is recorded in table for failure-frequency；
In the storage value-at-risk R of calculate nodeiWhen, memory capacity corresponding to above-mentioned node is read from table, operation is born
It carries and failure-frequency is calculated according to above-mentioned formula (a).
(2) m node is lined up into an ordered queue according to the sequence of storage value-at-risk from low to high, it will be in ordered queue
First node of head of the queue and the last one node of tail of the queue, which take out, constitutes the first storage grouping, the orderly team constituted for remaining node
Column continue in the manner described above constitutes subsequent multiple storage groupings, until only 2 or 3 nodes in ordered queue, i.e.,
It is the situation of even number and odd number corresponding to m, then is grouped using above-mentioned 2 or 3 nodes as a storage, final m node point
It is grouped at k storage；
(3) when system receives data storage request, data are cut into k number according to segment, to each data slot
Duplication obtains the grouping of k group data slot, includes data slot replicate data corresponding with its in every group of data slot grouping
Segment；
(4) by a data fragment packet data slot and corresponding replicate data segment be respectively stored into one and deposit
In two nodes in storage grouping, until in the grouping of k group data slot data slot and replicate data segment all store that arrive k a
In storage grouping；
Due in each grouping include a higher node of storage value-at-risk and a lower node of storage value-at-risk,
So that the average storage value-at-risk relative equilibrium in each grouping, several nodes for avoiding redundant data distribution are failure frequency
Redundant data is distributed in the grouping of storage risk balance in this way, facilitates data maintenance and reduction by the case where node of hair
Loss of data risk.
(5) it when a nodes break down in storage grouping, is deposited according to another node in above-mentioned storage grouping
The data slot or replicate data segment of storage, repair the above-mentioned node to break down；
The speed of repair data will be greatly improved as a result, shorten repair time, it is right when multiple nodes break downs
Multiple nodes are repaired parallel, and the data slot and corresponding replicate data segment in a data fragment packet are according to random
Mode is stored into two nodes in a storage grouping.It is whole when machine quantity is greater than the quantity in the library on failed machines
The time-consuming of a repair process is usually only necessary to dozens of minutes, solves the problems, such as that data efficient is automatically repaired.
(6) one-to-one port is arranged to each node, when a node fails, the corresponding port of above-mentioned node is automatic
It closes, after the node to break down is successfully repaired, automatically turns on the corresponding port of above-mentioned node.To ensure data just
The problem of really reading, and avoiding using error listing access port.
By the above method, multiple nodes are grouped by the present invention, include storage value-at-risk in each grouping compared with
High node and a lower node of storage value-at-risk, so that the average storage value-at-risk relative equilibrium in each grouping, keeps away
Redundant data is distributed to storage wind in this way by the case where several nodes for having exempted from redundant data distribution are the node of Frequent Troubles
In the balanced grouping in danger, facilitates data maintenance and reduce loss of data risk.
The above description is only a preferred embodiment of the present invention, thus it is all according to the configuration described in the scope of the patent application of the present invention,
The equivalent change or modification that feature and principle are done, is included in the scope of the patent application of the present invention.
Claims (3)
1. date storage method in a kind of distributed system, which is characterized in that this method comprises the following steps:
(1) by m node i in distributed system according to memory capacity, operation load and failure-frequency according to the following equation (a)
Calculate the storage value-at-risk R for determining each nodei:
Ri=Si×Pi+1/Fi×Qi+Gi×Ti(a)；
Wherein SiIndicate the memory capacity of node i, PiIndicate weight corresponding to memory capacity, FiIndicate the history run of node i
In highest system run load, QiIndicate the corresponding weight of operation load, GiIndicate the failure in the history run of node i
Frequency, TiIndicate weight corresponding to failure-frequency, and Pi> 1, Qi> 1, Gi>1；
(2) m node is lined up into an ordered queue according to the sequence of storage value-at-risk from low to high, by head of the queue in ordered queue
First node and the last one node of tail of the queue, which take out, constitutes the first storage grouping, and the ordered queue constituted for remaining node is pressed
It is continued with according to aforesaid way and constitutes subsequent multiple storage groupings, until only 2 or 3 nodes in ordered queue, then it will be upper
It states 2 or 3 nodes is grouped as a storage, final m node is divided into k storage grouping；
(3) when system receives data storage request, data is cut into k number according to segment, each data slot is replicated,
The grouping of k group data slot is obtained, includes data slot replicate data segment corresponding with its in every group of data slot grouping；
(4) by a data fragment packet data slot and corresponding replicate data segment be respectively stored into one storage point
In two nodes in group, until k group data slot grouping in data slot and replicate data segment all store to k store
In grouping；
(5) when a nodes break down in storage grouping, according to another node storage in above-mentioned storage grouping
Data slot or replicate data segment repair the above-mentioned node to break down；
(6) one-to-one port is arranged to each node, when a node fails, the corresponding port of above-mentioned node is closed automatically
It closes, after the node to break down is successfully repaired, automatically turns on the corresponding port of above-mentioned node.
2. date storage method in distributed system according to claim 1, which is characterized in that when event occurs for multiple nodes
When barrier, multiple nodes are repaired parallel.
3. date storage method in distributed system according to claim 1, which is characterized in that a data fragment packet
In data slot and corresponding replicate data segment according to random fashion store to one storage grouping in two nodes in.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611097372.6A CN106776952B (en) | 2016-12-02 | 2016-12-02 | Date storage method in a kind of distributed system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611097372.6A CN106776952B (en) | 2016-12-02 | 2016-12-02 | Date storage method in a kind of distributed system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106776952A CN106776952A (en) | 2017-05-31 |
| CN106776952B true CN106776952B (en) | 2019-03-22 |
Family
ID=58884165
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611097372.6A Active CN106776952B (en) | 2016-12-02 | 2016-12-02 | Date storage method in a kind of distributed system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106776952B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110704232B (en) * | 2019-10-10 | 2023-03-14 | 广东工业大学 | Method, device and equipment for repairing failed node in distributed system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1526108A (en) * | 2001-02-14 | 2004-09-01 | 无线谷通讯有限公司 | Method and system for modeling and managing terrain, buildings, and infrastructure |
| CN102710763A (en) * | 2012-05-29 | 2012-10-03 | 新华网股份有限公司 | Method and system for pooling, slicing and fault transfer of distributed cache |
| CN102857554A (en) * | 2012-07-26 | 2013-01-02 | 福建网龙计算机网络信息技术有限公司 | Data redundancy processing method based on distributed storage system |
| CN105335384A (en) * | 2014-06-30 | 2016-02-17 | 中航商用航空发动机有限责任公司 | Storage method for monitoring data, reproduction method for monitoring data and devices |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8589637B2 (en) * | 2009-10-30 | 2013-11-19 | Cleversafe, Inc. | Concurrent set storage in distributed storage network |
-
2016
- 2016-12-02 CN CN201611097372.6A patent/CN106776952B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1526108A (en) * | 2001-02-14 | 2004-09-01 | 无线谷通讯有限公司 | Method and system for modeling and managing terrain, buildings, and infrastructure |
| CN102710763A (en) * | 2012-05-29 | 2012-10-03 | 新华网股份有限公司 | Method and system for pooling, slicing and fault transfer of distributed cache |
| CN102857554A (en) * | 2012-07-26 | 2013-01-02 | 福建网龙计算机网络信息技术有限公司 | Data redundancy processing method based on distributed storage system |
| CN105335384A (en) * | 2014-06-30 | 2016-02-17 | 中航商用航空发动机有限责任公司 | Storage method for monitoring data, reproduction method for monitoring data and devices |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106776952A (en) | 2017-05-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100392611C (en) | Storage control apparatus and method | |
| JP4767139B2 (en) | Storage management program, storage management device, and storage management method | |
| US9395938B2 (en) | Storage control device and method for controlling storage devices | |
| CN102184129B (en) | Fault tolerance method and device for disk arrays | |
| CN110188084B (en) | Distributed file storage system and file storage method | |
| US20100229033A1 (en) | Storage management device, storage management method, and storage system | |
| JP2008250566A (en) | Disk array device, and method and program for operating this device | |
| US20110289349A1 (en) | System and Method for Monitoring and Repairing Memory | |
| CN105892934A (en) | Method and device used for memory equipment management | |
| CN104503781A (en) | Firmware upgrading method for hard disk and storage system | |
| CN110597655A (en) | Fast predictive restoration method for coupling migration and erasure code-based reconstruction and implementation | |
| CN106776952B (en) | Date storage method in a kind of distributed system | |
| CN104580381A (en) | Method for performing job task in multi-node network | |
| CN108153615B (en) | A kind of fault data restoration methods | |
| US20080208930A1 (en) | Management of redundancy in data arrays | |
| CN108156189B (en) | Redundant data processing method in a kind of multi-node system | |
| CN110837660A (en) | Data storage method and system and data verification method and system | |
| CN106708445B (en) | Link selecting method and device | |
| EP2645250A2 (en) | Storage system, storage apparatus, and data restoration method | |
| US20150379788A1 (en) | Method for managing fault messages of a motor vehicle | |
| CN116414661B (en) | Processing method and device for solid state disk of distributed storage | |
| CN107122254B (en) | Computer repair control method and system and repair method and system | |
| CN108334279B (en) | Data reading and writing method based on load balancing mode | |
| US10168935B2 (en) | Maintaining access times in storage systems employing power saving techniques | |
| CN103176745A (en) | Hard disc array takeover method of storage system with double controllers |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 101399 No. 2 East Airport Road, Shunyi Airport Economic Core Area, Beijing (1st, 5th and 7th floors of Industrial Park 1A-4) Applicant after: Zhongke Star Map Co., Ltd. Address before: 101399 Building 1A-4, National Geographic Information Technology Industrial Park, Guomen Business District, Shunyi District, Beijing Applicant before: Space Star Technology (Beijing) Co., Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CB03 | Change of inventor or designer information |
Inventor after: Wu Fangcai Inventor after: Liu Bin Inventor after: Chu Yongquan Inventor before: Liu Bin Inventor before: Wu Fangcai Inventor before: Chu Yongquan |
|
| CB03 | Change of inventor or designer information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Song Defeng Inventor after: Wu Fangcai Inventor after: Liu Bin Inventor after: Chu Yongquan Inventor before: Wu Fangcai Inventor before: Liu Bin Inventor before: Chu Yongquan |
|
| CB03 | Change of inventor or designer information |