CN106776952A - Date storage method in a kind of distributed system - Google Patents
Date storage method in a kind of distributed system Download PDFInfo
- Publication number
- CN106776952A CN106776952A CN201611097372.6A CN201611097372A CN106776952A CN 106776952 A CN106776952 A CN 106776952A CN 201611097372 A CN201611097372 A CN 201611097372A CN 106776952 A CN106776952 A CN 106776952A
- Authority
- CN
- China
- Prior art keywords
- node
- storage
- packet
- data
- risk
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present invention relates to date storage method in a kind of distributed system, according to memory capacity, operation load and failure-frequency calculate the storage value-at-risk for determining each node, multiple nodes are grouped, each packet includes storage value-at-risk node and the relatively low node of a storage value-at-risk higher, so that the average storage value-at-risk relative equilibrium in each packet, avoid redundant data distribution some nodes be Frequent Troubles node situation, redundant data is distributed in so packet of storage risk balance, facilitate data maintenance and reduce loss of data risk.
Description
【Technical field】
The invention belongs to date storage method under field of data storage, more particularly to distributed system environment.
【Background technology】
Generally, an external highest of handling up for providing of machine also can only achieve 200MBps, according to common machine
Device mirror image is the completely the same mode of data on several machines, and repairing the data of 12TB needs to take more than 20 hour, it is considered to
To normal service pressure, the time of reparation is up to tens hours.
Distributed storage mode is proposed in the prior art, is multiple storehouses by data cutting, and replicate several parts of redundancies, it is same
The different redundant distributions in storehouse are stored in the diverse location of different machines, improve reparation speed.But in the prior art by redundancy
When data distribution is on different machines, random fashion is used, different machines are not selected, cause identical data
The machine of distribution is all probably Frequent Troubles machine, thus results in data maintenance difficulty and improves loss of data risk.
Based on above mentioned problem, a kind of new distributed system storage method is needed badly now, by node according to storage value-at-risk
It is grouped so that the average storage value-at-risk relative equilibrium of each packet, redundant data is distributed in each packet, it is convenient
Data maintenance and reduce loss of data risk.
【The content of the invention】
In order to solve above mentioned problem of the prior art, the present invention proposes data storage side in a kind of distributed system
Method.
The technical solution adopted by the present invention is as follows：
A kind of date storage method in distributed system, the method comprises the following steps：
(1) m node i in distributed system is loaded according to memory capacity, operation and failure-frequency is according to following public affairs
Formula (a) calculates the storage value-at-risk R for determining each nodei:
Ri=Si×Pi+1/Fi×Qi+Gi×Ti(a)；
Wherein SiRepresent the memory capacity of node i, PiRepresent the weight corresponding to memory capacity, FiRepresent the operation of node i
Highest system operation load in history, QiRepresent the weight corresponding to operation load, GiIn the history run of expression node i
Failure-frequency, TiRepresent the weight corresponding to failure-frequency, and Pi, Qi, Gi>1；
(2) m node is lined up an ordered queue by the order according to storage value-at-risk from low to high, by ordered queue
First node of head of the queue and tail of the queue last node taking-up constitute the first storage packet, for the orderly team that remaining node is constituted
Row continue with the follow-up multiple storage packets of composition in the manner described above, until only 2 or 3 nodes in ordered queue, then
Using above-mentioned 2 or 3 nodes as a storage packet, final m node is divided into k storage packet；
(3) when system receives data storage request, data are cut into k according to fragment, to each data slot
Replicate, obtain the packet of k groups data slots, every group of data slot is grouped includes data slot replicate data corresponding with its
Fragment；
(4) data slot in a data fragment packet and corresponding replicate data fragment are respectively stored into one to deposit
In two nodes in storage packet, until the data slot and replicate data fragment in the packet of k groups data slot all store k
In storage packet；
(5) when a nodes break down in storing packet, another node in above-mentioned storage packet is deposited
The data slot or replicate data fragment of storage, repair to the above-mentioned node for breaking down；
(6) man-to-man port is set to each node, when a node fails, the corresponding port of above-mentioned node is automatic
Close, after the node for breaking down successfully is repaired, automatically turn on the corresponding port of above-mentioned node.
Beneficial effects of the present invention include：Multiple nodes are grouped, each packet includes a storage value-at-risk
Node higher and the relatively low node of a storage value-at-risk so that the average storage value-at-risk relative equilibrium in each packet,
Avoid redundant data distribution some nodes be Frequent Troubles node situation, by redundant data be distributed to so storage
In the packet of risk balance, facilitate data maintenance and reduce loss of data risk.
【Brief description of the drawings】
Accompanying drawing described herein be for providing a further understanding of the present invention, constituting the part of the application, but
Inappropriate limitation of the present invention is not constituted, in the accompanying drawings：
Fig. 1 is the structure chart of distributed system of the present invention.
Fig. 2 is the flow chart of date storage method in distributed system of the present invention.
【Specific embodiment】
Describe the present invention in detail below in conjunction with accompanying drawing and specific embodiment, illustrative examples therein and say
It is bright to be only used for explaining the present invention but not as a limitation of the invention.
It is the distributed system applied of the invention referring to accompanying drawing 1, the system includes multiple calculate nodes.
Referring to accompanying drawing 2, date storage method in a kind of distributed system, the method comprises the following steps：
(1) m node i in distributed system is loaded according to memory capacity, operation and failure-frequency is according to following public affairs
Formula (a) calculates the storage value-at-risk R for determining each nodei:
Ri=Si×Pi+1/Fi×Qi+Gi×Ti(a)；
Wherein SiRepresent the memory capacity of node i, PiRepresent the weight corresponding to memory capacity, FiRepresent the operation of node i
Highest system operation load in history, QiRepresent the weight corresponding to operation load, GiIn the history run of expression node i
Failure-frequency, TiRepresent the weight corresponding to failure-frequency, and Pi, Qi, Gi>1；
Wherein, memory capacity, operation load and failure-frequency be influence respectively node storage value-at-risk it is different because
Element, wherein memory capacity represent a storage capacity for node, and memory capacity is bigger, then above-mentioned node is because data storage pressure
The failure risk for causing is lower, otherwise higher；The operation load of one node is bigger, then above-mentioned node is because operation excess load institute
The failure risk for causing is bigger, otherwise lower；The failure-frequency that one node occurs within the history run cycle is higher, then show
The possibility that above-mentioned node breaks down in the cycle afterwards is higher, otherwise lower.
In one embodiment, memory capacity, operation load and failure-frequency are recorded in a table, each node
Memory capacity can refer to its hard disc of computer memory capacity, and the memory capacity of each node is recorded in table, monitor that each is saved
Be run multiple times shared system resource in point predetermined time cycle, and using the most system resources shared by operation as
Highest system operation load record monitors the failure frequency in each node predetermined time cycle in above-mentioned table, makees
It is that failure-frequency is recorded in table；
In the storage value-at-risk R of calculate nodeiWhen, the memory capacity corresponding to above-mentioned node, operation are read from table and is born
Carry and failure-frequency is calculated according to above-mentioned formula (a).
(2) m node is lined up an ordered queue by the order according to storage value-at-risk from low to high, by ordered queue
First node of head of the queue and tail of the queue last node taking-up constitute the first storage packet, for the orderly team that remaining node is constituted
Row continue with the follow-up multiple storage packets of composition in the manner described above, until only 2 or 3 nodes in ordered queue, i.e.,
Corresponding to the situation that m is even number and odd number, then using above-mentioned 2 or 3 nodes as a storage packet, final m node divides
Into k storage packet；
(3) when system receives data storage request, data are cut into k according to fragment, to each data slot
Replicate, obtain the packet of k groups data slots, every group of data slot is grouped includes data slot replicate data corresponding with its
Fragment；
(4) data slot in a data fragment packet and corresponding replicate data fragment are respectively stored into one to deposit
In two nodes in storage packet, until the data slot and replicate data fragment in the packet of k groups data slot all store k
In storage packet；
Because each packet includes storage value-at-risk node and the relatively low node of a storage value-at-risk higher,
So that the average storage value-at-risk relative equilibrium in each packet, it is to avoid some nodes of redundant data distribution are failure frequently
The situation of the node of hair, redundant data is distributed in so packet of storage risk balance, facilitates data maintenance and reduction
Loss of data risk.
(5) when a nodes break down in storing packet, another node in above-mentioned storage packet is deposited
The data slot or replicate data fragment of storage, repair to the above-mentioned node for breaking down；
Thus, it will greatly improve the speed of repair data, repair time is shortened, it is right when multiple nodes break downs
Multiple nodes are repaired parallel, and the data slot and corresponding replicate data fragment in a data fragment packet are according to random
Mode is stored in two nodes in a storage packet.It is whole in quantity of the machine quantity more than the storehouse on failed machines
The time-consuming of individual repair process is usually only necessary to dozens of minutes, solves the problems, such as that data efficient is repaired automatically.
(6) man-to-man port is set to each node, when a node fails, the corresponding port of above-mentioned node is automatic
Close, after the node for breaking down successfully is repaired, automatically turn on the corresponding port of above-mentioned node.So as to ensure data just
Really read, and avoid the problem for using error listing access port.
By the above method, be grouped for multiple nodes by the present invention, each packet include one store value-at-risk compared with
Node high and the relatively low node of a storage value-at-risk so that the average storage value-at-risk relative equilibrium in each packet, keep away
Exempted from redundant data distribution some nodes be Frequent Troubles node situation, by redundant data be distributed to so storage wind
During danger is grouped in a balanced way, facilitates data maintenance and reduce loss of data risk.
The above is only better embodiment of the invention, therefore all constructions according to described in present patent application scope,
The equivalent change or modification that feature and principle are done, is included in the range of present patent application.
Claims (3)
1. date storage method in a kind of distributed system, it is characterised in that the method comprises the following steps：
(1) by m node i in distributed system according to memory capacity, operation load and failure-frequency according to the following equation (a)
Calculate the storage value-at-risk R for determining each nodei:
Ri=Si×Pi+1/Fi×Qi+Gi×Ti(a)；
Wherein SiRepresent the memory capacity of node i, PiRepresent the weight corresponding to memory capacity, FiRepresent the history run of node i
In highest system operation load, QiRepresent the weight corresponding to operation load, GiRepresent the failure in the history run of node i
Frequency, TiRepresent the weight corresponding to failure-frequency, and Pi, Qi, Gi>1；
(2) m node is lined up an ordered queue by the order according to storage value-at-risk from low to high, by head of the queue in ordered queue
First node and tail of the queue last node taking-up constitute the first storage packet, are pressed for the ordered queue that remaining node is constituted
The follow-up multiple storage packets of composition are continued with according to aforesaid way, until only 2 or 3 nodes in ordered queue, then will be upper
State 2 or 3 nodes are grouped as a storage, final m node is divided into k storage packet；
(3) when system receives data storage request, data are cut into k according to fragment, each data slot are replicated,
Obtain the packet of k groups data slots, every group of data slot is grouped includes data slot replicate data fragment corresponding with its；
(4) data slot in a data fragment packet and corresponding replicate data fragment are respectively stored into a storage point
In two nodes in group, until the data slot and replicate data fragment in the packet of k groups data slot all store k storage
In packet；
(5) when a nodes break down in storing packet, what another node in above-mentioned storage packet was stored
Data slot or replicate data fragment, repair to the above-mentioned node for breaking down；
(6) man-to-man port is set to each node, when a node fails, the corresponding port of above-mentioned node is closed automatically
Close, after the node for breaking down successfully is repaired, automatically turn on the corresponding port of above-mentioned node.
2. date storage method in distributed system according to claim 1, it is characterised in that occur when multiple nodes therefore
During barrier, multiple nodes are repaired parallel.
3. date storage method in distributed system according to claim 1, it is characterised in that a data fragment packet
In data slot and corresponding replicate data fragment according to random fashion store to one storage packet in two nodes in.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611097372.6A CN106776952B (en) | 2016-12-02 | 2016-12-02 | Date storage method in a kind of distributed system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611097372.6A CN106776952B (en) | 2016-12-02 | 2016-12-02 | Date storage method in a kind of distributed system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106776952A true CN106776952A (en) | 2017-05-31 |
| CN106776952B CN106776952B (en) | 2019-03-22 |
Family
ID=58884165
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611097372.6A Active CN106776952B (en) | 2016-12-02 | 2016-12-02 | Date storage method in a kind of distributed system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106776952B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110704232A (en) * | 2019-10-10 | 2020-01-17 | 广东工业大学 | Method, device and equipment for repairing failed node in distributed system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1526108A (en) * | 2001-02-14 | 2004-09-01 | 无线谷通讯有限公司 | Method and system for modeling and managing terrain, buildings, and infrastructure |
| US20110107026A1 (en) * | 2009-10-30 | 2011-05-05 | Cleversafe, Inc. | Concurrent set storage in distributed storage network |
| CN102710763A (en) * | 2012-05-29 | 2012-10-03 | 新华网股份有限公司 | Method and system for pooling, slicing and fault transfer of distributed cache |
| CN102857554A (en) * | 2012-07-26 | 2013-01-02 | 福建网龙计算机网络信息技术有限公司 | Data redundancy processing method based on distributed storage system |
| CN105335384A (en) * | 2014-06-30 | 2016-02-17 | 中航商用航空发动机有限责任公司 | Storage method for monitoring data, reproduction method for monitoring data and devices |
-
2016
- 2016-12-02 CN CN201611097372.6A patent/CN106776952B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1526108A (en) * | 2001-02-14 | 2004-09-01 | 无线谷通讯有限公司 | Method and system for modeling and managing terrain, buildings, and infrastructure |
| US20110107026A1 (en) * | 2009-10-30 | 2011-05-05 | Cleversafe, Inc. | Concurrent set storage in distributed storage network |
| CN102710763A (en) * | 2012-05-29 | 2012-10-03 | 新华网股份有限公司 | Method and system for pooling, slicing and fault transfer of distributed cache |
| CN102857554A (en) * | 2012-07-26 | 2013-01-02 | 福建网龙计算机网络信息技术有限公司 | Data redundancy processing method based on distributed storage system |
| CN105335384A (en) * | 2014-06-30 | 2016-02-17 | 中航商用航空发动机有限责任公司 | Storage method for monitoring data, reproduction method for monitoring data and devices |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110704232A (en) * | 2019-10-10 | 2020-01-17 | 广东工业大学 | Method, device and equipment for repairing failed node in distributed system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106776952B (en) | 2019-03-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11487618B2 (en) | Data storage method, apparatus, and system | |
| US9891994B1 (en) | Updated raid 6 implementation | |
| EP2856318B1 (en) | Method and apparatus for single point of failure elimination for cloud-based applications | |
| CN105897499A (en) | Distributed storage system node state monitoring method, center code and system | |
| CN102857554A (en) | Data redundancy processing method based on distributed storage system | |
| CN1208891A (en) | Disc memory device data renewing method and disc memory controlling device | |
| JP2011159241A (en) | Storage apparatus, controller of the same and method for allocating storage area in storage apparatus | |
| CN109951537A (en) | A kind of load balance distribution method towards block chain | |
| CN101079896A (en) | A multi-availability mechanism coexistence framework of concurrent storage system | |
| US20190220377A1 (en) | Method and device of data rebuilding in storage system | |
| CN110413203A (en) | For managing the method, equipment and computer program product of storage system | |
| CN116340005B (en) | Container cluster scheduling method, device, equipment and storage medium | |
| CN110493060A (en) | A kind of virtual IP address distribution method and relevant apparatus | |
| US9602381B1 (en) | Real time adaptive monitoring | |
| CN106776952A (en) | Date storage method in a kind of distributed system | |
| CN107181636A (en) | Health examination method and device in a kind of SiteServer LBS | |
| CN107544874A (en) | Method for processing business and device | |
| CN103034457B (en) | Data storage method of storage system formed by multiple hard disks | |
| CN108153615B (en) | A kind of fault data restoration methods | |
| CN109766249A (en) | A kind of state display device of array hard disk | |
| CN105450679A (en) | Method and system for performing data cloud storage | |
| CN108156189B (en) | Redundant data processing method in a kind of multi-node system | |
| CN115878047B (en) | Data consistency verification method, device, equipment and storage medium | |
| US9153347B2 (en) | Method and system for reliable big capacity storage system protected by triple protection | |
| CN106844487A (en) | A kind of distributed memory system replicated based on storehouse |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 101399 No. 2 East Airport Road, Shunyi Airport Economic Core Area, Beijing (1st, 5th and 7th floors of Industrial Park 1A-4) Applicant after: Zhongke Star Map Co., Ltd. Address before: 101399 Building 1A-4, National Geographic Information Technology Industrial Park, Guomen Business District, Shunyi District, Beijing Applicant before: Space Star Technology (Beijing) Co., Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CB03 | Change of inventor or designer information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Wu Fangcai Inventor after: Liu Bin Inventor after: Chu Yongquan Inventor before: Liu Bin Inventor before: Wu Fangcai Inventor before: Chu Yongquan |
|
| CB03 | Change of inventor or designer information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Song Defeng Inventor after: Wu Fangcai Inventor after: Liu Bin Inventor after: Chu Yongquan Inventor before: Wu Fangcai Inventor before: Liu Bin Inventor before: Chu Yongquan |