CN109407976A - A kind of distributed storage method and distributed storage devices - Google Patents
A kind of distributed storage method and distributed storage devices Download PDFInfo
- Publication number
- CN109407976A CN109407976A CN201811117783.6A CN201811117783A CN109407976A CN 109407976 A CN109407976 A CN 109407976A CN 201811117783 A CN201811117783 A CN 201811117783A CN 109407976 A CN109407976 A CN 109407976A
- Authority
- CN
- China
- Prior art keywords
- node
- memory
- memory node
- emergency
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0626—Reducing size or complexity of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the present application provides a kind of distributed storage method and distributed storage devices, the described method includes: when detecting that the first memory node reaches full load state in storage device: obtaining the status information of remaining memory node, the status information includes at least residual memory space information；Determine the second memory node as the emergency node for carrying out dilatation to first memory node according to the state information.Distributed storage method in embodiments herein can be avoided effectively in storage device because the phenomenon that leading to storage device when a certain node reaches full load state integrally and cannot continue the read-write of new data, influence user's use occurs.
Description
Technical field
The invention relates to field of data storage, in particular to this kind of method of a kind of distributed storage method and application
Distributed storage devices.
Background technique
Generally it is distributed and positions more copy datas using consistency hash algorithm in distributed storage devices at present, but
It is that consistency Hash does not ensure that data can be completely uniformly distributed, so often going out in distributed storage devices
Now when the memory space utilization rate of some node reaches 100% i.e. full load, the memory space of other nodes is used
Rate may can use only 70% there are also many memory spaces.And essentially all distributed storage devices all can be because at present
When reaching full load for the memory space of some node, just assert that cluster generally in full load state, causes cluster can not
Continue to read and write new data, causes the interruption of client traffic.
For above-mentioned technical problem, there is presently no preferable solutions.What can be accomplished in currently available technology is
It alerts in advance, adds timing monitoring device to each memory node, with the memory space utilization rate of monitoring node, when timing monitors
When equipment detects that the memory space utilization rate of certain node reaches some threshold value, such as when adjacent to full load, hair
Mail carries out cluster dilatation to device manager, alarm set administrator in time.But alert in advance, memory space can only be prevented
The case where full load causes client traffic to interrupt is reached, but there is no solve this problem.If there is no new set after warning
Standby timely dilatation cluster, once the memory space of some node reaches full load, cluster can not still continue to read and write new data,
Client traffic still can interrupt.
It has been used to ensure that client traffic will not interrupt suddenly there are also a kind of way at present, therefore has been rejected from storage device
Full load node, but certain sections are certainly existed when some node reaches full load in distributed storage devices
Point be also substantially saturated, these nodes will appear the node of next full load quickly in data recovery procedure, thus at present this
Kind method disadvantage is more, and risk is big, is not suitable for being widely used.
Summary of the invention
Embodiments herein provides one kind can be in the storage device because a certain memory node reaches full load state
When, storage device integrally remains to continue the distribution of this kind of method of distributed storage method and application of the read-write of new data
Storage device.
In order to solve the above-mentioned technical problem, the embodiment of the present application provides the following technical solution:
A kind of distributed storage method, comprising:
When detecting that the first memory node reaches full load state in storage device:
The status information of remaining memory node is obtained, the status information includes at least residual memory space information；
Determine the second memory node as carrying out dilatation to first memory node according to the state information
Emergency node.
Preferably, the history that the status information further includes the memory space of the corresponding memory node uses letter
Breath；
It is described to determine that the second memory node is used as expanding first memory node according to the state information
The emergency node of appearance specifically:
Candidate memory node is determined according to the residual memory space information；
The emergency node is determined from the candidate memory node according to the history use information.
Preferably, the status information further includes for showing whether the corresponding memory node is that the third is deposited
Store up the first mark information of the emergency node of node；
It is described to determine that the second memory node is used as expanding first memory node according to the state information
The emergency node of appearance specifically:
Candidate memory node is determined according to the residual memory space information；
The emergency node is determined from the candidate memory node according to first mark information.
Preferably, further include:
The number newly read and write for storing first memory node is determined according to the storage space information of the emergency node
According to emergency memory space；
The emergency memory space is mapped as to the memory space of first memory node.
Preferably, further include:
Determine whether second memory node has the second mark information, second mark information is described for showing
Second memory node has been the emergency node of first memory node；
If not having, it will determine that second memory node is the emergency node；
If having, the emergency memory space is carried out according to the residual memory space information of second memory node
Dilatation.
Preferably, further include:
Generate the hint instructions for prompting storage device described in the timely dilatation of user；
Respond the hint instructions.
Preferably, the method also includes:
After the storage device dilatation, the data of first memory node stored in the emergency node are shifted
Into the memory space of first memory node；
Determine whether the data all shift；
If so, unmapped relationship and discharging the emergency memory space
The embodiment of the present application provides a kind of distributed storage devices simultaneously, comprising:
Memory node, to be multiple, for cooperating distributed storage data；
Management node is used to obtain remaining memory node when detecting that the first memory node reaches full load state
Status information, and according to status information determine the second memory node as to the first memory node carry out dilatation emergency
Node, wherein status information includes at least residual memory space information.
Preferably, the history that the status information further includes the memory space of the corresponding memory node uses letter
Breath；
The management node determines candidate memory node according to the residual memory space information, and is made according to the history
The emergency node is determined from the candidate memory node with information.
Preferably, the status information further includes for showing whether the corresponding memory node is that the third is deposited
Store up the first mark information of the emergency node of node；
The management node determines candidate memory node according to the residual memory space information, and according to first mark
Remember that information determines the emergency node from the candidate memory node.
Disclosure based on the above embodiment can know, the embodiment of the present application have it is following the utility model has the advantages that
There is memory node to reach full load state when encountering in distributed storage devices, and can not carry out in time instantly
When the case where cluster dilatation, emergency section can be determined by the current state information of remaining memory node in identification cluster
Point makes that the memory node of the full load state is assisted to continue to store the data of user's input by emergency node, to ensure to deposit
Storage device is integrally to the normal read-write of data, to guarantee that user is not in interrupt in the way using distributed storage devices
Phenomenon, while the dilatation of memory space is also carried out to storage device to the cluster administrator additional time.
Detailed description of the invention
Fig. 1 is the method flow diagram of an embodiment in the application distributed storage method.
Fig. 2 is the method flow diagram of an embodiment in the application distributed storage method.
Fig. 3 is the method flow diagram of an embodiment in the application distributed storage method.
Fig. 4 is the method flow diagram of an embodiment in the application distributed storage method.
Fig. 5 is the structural block diagram of an embodiment of the application distributed storage devices.
Specific embodiment
In the following, being described in detail in conjunction with specific embodiment of the attached drawing to the application, but not as the restriction of the application.
It should be understood that various modifications can be made to disclosed embodiments.Therefore, description above should not regard
To limit, and only as the example of embodiment.Those skilled in the art will expect within the scope and spirit of this
Other modifications.
The attached drawing being included in the description and forms part of the description shows embodiment of the disclosure, and with it is upper
What face provided is used to explain the disclosure together to substantially description and the detailed description given below to embodiment of the disclosure
Principle.
By the description of the preferred form with reference to the accompanying drawings to the embodiment for being given as non-limiting example, the application's
These and other characteristic will become apparent.
It is also understood that although the application is described referring to some specific examples, those skilled in the art
Member realizes many other equivalents of the application in which can determine, they have feature as claimed in claim and therefore all
In the protection scope defined by whereby.
When read in conjunction with the accompanying drawings, in view of following detailed description, above and other aspect, the feature and advantage of the disclosure will become
It is more readily apparent.
The specific embodiment of the disclosure is described hereinafter with reference to attached drawing；It will be appreciated, however, that the disclosed embodiments are only
Various ways implementation can be used in the example of the disclosure.Known and/or duplicate function and structure and be not described in detail to avoid
Unnecessary or extra details makes the disclosure smudgy.Therefore, specific structural and functionality disclosed herein is thin
Section is not intended to restrictions, but as just the basis of claim and representative basis be used to instructing those skilled in the art with
Substantially any appropriate detailed construction diversely uses the disclosure.
This specification can be used phrase " in one embodiment ", " in another embodiment ", " in another embodiment
In " or " in other embodiments ", it can be referred to one or more of the identical or different embodiment according to the disclosure.
In the following, the embodiment of the present application is described in detail in conjunction with attached drawing.
Fig. 1 is the method flow diagram of one embodiment of the application.As shown in Figure 1, the embodiment of the present application discloses a kind of distribution
Storage method, comprising:
When detecting that the first memory node reaches full load state in storage device:
The status information of remaining memory node is obtained, status information includes at least residual memory space information；
Determine the second memory node as the emergency node for carrying out dilatation to the first memory node according to status information.
By the above method, storage device (i.e. distributed storage devices) is when carrying out distributed storage to data, if hair
Existing first memory node reaches full load state, then is determined according to the residual memory space information of remaining memory node of acquisition
Second memory node enables the second memory node as the emergency node of the memory space for the first memory node of dilatation.
That is, the information that the first memory node is newly read and write can be directly stored into the emergency node, make in distributed storage devices
The memory spaces of all memory nodes can maximize use as far as possible, avoid as in the prior art only because of some storage
The memory space of node uses totally, and makes entire storage device using the state of the memory node as current state, causes
Entire storage device can not continue to read and write new data, influence user to the normal use of storage device.That is, passing through the application
Method can the memory node of some in distributed storage devices reach full load state when will not directly interrupt new data
Read-write, influence the normal use of user, but at least portion of the memory node of full load state can be not in by other
Divide memory space of the memory space as the full load memory node, so that storage device can also continue to read and write newly writing for user
The data entered guarantee the normal use of user.
Specifically, executing step 1: when detecting in storage device that the first memory node reaches full load state, this
One memory node can be for any one memory node in storage device, the memory node including reaching full load state for the first time,
Or n-th (n is the integer greater than 1) reaches the memory node of full load state.In addition, detection mode is not also unique, such as deposit
Storage device (the concretely management node in storage device) can actively examine the memory space of each memory node in real time
It surveys or timing detects, be also possible to actively issue signal when a certain memory node reaches full load state, make storage device (tool
Body can be the management node in storage device) it can learn that the memory node will reach or have arrived at full load according to the signal
State.
Executing step 2: when obtaining the status information of remaining memory node, the status information of remaining each memory node can
Real-time or timing detection is actively carried out by storage device to obtain, it can also be from each memory node in real time or periodically to storage device
The state that (the concretely management node in storage device) is sent and storage device is made to obtain remaining each memory node is believed
Breath.Moreover, the execution of the step can carry out simultaneously with step 1；Can also after in step 1 carry out, that is, detecting full load shape
It is just carried out when the memory node of state；It can also independently be carried out with step 1, that is, the execution of step 2 is not based on step 1, also not
It is affected by it.
Status information in the present embodiment step 2 includes at least the residual memory space information of remaining each memory node.Base
In this, electronic equipment execute step 3: according to status information determine the second memory node be used as to the first memory node into
When the emergency node of row dilatation, it can determine that residual memory space is most by comparing the remaining space information of each memory node
Memory node as final emergency node；It may be based on the remaining space information of each memory node and by multiple memory nodes
Spatially size is ranked up, and is based even on the ordering relation and is determined candidate memory node, later based in status information
Other information and determine final emergency node from multiple candidate memory nodes.
For example, the memory node is being handled all when certain memory node reaches full load state in storage device
Or I/O request to be processed is hung up, and the management node into storage device initiates request.Management node receives request
Afterwards, the stored copies quantity that current memory device has is judged, when stored copies quantity at least two (that is, user is defeated
When entering a data, which can be at least stored in two memory nodes simultaneously by storage device, make to be formed at least two about
The stored copies of the data), determine that data backup is stablized, it can be ensured that no data is lost, and management node just will start emergency at this time
(after emergency plan starting, the stored copies quantity for the data being newly written may be constant, it is also possible to reduce by a storage for processing scheme
Copy), namely step 2 and step 3 are successively executed: the residual memory space of remaining each memory node in inquiry storage device, to
After each memory node returns to status information, emergency node is filtered out accordingly.
Further, in order to be adapted to more kinds of applicable cases, while ensuring that data will not be merely stored in a certain storage
Node leads to Single Point of Faliure and generates data storage security hidden danger, and the status information in the present embodiment step 2 not only includes it
The residual memory space information of remaining each memory node, may also include one of following information or a variety of, specifically:
Embodiment one:
Status information further includes the history use information of the memory space of corresponding memory node；
When status information further includes above-mentioned history use information, storage device is executing step 3: true according to status information
When fixed second memory node is used as to the emergency node of the first memory node progress dilatation specifically:
Candidate memory node is determined according to residual memory space information；
Emergency node is determined from candidate memory node according to history use information.
By taking multiple ATM machine that storage device is applied to A bank are handled in the deposit or the withdrawal as an example, it is used in each ATM machine
The storage disk (entity disk or virtual disk) of the data of storage user's input is regarded as a memory node, or
Person's storage device can also be cloud storage device, that is, each ATM machine is communicated to connect with cloud processing unit, and user is in ATM machine
The data of input can be conveyed directly to cloud and carry out storage processing, multiple entities or virtual storage dress in cloud storage device
It sets to form multiple memory nodes.When the management node of storage device has detected that a memory node reaches full load state, simultaneously
It detects in the status information of remaining memory node of acquisition not only comprising residual memory space information, also includes each memory node
Memory space history use information when, storage device can first according to the residual memory space information of remaining memory node according to
Preset rules determine candidate memory node, such as residual memory space capacity successively sort descending, and enable first five memory node
As candidate memory node.Each candidate storage section is determined according to the history use information of the memory space of each memory node later
Which has overfill load condition once occurred in point, then determines emergency node from multiple candidate memory nodes accordingly.It should
The condition that emergency node need to meet needs residual memory space as big as possible, and includes at least the memory node and do not occur overfill once
Load condition.The reason of emergency node is arranged such avoids the memory node from having other while being to meet storage demand
Emergency node, leads to the in disorder complexity of relationship between memory node, influences the distributed storage of data.
Embodiment two:
Status information further include for show corresponding memory node whether be third memory node emergency node the
One mark information；
When status information further includes above-mentioned first mark information, storage device is executing step 3: true according to status information
When fixed second memory node is used as to the emergency node of the first memory node progress dilatation specifically:
Candidate memory node is determined according to residual memory space information；
Emergency node is determined from candidate memory node according to the first mark information.
Continue above-described embodiment, it, can be excellent when storage device detects the mark information of acquisition comprising the first mark information
It is first ordered from large to small by residual memory space, first five memory node is then determined as candidate by preset requirement and is deposited
Store up node.Later, whether successively detect according to collating sequence has the first label letter in the status information of each candidate memory node
Breath is finally determined without the first mark information and the maximum candidate memory node of residual memory space is final emergency section
Point.Emergency node after being determined using this kind of mode can guarantee distributed storage Information Security, it is ensured that same part data it is more
A copy will not be fallen on the same memory node, lead to the memory node failure.
Embodiment three:
Fig. 2 is the method flow diagram of an embodiment in the application distributed storage method.As shown in Fig. 2, in the present embodiment
Status information simultaneously include history use information and the first mark information in embodiment one and embodiment two.
Specifically, still by taking above-described embodiment as an example, when storage device determines that candidate deposits according to residual memory space information
After storing up node, the history use information and the first mark information of each candidate memory node are sequentially successively detected, is finally determined same
When meet big memory space, non-third memory node emergency node and do not occur the candidate memory node of overfill load condition and be
Emergency node.
Further, the distributed storage method after determining emergency node, in the present embodiment further include:
The emergency for storing the data that the first memory node is newly read and write is determined according to the storage space information of emergency node
Memory space；
Memory space of meeting an urgent need is mapped as to the memory space of the first memory node.
When it is implemented, can be determined according to the storage space information of the emergency node in the residual memory space of emergency node
It can release the emergency memory space for the data that how many spaces are newly written as the first memory node of storage, such as can determine this
Meet an urgent need node residual memory space in half or one third, three/it is second-class be used as emergency memory space.The emergency is deposited
Storage space can not occupy the residual memory space of emergency node 80 percent to very, it is necessary to ensure that the emergency node from
Body can also normally complete the write-in of new data.Later, mapping tool can be used the emergency memory space is mapped as first and deposit
The memory space for storing up node, that is, the data that the first memory node is newly written can be written directly in the emergency memory space.
Further, Fig. 3 is the method flow diagram of an embodiment in the application distributed storage method., as shown in figure 3,
Distributed storage method in the present embodiment further include:
Determine whether the second memory node has the second mark information, the second mark information is for showing the second memory node
It has been the emergency node of the first memory node；
If not having, it will determine that the second memory node is emergency node；
If having, dilatation is carried out to emergency memory space according to the residual memory space information of the second memory node.
Specifically, after filtering out emergency node, the control node in storage device sends a request to emergency node, answers
Whether anxious node has the second mark information according to request detection itself, with judge this node (i.e. emergency node itself) whether it
Preceding has been the emergency node of the first memory node.If this node does not have the second mark information, prove not to be fully loaded section
The emergency node of point, therefore this section point is marked to the emergency node (forming the second mark information) for becoming fully loaded node.
Immediately, the file for for example accounting for its residual memory space half size is created in the bottom document device of emergency node, it should
File occupies disk space of corresponding size using pre-assigned mode, read and write these disk spaces will not by emergency node
Data use.Then the first memory node is sent a request to, makes it that this document mapping is become its local disk, to use
The disk extends its memory space.If it is judged that this node has been the emergency node of the first memory node, then before extending
The file with mapping relations space, propagation can for emergency node current residual storage space half.Then to
One memory node sends request, so that it rescans the file after extension, to update the memory space of its local disk.
Further, although storage device starts emergency plan, so that arrived the emergency node of full load state still
It can continue to read and write new data by the residual memory space of other memory nodes, but the inevitable situation that will appear is urgent, such as meets with
Data store full pressure period, and storage device read-write data frequency is higher, and residual memory space is not enough to continue to support for a long time.Therefore it is
It avoids the occurrence of data-storage interrupt phenomenon, influences user's use, the method in the present embodiment further include:
Generate the hint instructions for prompting the timely dilatation storage device of user；
Respond the hint instructions.
For example, the residual memory space amount of each memory node can be monitored in real time after storage device starts emergency plan,
After the residual memory space size for the memory node for detecting preset quantity is lower than preset value, hint instructions are just generated, are stored
Device can send to the devices such as exportable text or picture or sound according to the hint instructions and request, and make device output prompt letter
Breath or prompt tone, to notify the administrator of storage device to carry out dilatation in time to storage device.Certainly, the generation item of prompt information
Part might not be based on above-mentioned condition, such as can also be the size of data writing in storage device monitoring preset time, with this
Determine the generation time of prompt information.
Further, Fig. 4 is the method flow diagram of an embodiment in the application distributed storage method., as shown in figure 4,
Method after administrator carries out dilatation to storage device, in the present embodiment further include:
After the storage device dilatation, the data of first memory node stored in the emergency node are shifted
Into the memory space of first memory node；
Determine whether the data all shift；
If so, unmapped relationship and discharging the emergency memory space.
It is, it is remaining that the first memory node detects that local storage space occurs, at this time after storage device is by dilatation
Just the data in the emergency memory space in node of meeting an urgent need are transferred in local storage space.The data volume of transfer can be according to this
Depending on the surplus of secondary memory space, if current spatial surplus not enough receives the total data in emergency memory space can root
Keep the status after the data of transfer part for the time being according to actual conditions, that is, emergency node there are still.Or continue to notify administrator into
It is complete to shift this when the first memory node local storage space receives the total data in emergency memory space enough for row dilatation
Portion data, the mapping relations between the node that releases and meet an urgent need later, the node that makes to meet an urgent need when being determined as meeting an urgent need node by being switched to
Emergency rating restores to initial normal condition, while deleting file where emergency memory space, extremely with release emergency memory space
In the local storage space of the memory node.The step is equally applicable to all depositing with aforementioned mapping relations in storage device
Between storage node.
Fig. 5 is the structural block diagram of an embodiment of the application distributed storage devices.As shown in figure 5, the embodiment of the present application
A kind of distributed storage devices are provided simultaneously, comprising:
Memory node, to be multiple, for cooperating distributed storage data；
Management node is used to obtain remaining memory node when detecting that the first memory node reaches full load state
Status information, and according to status information determine the second memory node as to the first memory node carry out dilatation emergency
Node, wherein status information includes at least residual memory space information.
It is, storage device (i.e. distributed storage devices) to data carry out distributed storage when, if discovery first
Memory node reaches full load state, then determines that second deposits according to the residual memory space information of remaining memory node of acquisition
Node is stored up, enables the second memory node as the emergency node of the memory space for the first memory node of dilatation.That is, the
The information that one memory node is newly read and write can be directly stored into the emergency node, and make in distributed storage devices all deposits
The memory space of storage node can maximize use as far as possible, avoid as in the prior art only depositing because of some memory node
It stores up space to use totally, and makes entire storage device using the state of the memory node as current state, cause entirely to store
Device can not continue to read and write new data, influence user to the normal use of storage device.That is, can by the present processes
Some memory node in distributed storage devices will not directly interrupt the read-write of new data, shadow when reaching full load state
The normal use of user is rung, but at least partly memory space of the memory node of full load state can be not in by other
As the memory space of the full load memory node, so that storage device can also continue to read and write the data of user being newly written,
Guarantee the normal use of user.
Specifically, management node (executes when the first memory node reaches full load state in detecting storage device
When step 1), which can be any one memory node in storage device, including reach full load shape for the first time
The memory node or n-th (n is the integer greater than 1) of state reach the memory node of full load state.In addition, detection mode
It is not unique, such as storage device (the concretely management node in storage device) can be actively empty to the storage of each memory node
Between be measured in real time or timing detect, be also possible to a certain memory node reach full load state when actively issue signal,
Make storage device (the concretely management node in storage device) can be learnt according to the signal memory node will reach or
Have arrived at full load state.
When obtaining the status information of remaining memory node (when executing step 2), which saves management node
The status information of point can actively be carried out by storage device in real time or timing detection obtains, can also be in real time or fixed by each memory node
When sent to management node and the status information that makes storage device obtain remaining each memory node.Moreover, the step is held
Row can carry out simultaneously with step 1；Can also after in step 1 carry out, that is, in the memory node Shi Caijin for detecting full load state
Row；It can also independently be carried out with step 1, that is, the execution of step 2 is not based on step 1, also unaffected.
Status information in the present embodiment step 2 includes at least the residual memory space information of remaining each memory node.Base
In this, management node execute step 3: according to status information determine the second memory node be used as to the first memory node into
When the emergency node of row dilatation, it can determine that residual memory space is most by comparing the remaining space information of each memory node
Memory node as final emergency node；It may be based on the remaining space information of each memory node and by multiple memory nodes
Spatially size is ranked up, and is based even on the ordering relation and is determined candidate memory node, later based in status information
Other information and determine final emergency node from multiple candidate memory nodes.
For example, the memory node is being handled all when certain memory node reaches full load state in storage device
Or I/O request to be processed is hung up, and the management node into storage device initiates request.Management node receives request
Afterwards, the stored copies quantity that current memory device has is judged, when stored copies quantity at least two (that is, user is defeated
When entering a data, which can be at least stored in two memory nodes simultaneously by storage device, make to be formed at least two about
The stored copies of the data), determine that data backup is stablized, it can be ensured that no data is lost, and management node just will start emergency at this time
(after emergency plan starting, the stored copies quantity for the data being newly written may be constant, it is also possible to reduce by a storage for processing scheme
Copy), namely step 2 and step 3 are successively executed: the residual memory space of remaining each memory node in inquiry storage device, to
After each memory node returns to status information, emergency node is filtered out accordingly.
Further, in order to be adapted to more kinds of applicable cases, while ensuring that data will not be merely stored in a certain storage
Node leads to Single Point of Faliure and generates data storage security hidden danger, and the status information in the present embodiment step 2 not only includes it
The residual memory space information of remaining each memory node, may also include one of following information or a variety of, specifically:
Embodiment one:
Status information further includes the history use information of the memory space of corresponding memory node；
When status information further includes above-mentioned history use information, management node is executing step 3: true according to status information
When fixed second memory node is used as to the emergency node of the first memory node progress dilatation specifically:
Candidate memory node is determined according to residual memory space information；
Emergency node is determined from candidate memory node according to history use information.
By taking multiple ATM machine that storage device is applied to A bank are handled in the deposit or the withdrawal as an example, it is used in each ATM machine
The storage disk (entity disk or virtual disk) of the data of storage user's input is regarded as a memory node, or
Person's storage device can also be cloud storage device, that is, each ATM machine is communicated to connect with cloud processing unit, and user is in ATM machine
The data of input can be conveyed directly to cloud and carry out storage processing, multiple entities or virtual storage dress in cloud storage device
It sets to form multiple memory nodes.When the management node of storage device has detected that a memory node reaches full load state, simultaneously
It detects in the status information of remaining memory node of acquisition not only comprising residual memory space information, also includes each memory node
Memory space history use information when, storage device can first according to the residual memory space information of remaining memory node according to
Preset rules determine candidate memory node, such as residual memory space capacity successively sort descending, and enable first five memory node
As candidate memory node.Each candidate storage section is determined according to the history use information of the memory space of each memory node later
Which has overfill load condition once occurred in point, then determines emergency node from multiple candidate memory nodes accordingly.It should
The condition that emergency node need to meet needs residual memory space as big as possible, and includes at least the memory node and do not occur overfill once
Load condition.The reason of emergency node is arranged such avoids the memory node from having other while being to meet storage demand
Emergency node, leads to the in disorder complexity of relationship between memory node, influences the distributed storage of data.
Embodiment two:
Status information further include for show corresponding memory node whether be third memory node emergency node the
One mark information；
When status information further includes above-mentioned first mark information, management node is executing step 3: true according to status information
When fixed second memory node is used as to the emergency node of the first memory node progress dilatation specifically:
Candidate memory node is determined according to residual memory space information；
Emergency node is determined from candidate memory node according to the first mark information.
Continue above-described embodiment, it, can be excellent when storage device detects the mark information of acquisition comprising the first mark information
It is first ordered from large to small by residual memory space, first five memory node is then determined as candidate by preset requirement and is deposited
Store up node.Later, whether successively detect according to collating sequence has the first label letter in the status information of each candidate memory node
Breath is finally determined without the first mark information and the maximum candidate memory node of residual memory space is final emergency section
Point.Emergency node after being determined using this kind of mode can guarantee distributed storage Information Security, it is ensured that same part data it is more
A copy will not be fallen on the same memory node, lead to the memory node failure.
Embodiment three:
Status information in the present embodiment includes the history use information and first in embodiment one and embodiment two simultaneously
Mark information.
Specifically, still by taking above-described embodiment as an example, when management node determines that candidate deposits according to residual memory space information
After storing up node, the history use information and the first mark information of each candidate memory node are sequentially successively detected, is finally determined same
When meet big memory space, non-third memory node emergency node and do not occur the candidate memory node of overfill load condition and be
Emergency node.
Further, after determining emergency node, management node is also used to:
The emergency for storing the data that the first memory node is newly read and write is determined according to the storage space information of emergency node
Memory space；
Memory space of meeting an urgent need is mapped as to the memory space of the first memory node.
When it is implemented, can be determined according to the storage space information of the emergency node in the residual memory space of emergency node
It can release the emergency memory space for the data that how many spaces are newly written as the first memory node of storage, such as can determine this
Meet an urgent need node residual memory space in half or one third, three/it is second-class be used as emergency memory space.The emergency is deposited
Storage space can not occupy the residual memory space of emergency node 80 percent to very, it is necessary to ensure that the emergency node from
Body can also normally complete the write-in of new data.Later, mapping tool can be used the emergency memory space is mapped as first and deposit
The memory space for storing up node, that is, the data that the first memory node is newly written can be written directly in the emergency memory space.
Further, in this embodiment management node be also used to:
Determine whether the second memory node has the second mark information, the second mark information is for showing the second memory node
It has been the emergency node of the first memory node；
If not having, it will determine that the second memory node is emergency node；
If having, dilatation is carried out to emergency memory space according to the residual memory space information of the second memory node.
Specifically, after filtering out emergency node, the management node in storage device sends a request to emergency node, answers
Whether anxious node has the second mark information according to request detection itself, with judge this node (i.e. emergency node itself) whether it
Preceding has been the emergency node of the first memory node.If this node does not have the second mark information, prove not to be fully loaded section
The emergency node of point, therefore this section point is marked to the emergency node (forming the second mark information) for becoming fully loaded node.
Immediately, the file for for example accounting for its residual memory space half size is created in the bottom document device of emergency node, it should
File occupies disk space of corresponding size using pre-assigned mode, read and write these disk spaces will not by emergency node
Data use.Then the first memory node is sent a request to, makes it that this document mapping is become its local disk, to use
The disk extends its memory space.If it is judged that this node has been the emergency node of the first memory node, then before extending
The file with mapping relations space, propagation can for emergency node current residual storage space half.Then to
One memory node sends request, so that it rescans the file after extension, to update the memory space of its local disk.
Further, although management node starts the emergency plan of storage device, so that arrived full load state
Emergency node can still continue to read and write new data by the residual memory space of other memory nodes, but it is tight inevitably to will appear situation
It is anxious, it such as meets with data and stores full pressure period, storage device read-write data frequency is higher, and residual memory space is not enough to continue for a long time
Support.Therefore in order to avoid there is data-storage interrupt phenomenon, user's use is influenced, the management node in the present embodiment is also used to:
Generate the hint instructions for prompting the timely dilatation storage device of user；
The hint instructions are sent to response apparatus.
For example, the residual memory space amount of each memory node can be monitored in real time after storage device starts emergency plan,
After the residual memory space size for the memory node for detecting preset quantity is lower than preset value, hint instructions are just generated, are stored
Device can send to the devices such as exportable text or picture or sound according to the hint instructions and request, and make device output prompt letter
Breath or prompt tone, to notify the administrator of storage device to carry out dilatation in time to storage device.Certainly, the generation item of prompt information
Part might not be based on above-mentioned condition, such as can also be the size of data writing in storage device monitoring preset time, with this
Determine the generation time of prompt information.
Further, after administrator carries out dilatation to storage device, the first memory node in the present embodiment is also used to:
After the storage device dilatation, the data of first memory node stored in the emergency node are shifted
Into the memory space of first memory node；
Determine whether the data all shift；
If so, unmapped relationship and discharging the emergency memory space.
It is, it is remaining that the first memory node detects that local storage space occurs, at this time after storage device is by dilatation
Just the data in the emergency memory space in node of meeting an urgent need are transferred in local storage space.The data volume of transfer can be according to this
Depending on the surplus of secondary memory space, if current spatial surplus not enough receives the total data in emergency memory space can root
Keep the status after the data of transfer part for the time being according to actual conditions, that is, emergency node there are still.Or continue to notify administrator into
It is complete to shift this when the first memory node local storage space receives the total data in emergency memory space enough for row dilatation
Portion data, the mapping relations between the node that releases and meet an urgent need later, the node that makes to meet an urgent need when being determined as meeting an urgent need node by being switched to
Emergency rating restores to initial normal condition.The step is equally applicable to all depositing with aforementioned mapping relations in storage device
Between storage node.
It is apparent to those skilled in the art that for convenience and simplicity of description, the data of foregoing description
The electronic equipment that processing method is applied to, can be with reference to the corresponding description in before-mentioned products embodiment, and details are not described herein.
Above embodiments are only the exemplary embodiment of the application, are not used in limitation the application, the protection scope of the application
It is defined by the claims.Those skilled in the art can make respectively the application in the essence and protection scope of the application
Kind modification or equivalent replacement, this modification or equivalent replacement also should be regarded as falling within the scope of protection of this application.
Claims (10)
1. a kind of distributed storage method, comprising:
When detecting that the first memory node reaches full load state in storage device:
The status information of remaining memory node is obtained, the status information includes at least residual memory space information；
Determine the second memory node as the emergency for carrying out dilatation to first memory node according to the state information
Node.
2. the method according to claim 1, wherein the status information further includes the corresponding memory node
Memory space history use information；
It is described to determine the second memory node as carrying out dilatation to first memory node according to the state information
Emergency node specifically:
Candidate memory node is determined according to the residual memory space information；
The emergency node is determined from the candidate memory node according to the history use information.
3. method according to claim 1 or 2, which is characterized in that the status information further includes corresponding for showing
The memory node whether be the third memory node emergency node the first mark information；
It is described to determine the second memory node as carrying out dilatation to first memory node according to the state information
Emergency node specifically:
Candidate memory node is determined according to the residual memory space information；
The emergency node is determined from the candidate memory node according to first mark information.
4. the method according to claim 1, wherein further include:
It is determined according to the storage space information of the emergency node for storing data that first memory node is newly read and write
Emergency memory space；
The emergency memory space is mapped as to the memory space of first memory node.
5. according to the method described in claim 4, it is characterized by further comprising:
Determine whether second memory node has the second mark information, second mark information is for showing described second
Memory node has been the emergency node of first memory node；
If not having, it will determine that second memory node is the emergency node；
If having, the emergency memory space is expanded according to the residual memory space information of second memory node
Hold.
6. according to the method described in claim 4, it is characterized by further comprising:
Generate the hint instructions for prompting storage device described in the timely dilatation of user；
Respond the hint instructions.
7. according to the method described in claim 6, it is characterized in that, the method also includes:
After the storage device dilatation, the data of first memory node that store in the emergency node are shifted to institute
In the memory space for stating the first memory node；
Determine whether the data all shift；
If so, unmapped relationship and discharging the emergency memory space.
8. a kind of distributed storage devices, comprising:
Memory node, to be multiple, for cooperating distributed storage data；
Management node is used to obtain the shape of remaining memory node when detecting that the first memory node reaches full load state
State information, and determine the second memory node as the emergency section for carrying out dilatation to the first memory node according to status information
Point, wherein status information includes at least residual memory space information.
9. distributed storage devices according to claim 1, which is characterized in that the status information further includes corresponding institute
State the history use information of the memory space of memory node；
The management node determines candidate memory node according to the residual memory space information, and uses letter according to the history
Breath determines the emergency node from the candidate memory node.
10. distributed storage devices according to claim 8 or claim 9, which is characterized in that the status information further includes being used for
Show the corresponding memory node whether be the third memory node emergency node the first mark information；
The management node determines candidate memory node according to the residual memory space information, and is believed according to first label
Breath determines the emergency node from the candidate memory node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811117783.6A CN109407976B (en) | 2018-09-21 | 2018-09-21 | Distributed storage method and distributed storage device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811117783.6A CN109407976B (en) | 2018-09-21 | 2018-09-21 | Distributed storage method and distributed storage device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109407976A true CN109407976A (en) | 2019-03-01 |
| CN109407976B CN109407976B (en) | 2021-09-14 |
Family
ID=65466358
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811117783.6A Active CN109407976B (en) | 2018-09-21 | 2018-09-21 | Distributed storage method and distributed storage device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109407976B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110134332A (en) * | 2019-04-28 | 2019-08-16 | 平安科技（深圳）有限公司 | A kind of date storage method and relevant apparatus |
| CN111526315A (en) * | 2020-04-28 | 2020-08-11 | 杭州海康威视数字技术股份有限公司 | Video data storage method, device, system and equipment |
| CN112395469A (en) * | 2020-11-27 | 2021-02-23 | 中国银联股份有限公司 | Biological characteristic storage method, device, equipment and storage medium |
| CN112835527A (en) * | 2021-02-05 | 2021-05-25 | 西藏宁算科技集团有限公司 | Expansion method and device based on MINIO cloud storage and electronic equipment |
| CN113867643A (en) * | 2021-09-29 | 2021-12-31 | 北京金山云网络技术有限公司 | Data storage method, device, equipment and storage medium |
| WO2024061108A1 (en) * | 2022-09-21 | 2024-03-28 | Zhejiang Dahua Technology Co., Ltd. | Distributed storage systems and methods thereof, device and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102279777A (en) * | 2011-08-18 | 2011-12-14 | 成都市华为赛门铁克科技有限公司 | Method and device for processing data redundancy and distributed storage system |
| CN103259740A (en) * | 2013-04-27 | 2013-08-21 | 杭州华三通信技术有限公司 | Load balancing processing method and device |
| US20160306765A1 (en) * | 2015-04-17 | 2016-10-20 | Avago Technologies General Ip (Singapore) Pte. Ltd. | Low power parallelization to multiple output bus widths |
| CN107015883A (en) * | 2016-01-28 | 2017-08-04 | 中国移动通信集团上海有限公司 | A kind of dynamic data backup method and device |
| CN107707619A (en) * | 2017-08-28 | 2018-02-16 | 北京小米移动软件有限公司 | Distributed storage method and device |
| CN107864209A (en) * | 2017-11-17 | 2018-03-30 | 北京联想超融合科技有限公司 | The method, apparatus and server of data write-in |
| CN108023967A (en) * | 2017-12-20 | 2018-05-11 | 联想(北京)有限公司 | A kind of management equipment in data balancing method, apparatus and distributed memory system |
-
2018
- 2018-09-21 CN CN201811117783.6A patent/CN109407976B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102279777A (en) * | 2011-08-18 | 2011-12-14 | 成都市华为赛门铁克科技有限公司 | Method and device for processing data redundancy and distributed storage system |
| CN103259740A (en) * | 2013-04-27 | 2013-08-21 | 杭州华三通信技术有限公司 | Load balancing processing method and device |
| US20160306765A1 (en) * | 2015-04-17 | 2016-10-20 | Avago Technologies General Ip (Singapore) Pte. Ltd. | Low power parallelization to multiple output bus widths |
| CN107015883A (en) * | 2016-01-28 | 2017-08-04 | 中国移动通信集团上海有限公司 | A kind of dynamic data backup method and device |
| CN107707619A (en) * | 2017-08-28 | 2018-02-16 | 北京小米移动软件有限公司 | Distributed storage method and device |
| CN107864209A (en) * | 2017-11-17 | 2018-03-30 | 北京联想超融合科技有限公司 | The method, apparatus and server of data write-in |
| CN108023967A (en) * | 2017-12-20 | 2018-05-11 | 联想(北京)有限公司 | A kind of management equipment in data balancing method, apparatus and distributed memory system |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110134332A (en) * | 2019-04-28 | 2019-08-16 | 平安科技（深圳）有限公司 | A kind of date storage method and relevant apparatus |
| CN110134332B (en) * | 2019-04-28 | 2022-03-08 | 平安科技（深圳）有限公司 | Data storage method and related device |
| CN111526315A (en) * | 2020-04-28 | 2020-08-11 | 杭州海康威视数字技术股份有限公司 | Video data storage method, device, system and equipment |
| CN111526315B (en) * | 2020-04-28 | 2023-03-14 | 杭州海康威视数字技术股份有限公司 | Video data storage method, device, system and equipment |
| CN112395469A (en) * | 2020-11-27 | 2021-02-23 | 中国银联股份有限公司 | Biological characteristic storage method, device, equipment and storage medium |
| CN112835527A (en) * | 2021-02-05 | 2021-05-25 | 西藏宁算科技集团有限公司 | Expansion method and device based on MINIO cloud storage and electronic equipment |
| CN113867643A (en) * | 2021-09-29 | 2021-12-31 | 北京金山云网络技术有限公司 | Data storage method, device, equipment and storage medium |
| WO2024061108A1 (en) * | 2022-09-21 | 2024-03-28 | Zhejiang Dahua Technology Co., Ltd. | Distributed storage systems and methods thereof, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109407976B (en) | 2021-09-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109407976A (en) | A kind of distributed storage method and distributed storage devices | |
| CN106547612B (en) | Multitasking method and device | |
| CN102521265B (en) | Dynamic consistency control method in massive data management | |
| CN106527981A (en) | Configuration-based data fragmentation method for adaptive distributed storage system | |
| US20090089499A1 (en) | Method of managing storage capacity in a storage system, a storage device and a computer system | |
| EP3163446B1 (en) | Data storage method and data storage management server | |
| US8245012B2 (en) | Adaptively preventing out of memory conditions | |
| US10133757B2 (en) | Method for managing data using in-memory database and apparatus thereof | |
| CN104035836A (en) | Automatic disaster tolerance recovery method and system in cluster retrieval platform | |
| CN105843937A (en) | Data processing method and device | |
| CN111857592A (en) | Data storage method and device based on object storage system and electronic equipment | |
| US20100138627A1 (en) | Efficient alert mechanism for overutilization of storage resources | |
| CN106681656A (en) | Dynamically-adjustable cloud storage system fault tolerance method | |
| CN109710456A (en) | A kind of data reconstruction method and device | |
| WO2017118318A1 (en) | Data storage and service processing method and device | |
| CN114461407B (en) | Data processing method, data processing device, distribution server, data processing system, and storage medium | |
| CN109656684A (en) | A kind of partition method of Kafka, partition system and relevant apparatus | |
| US11544615B2 (en) | Managing runtime qubit allocation for executing quantum services | |
| CN102722450A (en) | Storage method for redundancy deletion block device based on location-sensitive hash | |
| CN107368355B (en) | Dynamic scheduling method and device of virtual machine | |
| CN112612605A (en) | Thread allocation method and device, computer equipment and readable storage medium | |
| CN116204311A (en) | Pod cluster capacity expansion and contraction method and device, computer equipment and storage medium | |
| CN106354828A (en) | Data fragmentation method and device for distributed database | |
| CN111158931A (en) | Dynamic data balancing method and device for Kafka partition and storage medium | |
| CN109976662B (en) | Data storage method and device and distributed storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |