CN109508325A - A kind of capacity control method and device of cluster file system - Google Patents
A kind of capacity control method and device of cluster file system Download PDFInfo
- Publication number
- CN109508325A CN109508325A CN201811347889.5A CN201811347889A CN109508325A CN 109508325 A CN109508325 A CN 109508325A CN 201811347889 A CN201811347889 A CN 201811347889A CN 109508325 A CN109508325 A CN 109508325A
- Authority
- CN
- China
- Prior art keywords
- capacity
- data
- file system
- cluster file
- object storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 230000005012 migration Effects 0.000 claims abstract description 32
- 238000013508 migration Methods 0.000 claims abstract description 32
- 230000002567 autonomic effect Effects 0.000 claims abstract description 28
- 238000001514 detection method Methods 0.000 claims description 25
- 238000012544 monitoring process Methods 0.000 claims description 12
- 230000004048 modification Effects 0.000 claims description 11
- 238000012986 modification Methods 0.000 claims description 11
- 230000007246 mechanism Effects 0.000 claims description 9
- 230000002159 abnormal effect Effects 0.000 claims description 8
- 230000007704 transition Effects 0.000 claims description 7
- GOLXNESZZPUPJE-UHFFFAOYSA-N spiromesifen Chemical compound CC1=CC(C)=CC(C)=C1C(C(O1)=O)=C(OC(=O)CC(C)(C)C)C11CCCC1 GOLXNESZZPUPJE-UHFFFAOYSA-N 0.000 claims description 3
- 238000004458 analytical method Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 6
- 230000009514 concussion Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000012217 deletion Methods 0.000 description 3
- 230000037430 deletion Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 239000000203 mixture Substances 0.000 description 2
- 230000008439 repair process Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 241000208340 Araliaceae Species 0.000 description 1
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 1
- 235000003140 Panax quinquefolius Nutrition 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 235000008434 ginseng Nutrition 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000004904 shortening Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1415—Saving, restoring, recovering or retrying at system level
- G06F11/1435—Saving, restoring, recovering or retrying at system level using file system or storage system metadata
Abstract
The present invention provides the capacity control method and device of a kind of cluster file system, this method comprises: carrying out the setting of forbidden data Autonomic Migration Framework to cluster file system；Deleting expires at least partly data of PG in the object storage device of state in capacity, expires state to release the capacity of object storage device；The setting of turn-on data Autonomic Migration Framework is carried out, to cluster file system at least partly data of PG to undelete.It can be seen that this method can be solved while guaranteeing data consistency data it is unbalanced caused by capacity overrun issues.
Description
Technical field
The present invention relates to field of computer technology, in particular to the capacity control method and dress of a kind of cluster file system
It sets.
Background technique
With the continuous increase to data volume demand, distributed storage technology is come into being, and is widely used in video prison
The every field such as control, broadcasting and TV matchmaker official communication, biotechnology, traffic operation.
Cluster file system (CFS, Cluster File System) is one of distributed storage technology, each
Multiple object storage devices (Object Storage Device, OSD), each object storage device are distributed on clustered node
For providing data service, and using the unified management of meta data server progress client and object storage device.It is applying
In the process, with the continuous expansion of system scale, the memory capacity of cluster file system is continuously increased, due to distributed storage system
There is data balancing in system, not can guarantee the harmony of the memory capacity in each object storage device, and when one
Or multiple object storage devices, when not starting normally, the data on the OSD can migrate, and may result in the magnetic of other OSD
Disk space capacity transfinites, and leads to the exception of clustered node, can not be normally carried out the operation of data write-in, and the positioning of failure and asks
The solution of topic requires longer time to complete, and influences system normal operation.
Summary of the invention
In view of this, the purpose of the present invention is to provide a kind of capacity control method of cluster file system and device, it can
With solved while guaranteeing data consistency data it is unbalanced caused by capacity overrun issues.
To achieve the above object, the present invention has following technical solution:
A kind of capacity control method of cluster file system, when obtain the cluster file system occur capacity transfinite it is different
When normal information, volume controlled is carried out, the exception information that the capacity transfinites includes: that multiple object storage devices are in capacity and expire shape
State, system can not be written data and client can normal carry, the method for the volume controlled includes:
The setting of forbidden data Autonomic Migration Framework is carried out to cluster file system；
Deleting expires at least partly data of PG in the object storage device of state in capacity, is deposited with releasing the object
The capacity of storage equipment expires state；
The setting of turn-on data Autonomic Migration Framework is carried out, to cluster file system at least partly PG's described in undeleting
Data.
Optionally it is determined that the method that capacity, which occurs, for cluster file system transfinites includes:
Obtain the detection information of the operating condition of cluster file system；
When the detection information is abnormal by proper transition, system detection message file and OSD log are divided
Analysis, if multiple object storage devices be in that capacity expires state, data can not be written in system and client can normal carry, recognize
Capacity, which occurs, for cluster file system transfinites.
Optionally, before deleting at least partly data in the object storage device that capacity expires state, further includes:
Carry out the backup of system configuration file；
Carry out the modification of anti-oscillating relevant parameter in system configuration file.
Optionally, the anti-oscillating relevant parameter includes:
Store one of quantity of documents parameter, thread timeout parameter, OSD heartbeat mechanism parameter and OSD monitoring parameter
Or it is a variety of.
Optionally, it deletes after expiring at least partly data of PG in the object storage device of state in capacity, to collection
Group's file system carries out before the setting of turn-on data Autonomic Migration Framework, further includes:
Restarting for the cluster file system is carried out, and is waited, until no data changes.
Optionally, after at least partly data of PG to undelete, further includes:
Carry out the data balancing of the cluster file system.
A kind of capacity control device of cluster file system, comprising:
For obtaining the cluster file system exception information that capacity transfinites occurs for exception information acquiring unit, described
The exception information that capacity transfinites includes: that multiple object storage devices are in that capacity expires state, data and client can not be written in system
End can normal carry；
Forbid migrating setting unit, for carrying out cluster file system the setting of forbidden data Autonomic Migration Framework；
Data delete unit, for deleting at least partly number of PG expired in the object storage device of state in capacity
According to expiring state to release the capacity of the object storage device；
Migration setting unit is opened, for carrying out the setting of turn-on data Autonomic Migration Framework to cluster file system, to restore
At least partly data of PG deleted.
Optionally, in the exception information acquiring unit, the method for determining that cluster file system generation capacity transfinites includes:
Obtain the detection information of the operating condition of cluster file system；
When the detection information is abnormal by proper transition, system detection message file and OSD log are divided
Analysis, if multiple object storage devices be in that capacity expires state, data can not be written in system and client can normal carry, recognize
Capacity, which occurs, for cluster file system transfinites.
Optionally, further includes:
Backup units, for carrying out the backup of system configuration file；
Configuration file modifies unit, for carrying out the modification of anti-oscillating relevant parameter in system configuration file.
Optionally, further includes:
Restart and wait unit, for carrying out restarting for the cluster file system, and waited, until no data becomes
It is dynamic.
The capacity control method and device of cluster file system provided in an embodiment of the present invention, by cluster file system
Carry out the setting of forbidden data Autonomic Migration Framework；Deleting expires at least partly number of PG in the object storage device of state in capacity
According to expiring state to release the capacity of object storage device；The setting of turn-on data Autonomic Migration Framework is carried out to cluster file system, with
At least partly data of PG to undelete.It can be seen that this method system occur capacity transfinite exception when, by forbidding
Data Autonomic Migration Framework；Then, deleting expires the partial data in the OSD of state in capacity, so that OSD releases the full shape of capacity
State；Finally, deleted data are backfilled, with solved while guaranteeing data consistency data it is unbalanced caused by
Capacity overrun issues.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is the present invention
Some embodiments for those of ordinary skill in the art without creative efforts, can also basis
These attached drawings obtain other attached drawings.
Fig. 1 shows a kind of determining cluster file system provided according to embodiments of the present invention and the method that capacity transfinites occurs
Flow chart；
Fig. 2 shows a kind of processes of the capacity control method of cluster file system of offer according to an embodiment of the present invention
Figure；
Fig. 3 shows a kind of method of the configuration file of modification cluster file system of offer according to an embodiment of the present invention
Flow chart；
Fig. 4 shows a kind of composition of the capacity control device of cluster file system of offer according to an embodiment of the present invention
Schematic diagram.
Specific embodiment
In order to make the foregoing objectives, features and advantages of the present invention clearer and more comprehensible, with reference to the accompanying drawing to the present invention
Specific embodiment be described in detail.
In the following description, numerous specific details are set forth in order to facilitate a full understanding of the present invention, but the present invention can be with
Implemented using other than the one described here other way, those skilled in the art can be without prejudice to intension of the present invention
In the case of do similar popularization, therefore the present invention is not limited by the specific embodiments disclosed below.
The embodiment of the present application can be applied in the scene that the capacity of cluster file system transfinites.
Based on the problems of the prior art, the embodiment of the present application provides a kind of volume controlled side of cluster file system
Method, to solve the problems of the prior art.
The capacity control method of a kind of cluster file system provided by the embodiments of the present application, when the acquisition group document system
When the exception information that capacity transfinites occurs for system, volume controlled is carried out.The exception information that the capacity transfinites includes: that multiple objects are deposited
Storage equipment is in that capacity expires state, data can not be written in system and client can normal carry.
The technical solution and technical effect of the application in order to better understand, first to the application based on cluster file system
(CFS, Cluster File System) is illustrated, and the CFS of the application is based on distributed storage technology, which includes
Multiple OSD, monitoring service (MON, Monitor) and Metadata Service (MDS, master are distributed on each clustered node
Copy of Cluster map), wherein put in order the minimum unit that group (PG, Placement Group) is object storage.
In the present embodiment, developer then may be used when obtaining the exception information that cluster file system generation capacity transfinites
It is controlled with the capacity to the cluster file system.Wherein, the exception information that capacity transfinites may include: multiple in system
Object storage device expires (full) state all in capacity, system can not be normally written data and client can normal carry,
Read the information of data.
In a kind of implementation of the present embodiment, referring to Fig. 1, the figure shows provided by the embodiments of the present application a kind of true
Determine cluster file system and the method flow diagram that capacity transfinites occur, may include steps of S101-S102:
S101: the detection information of the operating condition of cluster file system is obtained.
In the present embodiment, the fortune of system can be automatically detected by the monitoring service that cluster file system provides
Market condition, and generate the detection information of the operating condition of cluster file system.Then, available to this group file system of developer
The detection information of the operating condition of system.
S102: when detection information is abnormal by proper transition, system detection message file and OSD log are divided
Analysis, if multiple object storage devices be in that capacity expires state, data can not be written in system and client can normal carry, recognize
Capacity, which occurs, for cluster file system transfinites.
In the present embodiment, when the detection information of acquisition is abnormal by proper transition, then developer is to system detection
Message file and OSD log are analyzed, if learn by analysis multiple object storage devices be in capacity expire state, with
And system can not be written data and client can normal carry, it may be considered that capacity, which occurs, for cluster file system transfinites.Its
In, by analysis system detection information file and OSD log, the OSD of state can be expired to be in capacity in quick positioning system,
And then determine that cluster file system occurs capacity and transfinites, the tedious steps of fault location are optimized, the time required to shortening positioning.
In concrete implementation scene, when the service of CFS system is abnormal, system mode can be by normal (OK) state
Exception (WARN or ERR) state is converted to, then, developer can be by system detection message file and OSD log
It is analyzed, quickly to position the usage amount of OSD, and determines the OSD for expiring state in system in capacity.If by analysis really
Fixed multiple OSD be in capacity expire state and system can not be written data and client can normal carry, it may be considered that collecting
Capacity, which occurs, for group's file system transfinites.In general, these, which are in capacity and expire the OSD of state, concentrates on a cluster section
On point, belong to a failure domain.
When obtaining the exception information that cluster file system generation capacity transfinites, volume controlled can be carried out.Referring to fig. 2,
The figure shows a kind of flow charts of the capacity control method of cluster file system, may include steps of S201-S203:
S201: the setting of forbidden data Autonomic Migration Framework is carried out to cluster file system.
In actual scene, object storage device in the state that capacity is full can automatically by the Data Migration of itself to other
Object storage device in.Therefore, in this embodiment, occur data Autonomic Migration Framework in order to prevent, it can be to cluster file system
Carry out the setting of forbidden data Autonomic Migration Framework.
In concrete implementation scene, can by cluster file system be arranged " not restoring (norecover) " and
The mode of " not backfilling (nobackfill) " order, the setting of forbidden data Autonomic Migration Framework is carried out to cluster file system.Such as:
The state that do not restore can be set by the order of " cfs osd set norecover " in cluster file system, passed through
The state not backfilled can be arranged in the order of " cfs osd set norebackfill " in cluster file system.
S202: deleting expires at least partly data of PG in the object storage device of state in capacity, described in releasing
The capacity of object storage device expires state.
In the present embodiment, the data of the part PG in the object storage device that capacity expires state can be deleted, with
The capacity for releasing object storage device expires state.The quantity of deletion can according to need to be arranged, for example can delete in appearance
Measure the data of 20% PG in the object storage device of full state.Wherein, putting in order group (Placement group, PG) can be with
It is the virtual unit for data storage created in the storage pool of cluster file system, storage pool can be in group document
Disk in system for storing data.
In a kind of implementation of the present embodiment, deleting in the object storage device that capacity expires state at least
Before partial data, referring to Fig. 3, the figure shows a kind of configuration texts for modifying cluster file system provided by the embodiments of the present application
The method flow diagram of part, may include steps of S301-S302:
S301: the backup of system configuration file is carried out.
In the present embodiment, before the configuration file to cluster file system is modified, it can first carry out system and match
Set the backup of file, with prevent from appearing in modification during fault is directed into former configuration file the problem of.
S302: the modification of anti-oscillating relevant parameter in system configuration file is carried out.
In the present embodiment, it can modify to anti-oscillating relevant parameter in system configuration file, to guarantee that system exists
The problem of being not in OSD concussion during reparation, and then avoid to be caused in system repair process by OSD concussion and be
There are other failures in system.Wherein, OSD concussion refer to OSD repeatedly carry out unlatching operation (OSD up) and (OSD out of service
Down operation).
In concrete implementation scene, anti-oscillating parameter can be increased in system configuration file, and delete it is all with
The related parameter of full.
In a kind of implementation of the present embodiment, anti-oscillating relevant parameter includes:
Store one of quantity of documents parameter, thread timeout parameter, OSD heartbeat mechanism parameter and OSD monitoring parameter
Or it is a variety of.
In the present embodiment, anti-oscillating relevant parameter includes: storage quantity of documents parameter, thread timeout parameter, OSD heartbeat
Scheme parameters and OSD monitoring parameter.One or more in aforementioned anti-oscillating relevant parameter system can be added to match
It sets in file.Wherein, storage quantity of documents parameter determines under a catalogue, the number of stored target file, thread time-out ginseng
Number is that OSD operates relevant thread parameter, and OSD heartbeat mechanism parameter is the relevant parameter of OSD heartbeat detection, OSD monitoring parameter
For parameter information relevant to report, election, request, extension lease and/or replacement MDS in OSD monitoring, guarantee restoring data
In the process, the problem of being not in OSD concussion, the generation of other failures when avoiding repair data.
In specific application scenarios, in one example, the setting of anti-oscillating relevant parameter, such as can be according to as follows
Setting carries out:
Filestore_split_multiple=100；//PG subdirectory divides multiplier
Filestore_merge_threshold=500；The minimum number of files that //PG subdirectory merges
Osd_op_thread_timeout=300；//OSD operates thread time-out time
Threadpool_default_timeout=800；// thread pool time default timeout
Osd_heartbeat_interval=60；// heartbeat mechanism interval
Osd_heartbeat_grace=300；// heartbeat mechanism the grace period
Mon_osd_min_down_reporters=10；OSD is out of service reports minimum value for the monitoring of //OSD finger daemon
mon_osd_min_down_reports；The minimum received reporting quantities out of service of // mono- OSD
Mon_lease_ack_timeout=200；The time-out time that // time-out re-elects
Osd_mon_ack_timeout=100；// know waiting time of static requests
Mon_election_timeout=100；// election contest control the time
Mon_lease=100；// extend lease time
Mds_beacon_grace=30；// time for identifying message and needing to replace MDS is not received
Wherein, storage quantity of documents parameter may include the minimum file that PG subdirectory division multiplier and PG subdirectory merge
Number, the number of stored target file under a catalogue is determined by the two parameters；Thread timeout parameter may include OSD behaviour
Make thread time-out time and thread pool time default timeout；OSD heartbeat mechanism parameter may include heartbeat mechanism interval and heartbeat
The mechanism grace period, by the operation of the two parameter monitorings OSD, in this example, every 60 seconds transmission heartbeat messages, if being more than
300 seconds time confiscated heartbeat message, then indicted that the OSD has failed；OSD monitoring parameter includes the monitoring of OSD finger daemon
The OSD time-out out of service for reporting minimum value, the minimum received reporting quantities out of service of an OSD, time-out to re-elect
Time, the election contest control time, extends lease time and does not receive mark message and need the waiting time for knowing static requests
Replace the time of MDS.
S203: carrying out the setting of turn-on data Autonomic Migration Framework to cluster file system, at least portion described in undeleting
Divide the data of PG.
In the present embodiment, release object storage device capacity expire state after, can to cluster file system into
The setting of row turn-on data Autonomic Migration Framework, so that the data of the PG deleted store in data and restore in less OSD.Such as it can be with
Aforementioned 20% deleted data are restored to other data to store in less OSD.It completes to having deleted the extensive of data
After multiple, cluster file system can be waited to restore calm.In this way, deleted copy is backfilled, and then guarantee data
Consistency.
It, can be by the way that " restoring (recover) " and " backfill be arranged in cluster file system in concrete implementation scene
(backfill) " mode ordered, the setting to cluster file system turn-on data Autonomic Migration Framework.Such as: pass through " cfs osd
The state of recovery can be arranged in the order of set recover " in cluster file system, pass through " cfs osd set
The state of backfill can be arranged in the order of rebackfill " in cluster file system.And it completes to the extensive of deletion data
After multiple, it can be ordered by watch cfs-s and cluster file system is waited to restore calm.
In a kind of implementation of the present embodiment, deleting expires at least portion in the object storage device of state in capacity
Divide after the data of PG, before the setting that turn-on data Autonomic Migration Framework is carried out to cluster file system, further includes: carry out cluster text
Part system is restarted, and is waited, until cluster file system enters the state that stable, no data changes.
In the present embodiment, after carrying out step S202, before step S203, cluster file system can also be carried out
Restart, and waited, until no data changes.
In concrete implementation mode, it can be ordered by watch cfs-s and continuous observation is carried out to cluster, to guarantee to collect
Group enters the state that stable, no data changes.
In a kind of implementation of the present embodiment, after at least partly data of PG to undelete, further includes: carry out
The data balancing of cluster file system.
In the present embodiment, after at least partly data of PG to undelete, cluster file system can also be carried out
Data balancing, the data capacity to guarantee each clustered node OSD in system is balanced.
In specific Scene realization, the data of system can be realized by reweight_by_crushtool.sh script
It is balanced.In reweight_by_crushtool.sh script include #reweight_by_crushtool.sh<num_rep><
Pg_num><pool_name>order, which can generate log in the implementation procedure of script, and be stored in/var/
Under log/icfs/reweight_log/ catalogue.Wherein, the num_rep in order indicates to carry out the storage pool of capacity equilibrium
Number of copies, pg_num indicate that the PG number of the storage pool, pool_name are the storage Pool names.It, can be with after script execution success
Corresponding message is exported on the screen, otherwise can export the error message of culture error reason.
To sum up, the capacity control method of cluster file system provided in an embodiment of the present invention, by cluster file system
Carry out the setting of forbidden data Autonomic Migration Framework；Deleting expires at least partly number of PG in the object storage device of state in capacity
According to expiring state to release the capacity of object storage device；The setting of turn-on data Autonomic Migration Framework is carried out to cluster file system, with
At least partly data of PG to undelete.It can be seen that this method passes through forbidden data Autonomic Migration Framework；Then, deletion is in
Capacity expires the partial data in the OSD of state, so that OSD releases the full state of capacity；Finally, by deleted data to guarantee
The principle of data consistency is backfilled, with solved while guaranteeing data consistency data it is unbalanced caused by capacity it is super
Limit problem.
Referring to fig. 4, the figure shows a kind of composition schematic diagram of the capacity control device of cluster file system, described devices
Include:
For obtaining the cluster file system exception information that capacity transfinites, institute occur for exception information acquiring unit 401
The exception information that stating capacity transfinites includes: that multiple object storage devices are in that capacity expires state, data and visitor can not be written in system
It family end can normal carry；
Forbid migrating setting unit 402, for carrying out cluster file system the setting of forbidden data Autonomic Migration Framework；
Data delete unit 403, for deleting at least partly PG's expired in the object storage device of state in capacity
Data expire state to release the capacity of the object storage device；
Migration setting unit 404 is opened, for carrying out the setting of turn-on data Autonomic Migration Framework to cluster file system, with extensive
At least partly data of PG deleted again.
In a kind of implementation of the present embodiment, in the exception information acquiring unit 401, cluster file system is determined
The method that transfinites of capacity, which occurs, includes:
Obtain the detection information of the operating condition of cluster file system；
When the detection information is abnormal by proper transition, system detection message file and OSD log are divided
Analysis, if multiple object storage devices be in that capacity expires state, data can not be written in system and client can normal carry, recognize
Capacity, which occurs, for cluster file system transfinites.
In a kind of implementation of the present embodiment, further includes:
Backup units, for carrying out the backup of system configuration file；
Configuration file modifies unit, for carrying out the modification of anti-oscillating relevant parameter in system configuration file.
In a kind of implementation of the present embodiment, further includes:
Restart and wait unit, for carrying out restarting for the cluster file system, and waited, until no data becomes
It is dynamic.
The above is only a preferred embodiment of the present invention, although the present invention has been disclosed in the preferred embodiments as above, so
And it is not intended to limit the invention.Anyone skilled in the art is not departing from technical solution of the present invention ambit
Under, many possible changes and modifications all are made to technical solution of the present invention using the methods and technical content of the disclosure above,
Or equivalent example modified to equivalent change.Therefore, anything that does not depart from the technical scheme of the invention, according to the present invention
Technical spirit any simple modification, equivalent variation and modification made to the above embodiment, still fall within the technology of the present invention side
In the range of case protection.
Claims (10)
1. a kind of capacity control method of cluster file system, which is characterized in that hold when obtaining the cluster file system
When measuring the exception information to transfinite, volume controlled is carried out, the exception information that the capacity transfinites includes: at multiple object storage devices
In capacity expires state, data can not be written in system and client can normal carry, the method for the volume controlled includes:
The setting of forbidden data Autonomic Migration Framework is carried out to cluster file system；
Deleting expires at least partly data of PG in the object storage device of state in capacity, is set with releasing the object storage
Standby capacity expires state；
The setting of turn-on data Autonomic Migration Framework is carried out, to cluster file system at least partly number of PG described in undeleting
According to.
2. the method according to claim 1, wherein determining that the method packet that capacity transfinites occurs for cluster file system
It includes:
Obtain the detection information of the operating condition of cluster file system；
When the detection information is abnormal by proper transition, system detection message file and OSD log are analyzed, if
Multiple object storage devices are in that capacity expires state, data can not be written in system and client can normal carry, then it is assumed that collection
Capacity, which occurs, for group's file system transfinites.
3. the method according to claim 1, wherein deleting in the object storage device that capacity expires state
At least partly data before, further includes:
Carry out the backup of system configuration file；
Carry out the modification of anti-oscillating relevant parameter in system configuration file.
4. according to the method described in claim 3, it is characterized in that, the anti-oscillating relevant parameter includes:
Store one of quantity of documents parameter, thread timeout parameter, OSD heartbeat mechanism parameter and OSD monitoring parameter or more
Kind.
5. the method according to claim 1, wherein deleting expires in the object storage device of state in capacity
At least partly after the data of PG, before the setting that turn-on data Autonomic Migration Framework is carried out to cluster file system, further includes:
Restarting for the cluster file system is carried out, and is waited, until no data changes.
6. the method according to claim 1, wherein after at least partly data of PG to undelete,
Further include:
Carry out the data balancing of the cluster file system.
7. a kind of capacity control device of cluster file system characterized by comprising
For obtaining the cluster file system exception information that capacity transfinites, the capacity occur for exception information acquiring unit
The exception information to transfinite includes: that multiple object storage devices are in that capacity expires state, data can not be written in system and client can
With normal carry；
Forbid migrating setting unit, for carrying out cluster file system the setting of forbidden data Autonomic Migration Framework；
Data delete unit, for deleting at least partly data of PG expired in the object storage device of state in capacity, with
The capacity for releasing the object storage device expires state；
Migration setting unit is opened, for carrying out the setting of turn-on data Autonomic Migration Framework to cluster file system, to undelete
At least partly data of PG.
8. device according to claim 7, which is characterized in that in the exception information acquiring unit, determine group document
The method that transfinites of capacity occurs for system
Obtain the detection information of the operating condition of cluster file system；
When the detection information is abnormal by proper transition, system detection message file and OSD log are analyzed, if
Multiple object storage devices are in that capacity expires state, data can not be written in system and client can normal carry, then it is assumed that collection
Capacity, which occurs, for group's file system transfinites.
9. device according to claim 7, which is characterized in that further include:
Backup units, for carrying out the backup of system configuration file；
Configuration file modifies unit, for carrying out the modification of anti-oscillating relevant parameter in system configuration file.
10. device according to claim 7, which is characterized in that further include:
Restart and wait unit, for carrying out restarting for the cluster file system, and waited, until no data changes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811347889.5A CN109508325A (en) | 2018-11-13 | 2018-11-13 | A kind of capacity control method and device of cluster file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811347889.5A CN109508325A (en) | 2018-11-13 | 2018-11-13 | A kind of capacity control method and device of cluster file system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109508325A true CN109508325A (en) | 2019-03-22 |
Family
ID=65748274
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811347889.5A Withdrawn CN109508325A (en) | 2018-11-13 | 2018-11-13 | A kind of capacity control method and device of cluster file system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109508325A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110502496A (en) * | 2019-07-19 | 2019-11-26 | 苏州浪潮智能科技有限公司 | A kind of distributed file system restorative procedure, system, terminal and storage medium |
| CN111290909A (en) * | 2020-01-19 | 2020-06-16 | 山东汇贸电子口岸有限公司 | System and method for monitoring and alarming ceph cluster |
| CN111680015A (en) * | 2020-05-29 | 2020-09-18 | 北京百度网讯科技有限公司 | File resource processing method, device, equipment and medium |
| CN112162886A (en) * | 2020-09-18 | 2021-01-01 | 北京浪潮数据技术有限公司 | Method, device, equipment and medium for switching back-end storage equipment |
| CN112363980A (en) * | 2020-11-03 | 2021-02-12 | 网宿科技股份有限公司 | Data processing method and device for distributed system |
-
2018
- 2018-11-13 CN CN201811347889.5A patent/CN109508325A/en not_active Withdrawn
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110502496A (en) * | 2019-07-19 | 2019-11-26 | 苏州浪潮智能科技有限公司 | A kind of distributed file system restorative procedure, system, terminal and storage medium |
| CN110502496B (en) * | 2019-07-19 | 2022-10-18 | 苏州浪潮智能科技有限公司 | Distributed file system repair method, system, terminal and storage medium |
| CN111290909A (en) * | 2020-01-19 | 2020-06-16 | 山东汇贸电子口岸有限公司 | System and method for monitoring and alarming ceph cluster |
| CN111680015A (en) * | 2020-05-29 | 2020-09-18 | 北京百度网讯科技有限公司 | File resource processing method, device, equipment and medium |
| CN111680015B (en) * | 2020-05-29 | 2023-08-11 | 北京百度网讯科技有限公司 | File resource processing method, device, equipment and medium |
| CN112162886A (en) * | 2020-09-18 | 2021-01-01 | 北京浪潮数据技术有限公司 | Method, device, equipment and medium for switching back-end storage equipment |
| CN112162886B (en) * | 2020-09-18 | 2023-12-22 | 北京浪潮数据技术有限公司 | Back-end storage device switching method, device, equipment and medium |
| CN112363980A (en) * | 2020-11-03 | 2021-02-12 | 网宿科技股份有限公司 | Data processing method and device for distributed system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109508325A (en) | A kind of capacity control method and device of cluster file system | |
| US10896102B2 (en) | Implementing secure communication in a distributed computing system | |
| US10642694B2 (en) | Monitoring containers in a distributed computing system | |
| CN109683826B (en) | Capacity expansion method and device for distributed storage system | |
| US10698866B2 (en) | Synchronizing updates across cluster filesystems | |
| US9858322B2 (en) | Data stream ingestion and persistence techniques | |
| US9794135B2 (en) | Managed service for acquisition, storage and consumption of large-scale data streams | |
| US10089148B1 (en) | Method and apparatus for policy-based replication | |
| KR20140038450A (en) | Automatic configuration of a recovery service | |
| US10620871B1 (en) | Storage scheme for a distributed storage system | |
| CN114780252B (en) | Resource management method and device of data warehouse system | |
| US7506117B2 (en) | Data recovery method for computer system | |
| CN111538719A (en) | Data migration method, device, equipment and computer storage medium | |
| CN114490677A (en) | Data synchronization in a data analysis system | |
| CN105354102B (en) | A kind of method and apparatus of file system maintenance and reparation | |
| CN112235405A (en) | Distributed storage system and data delivery method | |
| CN110647425A (en) | Database recovery method and device | |
| CN109947730A (en) | Metadata restoration methods, device, distributed file system and readable storage medium storing program for executing | |
| CN105988898A (en) | System backup device and backup method | |
| CN112579550B (en) | Metadata information synchronization method and system of distributed file system | |
| CN111338751B (en) | Cross-pool migration method and device for data in same ceph cluster | |
| CN113722154B (en) | Data management method and system, monitoring server and storage medium | |
| US20080313326A1 (en) | Information Processor and Information Processing System | |
| CN111352916B (en) | Data storage method, system and storage medium based on NAS storage system | |
| CN109947592A (en) | A kind of method of data synchronization, device and relevant device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20190322 |
|
| WW01 | Invention patent application withdrawn after publication |