CN105528053A - Management method and device for distributed system - Google Patents
Management method and device for distributed system Download PDFInfo
- Publication number
- CN105528053A CN105528053A CN201510923687.0A CN201510923687A CN105528053A CN 105528053 A CN105528053 A CN 105528053A CN 201510923687 A CN201510923687 A CN 201510923687A CN 105528053 A CN105528053 A CN 105528053A
- Authority
- CN
- China
- Prior art keywords
- group
- energy
- service
- disk
- conservation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
- G06F1/26—Power supply means, e.g. regulation thereof
- G06F1/32—Means for saving power
- G06F1/3203—Power management, i.e. event-based initiation of a power-saving mode
- G06F1/3234—Power saving characterised by the action undertaken
- G06F1/325—Power saving in peripheral device
- G06F1/3275—Power saving in memory, e.g. RAM, cache
Abstract
The invention discloses a management method and device for a distributed system. The distributed system comprises meta data service nodes and data nodes; the method comprises the following steps: grouping all disks in the distributed system to obtain multiple service groups and multiple energy saving groups; transferring read-write businesses of the multiple energy saving groups to the multiple service groups, and performing deceleration or dormancy on the disks in the multiple energy saving groups; enabling the multiple service groups to run the businesses at full speed. According to the management method and device, all the disks in the distributed system are divided into the plurality of service groups and the plurality of energy saving groups, the read-write businesses of the multiple energy saving groups are transferred into the multiple service groups, deceleration or dormancy is performed on the disks in the energy saving groups, and the service groups are enabled to rotate at full speed and normally serve; the problem of long business switching time caused by whole node shutdown for energy conservation is avoided, and energy conservation processing on the system can be effectively performed by using business characteristics.
Description
Technical field
The present invention relates to computer realm, particularly, relate to a kind of management method and device of distributed system.
Background technology
Distributed file system generally comprises client, meta data server and data server, and the access interface that client is responsible for file data is formulated, the layout of meta data server process file and attribute, the data content of data server storage file.
For distributed file system, can store mass data is its topmost feature, and mass data is just along with the existence of large batch of disk and back end, and the disk energy consumption in back end is very large, and the business of system is often according to the difference of time period and pressure varies in size, traffic pressure hour, a large amount of data disks is at light condition, create much unnecessary power consumption, directly cause the increase of the use cost of distributed file system.
For this problem, the solution bill that prior art adopts is different Physical Extents by different node division, node in subregion is shut down or machine energy-saving, but because node whole in this method is in shutdown and power save mode, when so accessing the data on it when service needed, its by shut down or power save mode transfer to switching time of full speed operation state can be long, so will directly cause the delay of upper access time to be amplified.
For the problems referred to above in correlation technique, at present effective solution is not yet proposed.
Summary of the invention
For the problems referred to above in correlation technique, the present invention proposes a kind of management method and device of distributed system, can avoid the problem of energy-conservation business length switching time brought of whole node shutdown, and can effectively utilize service feature to carry out the energy-efficient treatment of system.
Technical scheme of the present invention is achieved in that
According to an aspect of the present invention, provide a kind of management method of distributed system, wherein, distributed system comprises Metadata Service node, back end.
This management method comprises:
Disks all in distributed system are divided into groups, obtains multiple service group and multiple energy-conservation group;
By the read-write business migration of multiple energy-conservation group to multiple service group, and reduction of speed or dormancy are carried out to the disk in multiple energy-conservation group;
Make the business at multiple service group full speed running place.
Wherein, when all disks in distributed system are divided into groups, grouping according to comprise following one of at least:
Same file is arranged in same disk group;
The pressure of business;
The access mode of business.
In addition, this management method comprises further:
When reading and writing data to back end, back end is added up the traffic pressure value of each disk group corresponding to this back end and is sent to Metadata Service node;
The each pressure threshold of the traffic pressure value of each disk group with corresponding each disk group compares by Metadata Service node respectively;
When the traffic pressure value of disk group is greater than corresponding pressure threshold, this disk group is switched to energy-conservation group.
Optionally, this management method comprises further:
When writing new file to back end, new file is written in the disk in the same service group of this back end by meta data server.
Optionally, this management method comprises further:
When the file read from back end is arranged in the disk of energy-conservation group, back end by the multiple Suspend To Disks in this energy-conservation group to provide service.
Optionally, this management method comprises further:
The capacity service condition of each disk group corresponding to each back end of metadata node taken at regular intervals;
The difference of the active volume of the active volume of service group and energy-conservation group and capacity threshold are made comparisons;
When difference exceed capacity threshold value, this service group and this energy-conservation group are carried out state switching.
Wherein, when the difference of the active volume by the active volume of service group and energy-conservation group and capacity threshold are made comparisons, the difference of the active volume of the active volume and energy-conservation group that are subordinated to the service group of same back end and capacity threshold can be made comparisons.
According to a further aspect in the invention, provide a kind of management devices of distributed system, wherein, distributed system comprises Metadata Service node, back end.
This management devices comprises:
Grouping module, for dividing into groups to disks all in distributed system, obtains multiple service group and multiple energy-conservation group;
Modular converter, for by the read-write business migration of multiple energy-conservation group to multiple service group, and reduction of speed or dormancy are carried out to the disk in multiple energy-conservation group;
Run module, for making the business at multiple service group full speed running place.
In addition, this management devices comprises further:
Statistical module, for when reading and writing data to back end, adding up the traffic pressure value of each disk group corresponding to this back end and being sent to Metadata Service node;
First comparison module, for comparing each pressure threshold of the traffic pressure value of each disk group with corresponding each disk group respectively;
First handover module, for being greater than corresponding pressure threshold in the traffic pressure value of disk group, switches to energy-conservation group by this disk group.
Optionally, this management devices comprises further:
Acquisition module, for the capacity service condition of each disk group corresponding to each back end of taken at regular intervals;
Second comparison module, makes comparisons for the difference of the active volume by the active volume of service group and energy-conservation group and capacity threshold;
Second handover module, for when difference exceed capacity threshold value, carries out state switching by this service group and this energy-conservation group.
The present invention is by being divided into several service groups and energy-conservation group to disks all in distributed system, and service group of the read-write business of energy-conservation group being moved into, and to disk reduction of speed or dormancy in energy-conservation group, makes service packet rotate normal service at full speed.Both avoid the problem of energy-conservation business length switching time brought of whole node shutdown, and can effectively utilize again service feature to carry out the energy-efficient treatment of system.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment below, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
Fig. 1 is the process flow diagram of the management method of distributed system according to the embodiment of the present invention;
Fig. 2 is the process flow diagram of the management method of distributed system according to another embodiment of the present invention;
Fig. 3 is the block diagram of the management devices of distributed system according to the embodiment of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, be clearly and completely described the technical scheme in the embodiment of the present invention, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, the every other embodiment that those of ordinary skill in the art obtain, all belongs to the scope of protection of the invention.
According to embodiments of the invention, provide a kind of management method of distributed system, wherein, distributed system comprises Metadata Service node, back end, client node.
As shown in Figure 1, comprise according to the management method of the embodiment of the present invention:
Step S101, divides into groups to disks all in distributed system, obtains multiple service group and multiple energy-conservation group;
Step S103, by the read-write business migration of multiple energy-conservation group to multiple service group, and carries out reduction of speed or dormancy to the disk in multiple energy-conservation group;
Step S103, makes the business at multiple service group full speed running place.
Wherein, in one embodiment, when dividing into groups to all disks in distributed system, grouping according to comprise following one of at least:
Same file is arranged in same disk group;
The pressure of business;
The access mode of business.
Technique scheme for a better understanding of the present invention, is described in detail technique scheme of the present invention below in conjunction with the instantiation shown in Fig. 2.
As can be seen from Figure 2, technical scheme of the present invention is achieved in that
1, the disk of each back end is divided into several groupings by Metadata Service node, when selecting file layout, makes same file only fall into same grouping;
2, client node is to back end write or reading data;
3, back end is added up current traffic pressure (i.e. the traffic pressure value of each disk group that this back end is corresponding) and is sent to meta data server;
4, Metadata Service node regularly obtains the current traffic pressure value of its statistics to back end;
And according to the processing power of each disk stripping, select whether can energy-conservationly (to be specially: each pressure threshold of the traffic pressure value of each disk group with corresponding each disk group compares by Metadata Service node respectively; When the traffic pressure value of disk group is greater than corresponding pressure threshold, this disk group is switched to energy-conservation group, namely selecting can be energy-conservation; Otherwise do not switch);
If 5 can be energy-conservation, Metadata Service node then sends the energy-conservation order of intended target disk group (namely traffic pressure value is greater than the disk group of pressure threshold) to back end;
6, after back end receives this order, call the order of disk reduction of speed, make this group disk enter power save mode;
7, when there being new document creation to write, this new file is chosen in the disk stripping (i.e. service group) that state is service state by meta data server, and, this new file is written in the disk in the same service group of this back end;
If 8 client-access existing file falls into energy-conservation group;
9, back end is automatically by the multiple Suspend To Disks in energy-conservation group, provides service.
In addition, in another embodiment, cause that its capacity utilization differs too large with the capacity rate of energy-conservation group and performance that is that bring is unbalanced for preventing some disk to be in service state always, the management method according to the embodiment of the present invention also can comprise:
The capacity service condition of each disk group corresponding to each back end of collection of all meta data servers (i.e. metadata node) meeting timing;
The difference of the active volume of the active volume of service group and energy-conservation group and capacity threshold are made comparisons;
When difference exceed capacity threshold value, this service group and this energy-conservation group are carried out state switching.
Wherein, in one embodiment, when the difference of the active volume by the active volume of service group and energy-conservation group and capacity threshold are made comparisons, the difference of the active volume of the active volume and energy-conservation group that are subordinated to the service group of same back end and capacity threshold can be made comparisons.
In addition, for the number of disks that each disk group comprises, in one embodiment, the number of disks of each disk group can be determined according to the performance of the data redundancy proportioning of Operation system setting and disk reparation, in the present embodiment, the number of disks of each disk group can be set in and be not less than 96 pieces of disks and the half being not more than total disk number.
Pass through above-described embodiment, the present invention is by being divided into different disk strippings to disks all in distributed system, according to pressure and the access mode of business, disk stripping is divided into several service packet and energy-conservation grouping, and carry out state rotation according to the use capacity of disk stripping and uninterrupted service time, for energy-conservation grouping by disk reduction of speed or dormancy in it, service packet of read-write business being moved into, service packet rotates normal service at full speed.Reach meticulous and efficient energy conservation object by this power save mode, this kind of power-economizing method, both avoided the problem of energy-conservation business length switching time brought of whole node shutdown, and can effectively utilize again service feature to carry out the energy-efficient treatment of system.
According to embodiments of the invention, additionally provide a kind of management devices of distributed system, wherein, distributed system comprises Metadata Service node, back end.
As shown in Figure 3, comprise according to the management devices of the embodiment of the present invention:
Grouping module 31, for dividing into groups to disks all in distributed system, obtains multiple service group and multiple energy-conservation group;
Modular converter 32, for by the read-write business migration of multiple energy-conservation group to multiple service group, and reduction of speed or dormancy are carried out to the disk in multiple energy-conservation group;
Run module 33, for making the business at multiple service group full speed running place.
In addition, in one embodiment, comprise further according to the management devices of the embodiment of the present invention:
Statistical module (not shown), for when reading and writing data to back end, adding up the traffic pressure value of each disk group corresponding to this back end and being sent to Metadata Service node;
First comparison module (not shown), for comparing each pressure threshold of the traffic pressure value of each disk group with corresponding each disk group respectively;
First handover module (not shown), for being greater than corresponding pressure threshold in the traffic pressure value of disk group, switches to energy-conservation group by this disk group.
Optionally, in one embodiment, comprise further according to the management devices of the embodiment of the present invention:
Acquisition module (not shown), for the capacity service condition of each disk group corresponding to each back end of taken at regular intervals;
Second comparison module (not shown), makes comparisons for the difference of the active volume by the active volume of service group and energy-conservation group and capacity threshold;
Second handover module (not shown), for when difference exceed capacity threshold value, carries out state switching by this service group and this energy-conservation group.
In sum, by means of technique scheme of the present invention, by being divided into several service groups and energy-conservation group to disks all in distributed system, and service group that the read-write business of energy-conservation group is moved into, and to disk reduction of speed or dormancy in energy-conservation group, make service packet rotate normal service at full speed.Both avoided the problem of energy-conservation business length switching time brought of whole node shutdown, and service feature effectively can have been utilized again to carry out the energy-efficient treatment of system, and effectively can reduce unnecessary disk power dissipation overhead, the service efficiency of improving product.
The foregoing is only preferred embodiment of the present invention, not in order to limit the present invention, within the spirit and principles in the present invention all, any amendment done, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1. a management method for distributed system, described distributed system comprises Metadata Service node, and back end and client node, is characterized in that, comprising:
Disks all in described distributed system are divided into groups, obtains multiple service group and multiple energy-conservation group;
By the read-write business migration of described multiple energy-conservation group to described multiple service group, and reduction of speed or dormancy are carried out to the disk in described multiple energy-conservation group;
Make the business at described multiple service group full speed running place.
2. management method according to claim 1, is characterized in that, when all disks in described distributed system are divided into groups, grouping according to comprise following one of at least:
Same file is arranged in same disk group;
The pressure of business;
The access mode of business.
3. management method according to claim 1, is characterized in that, comprises further:
When reading and writing data to described back end, described back end is added up the traffic pressure value of each disk group corresponding to this back end and is sent to Metadata Service node;
The each pressure threshold of the traffic pressure value of each disk group with corresponding each disk group compares by described Metadata Service node respectively;
When the traffic pressure value of disk group is greater than corresponding pressure threshold, this disk group is switched to described energy-conservation group.
4. management method according to claim 1, is characterized in that, comprises further:
When writing new file to described back end, described new file is written in the disk in the same service group of this back end by meta data server.
5. management method according to claim 1, is characterized in that, comprises further:
When the file read from described back end is arranged in the disk of described energy-conservation group, described back end by the multiple Suspend To Disks in this energy-conservation group to provide service.
6. management method according to claim 1, is characterized in that, comprises further:
The capacity service condition of each disk group corresponding to each back end of described metadata node taken at regular intervals;
The difference of the active volume of the active volume of service group and energy-conservation group and capacity threshold are made comparisons;
When described difference exceedes described capacity threshold, this service group and this energy-conservation group are carried out state switching.
7. management method according to claim 6, is characterized in that, the difference of the active volume of the active volume of service group and energy-conservation group and capacity threshold being made comparisons comprises:
The difference of the active volume of the active volume and energy-conservation group that are subordinated to the service group of same back end and capacity threshold are made comparisons.
8. a management devices for distributed system, described distributed system comprises Metadata Service node, and back end is characterized in that, comprising:
Grouping module, for dividing into groups to disks all in described distributed system, obtains multiple service group and multiple energy-conservation group;
Modular converter, for by the read-write business migration of described multiple energy-conservation group to described multiple service group, and reduction of speed or dormancy are carried out to the disk in described multiple energy-conservation group;
Run module, for making the business at described multiple service group full speed running place.
9. management devices according to claim 8, is characterized in that, comprises further:
Statistical module, for when reading and writing data to described back end, adding up the traffic pressure value of each disk group corresponding to this back end and being sent to Metadata Service node;
First comparison module, for comparing each pressure threshold of the traffic pressure value of each disk group with corresponding each disk group respectively;
First handover module, for being greater than corresponding pressure threshold in the traffic pressure value of disk group, switches to described energy-conservation group by this disk group.
10. management devices according to claim 8, is characterized in that, comprises further:
Acquisition module, for the capacity service condition of each disk group corresponding to each back end of taken at regular intervals;
Second comparison module, makes comparisons for the difference of the active volume by the active volume of service group and energy-conservation group and capacity threshold;
Second handover module, for when described difference exceedes described capacity threshold, carries out state switching by this service group and this energy-conservation group.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510923687.0A CN105528053A (en) | 2015-12-11 | 2015-12-11 | Management method and device for distributed system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510923687.0A CN105528053A (en) | 2015-12-11 | 2015-12-11 | Management method and device for distributed system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105528053A true CN105528053A (en) | 2016-04-27 |
Family
ID=55770329
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510923687.0A Pending CN105528053A (en) | 2015-12-11 | 2015-12-11 | Management method and device for distributed system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105528053A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018086409A1 (en) * | 2016-11-08 | 2018-05-17 | 杭州海康威视数字技术股份有限公司 | Disk data reading/writing method and device |
| CN109164981A (en) * | 2018-08-15 | 2019-01-08 | 优刻得科技股份有限公司 | Disk management method, device, storage medium and equipment |
| CN111352592A (en) * | 2020-02-27 | 2020-06-30 | 腾讯科技（深圳）有限公司 | Disk read-write control method, device, equipment and computer readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101344812A (en) * | 2008-08-19 | 2009-01-14 | 华南理工大学 | Magnetic disk dynamic power supply management method based on embedded system |
| CN103576835A (en) * | 2013-10-30 | 2014-02-12 | 华为技术有限公司 | Data manipulation method and device for sleep disk |
| EP2829946A1 (en) * | 2012-05-14 | 2015-01-28 | Huawei Technologies Co., Ltd | Device and method for system power supply management and system having power supply energy-saving management function |
-
2015
- 2015-12-11 CN CN201510923687.0A patent/CN105528053A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101344812A (en) * | 2008-08-19 | 2009-01-14 | 华南理工大学 | Magnetic disk dynamic power supply management method based on embedded system |
| EP2829946A1 (en) * | 2012-05-14 | 2015-01-28 | Huawei Technologies Co., Ltd | Device and method for system power supply management and system having power supply energy-saving management function |
| CN103576835A (en) * | 2013-10-30 | 2014-02-12 | 华为技术有限公司 | Data manipulation method and device for sleep disk |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018086409A1 (en) * | 2016-11-08 | 2018-05-17 | 杭州海康威视数字技术股份有限公司 | Disk data reading/writing method and device |
| US11048601B2 (en) | 2016-11-08 | 2021-06-29 | Hangzhou Hikvision Digital Technology Co., Ltd. | Disk data reading/writing method and device |
| CN109164981A (en) * | 2018-08-15 | 2019-01-08 | 优刻得科技股份有限公司 | Disk management method, device, storage medium and equipment |
| CN109164981B (en) * | 2018-08-15 | 2020-06-16 | 优刻得科技股份有限公司 | Disk management method, device, storage medium and equipment |
| CN111352592A (en) * | 2020-02-27 | 2020-06-30 | 腾讯科技（深圳）有限公司 | Disk read-write control method, device, equipment and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102222085B (en) | Data de-duplication method based on combination of similarity and locality | |
| CN105653524B (en) | Data storage method, device and system | |
| CN104573119A (en) | Energy-saving-oriented Hadoop distributed file system storage policy in cloud computing | |
| CN101673191B (en) | Data writing method and device as well as data memory system | |
| CN102426475A (en) | Energy saving method, energy saving management server and system under desktop virtual environment | |
| CN105868004B (en) | Scheduling method and scheduling device of service system based on cloud computing | |
| CN105528053A (en) | Management method and device for distributed system | |
| Ranjana et al. | A survey on power aware virtual machine placement strategies in a cloud data center | |
| CN110633152A (en) | Method and device for realizing horizontal scaling of service cluster | |
| CN104298574A (en) | Data high-speed storage processing system | |
| CN108776690B (en) | Method for HDFS distributed and centralized mixed data storage system based on hierarchical governance | |
| CN104102301A (en) | 2U (unit) ultrahigh-density storage server | |
| CN101819459A (en) | Heterogeneous object memory system-based power consumption control method | |
| CN101504568B (en) | Method, apparatus and electronic equipment for operating hard disk | |
| CN108388471B (en) | Management method based on double-threshold constraint virtual machine migration | |
| CN107203256A (en) | Energy-conservation distribution method and device under a kind of network function virtualization scene | |
| CN105528054A (en) | Cluster system comprehensive scheduling energy saving method and device | |
| CN101853185B (en) | Blade server and service dispatching method thereof | |
| CN102681650B (en) | The storage system of a kind of Energy control power-economizing method and correspondence thereof | |
| CN104765572A (en) | Energy-saving virtual storage server system and scheduling method | |
| CN105808150A (en) | Solid state disk caching system for hybrid storage device | |
| CN101776946B (en) | Method for controlling power consumption on basis of object-based storage system | |
| US8281159B1 (en) | Systems and methods for managing power usage based on power-management information from a power grid | |
| CN103714022A (en) | Mixed storage system based on data block | |
| CN107479830A (en) | A kind of energy saving of hard disks control method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160427 |
|
| RJ01 | Rejection of invention patent application after publication |