CN103616944B - Consumption-reducing method based on anticipation green data classification policy in cloud storage system - Google Patents

Abstract

The invention discloses the consumption-reducing method based on anticipation green data classification policy in a kind of cloud storage system. The present invention is by the rate of people logging in of file and threshold value comparison, and it is exactly dsc data that rate of people logging in is greater than hot threshold value, and it is exactly cold data that rate of people logging in is less than cold threshold value, data between cold and hot threshold value are carried out anticipation, in the time that certain time period is hot, is just migrated to thermal region, otherwise put it into cool region. If there is new data to carry out write operation, and there are not these data in former cloud storage system, and the data in these data and former cloud storage system are carried out to correlation analysis, finds the data the highest with its degree of correlation, and new data is put into and the disk of legacy data same type. The present invention reasonably classifies data file well, substantially avoided the hot file that may occur the unreasonable classification in the situation that at cool region by rationally effectively classification, energy resource consumption and the file of cold file in thermal region reads the situation that time delay increases.

Description

Consumption-reducing method based on anticipation green data classification policy in cloud storage system

Technical field

The invention belongs to computer data management field, specifically in a kind of cloud storage system based on anticipation green dataThe consumption-reducing method of classification policy.

Background technology

More and more universal along with data-intensive applications and service, large-scale data center is consuming huge electric power moneySource, and rapid growth. The energy consumption summation of all data centers in 2011 of the U.S. is 100TWh according to statistics, according to common electricity charge meterValency method will spend 1,000 hundred million dollars, and the 40TWh consuming than 2005 has gone up more than twice, the energy consumption of data center1～2% of all energy consumptions of the Zhan Liao U.S.. And wherein storage system is the energy consumption rich and influential family of data center, account for 25～35%. In dataThe heart is the huge energy of autophage not only, the energy that cooling device consumes be also can not ignore (account for data center consumes energyAmount 1/3rd), more severe: except the economic loss that energy consumption itself is brought, its carbon dioxide giving out is alsoCan produce ill effect (can give off the carbon dioxide of 0.7kg at the U.S.'s 1 degree electricity) to environment. And along with various application are to depositingThe demand of storage equipment is expanding with annual 60% growth rate, and storage system more can not be ignored the energy of consumption. Therefore howThe energy consumption that reduces large-scale data center memory device is a problem in the urgent need to address.

Summary of the invention

The present invention puts into different disk regions for the existing technology of utilizing Data classification by different dataThe deficiency in energy consumption is saved in territory, introduced to new data and season dsc data concept, anticipation green data has been proposedClassification policy. Anticipation green data classification policy is exactly that rate of people logging in is greater than hot threshold value by the rate of people logging in of file and threshold value comparisonBe exactly dsc data, it is exactly cold data that rate of people logging in is less than cold threshold value, and the data between cold and hot threshold value are carried out anticipation, whenCertain time period, when hot, just migrated to thermal region, otherwise is put it into cool region. If there is new data to carry out write operation, andThere are not these data in former cloud storage system, the data in these data and former cloud storage system are carried out to correlation analysis, find withThe data that its degree of correlation is the highest, put into new data and the disk of legacy data same type.

The concrete steps of the inventive method are:

Step (1): to cold data, dsc data, season dsc data, and new data defines, and specifically defines as follows:

1) cold data: the average access number of operations of these data is less than cold threshold value in long-term process operation data.

2) dsc data: the average access number of operations of these data is greater than hot threshold value in long-term process operation data.

3) season dsc data: in long-term operating process, these data only have certain section or a few sections of time operations on average timeNumber is greater than hot threshold value, and data temperature presents the situation of fluctuation up and down, and data average access is grasped in whole process operation dataMake number of times between cold and hot threshold value.

4) new data: be illustrated in data not in cloud storage system, these data yet do not exist and grasped accordinglyThe number of times of doing.

Step (2): the definition to anticipation green data classification policy model:

Definition F={f₁,...,f_i,...,f_m, the set of F representation file, f_i＝{m_i,p_i, wherein p_iRepresent file f_iCurrentThe array that every month, rate of people logging in formed of the first two years on date, m_iRepresent file f_iProperty value, wherein attribute comprises: fileThe cryptographic Hash of keyword, the cryptographic Hash of file name, the cryptographic Hash of file content. p_ijRepresent array p_iIn j element,ap_iRepresent p_iThe mean value of array, it is the average access rate of visit data,

1) the average access frequency ap of judgement visit data_iWhether be 0, no, these data are legacy data, go to 2); Be,These data are new data, go to 4).

2) by the average access frequency values ap of legacy data_iCompare with cold and hot threshold value, if average access frequency is greater thanEqual hot threshold value th_h, these data are dsc data, and these data are put into Thermomagnetic disc; Data between cold and hot threshold value are considered as to seasonJoint dsc data, will be less than or equal to cold threshold value th_cData be considered as cold data, by cold data and season dsc data put into cold diskIn; If these data are dsc data in season, after putting into cold disk, go to 3).

3) by neutral net to season dsc data predict, the average access rate of predicting its next month, judges under itThe temperature of individual month. If the average access rate of its next month is more than or equal to hot threshold value th_h, these data are put into Thermomagnetic disc, noPut into cold disk.

4) property value of the legacy data in the property value of new data and cold and hot disk is carried out to correlation analysis, find relevantSpend maximum legacy data, new data is put into and the disk of legacy data same type.

Step (3): the structure of energy consumption model:

Make popular file and the number ratio of non-popular file be, make the request number ratio of popular file and non-popular fileFor η, S'_hS_c'＝k,0＜k＜1，S'_hRepresent the mean size of hot demand file, S_c' represent the mean size of cold demand file,Ordert^hRepresent hyperdisk transfer rate, unit is Mb/s, t^cRepresent disk transfer rate at a slow speed, unit is Mb/s，|D_h|/|D_c|＝γ，D_h＝{d₁,...,d_h,...,d_e}，D_hRepresent fast rotational disc-pack, D_c＝{d_f,...,d_c,...,d_n}，D_cRepresent spinning disk set at a slow speed.

In the cold and hot disk situation of differentiation, total power consumption is:

Wherein R_h＝{r₁,...r_h,...,r_b}，R_hThe set of representative request dsc data, t_ceshiRepresent in experimentation realThe time of testing, p^hRepresent the positive energy consumption of hyperdisk, unit is J/Mb, i^hRepresent the desirable moment energy consumption of hyperdisk, unit is J/s。

In the time not distinguishing cold and hot disk, total power consumption is:

The energy that can save taking the temperature of specified data by Data classification as:

Beneficial effect of the present invention:

The present invention proposes anticipation green data classification policy, this classification policy enters new data and Seasonal DataGo good location. This anticipation Data classification strategy is reasonably classified data file well, by rationally havingThe classification of effect has avoided the hot file that may occur the in the situation that of unreasonable classification at cool region substantially, and cold file is at thermal regionIn energy resource consumption and file read time delay increase situation.

Brief description of the drawings

Fig. 1 is the flow chart of data classification algorithm;

Fig. 2 is the affect lab diagram of number of disks on energy consumption;

Fig. 3 is number of disks the is saved percentage lab diagram that affects on energy;

Fig. 4 is the affect lab diagram of heat request number on energy consumption;

Fig. 5 is heat request number the is saved percentage lab diagram that affects on energy;

Fig. 6 be the comparison energy of number of disks consume affect lab diagram;

Fig. 7 is the lab diagram that affects that the comparison energy of number of disks is saved percentage;

Fig. 8 is that the mean size of hot file is on the lab diagram that affects of energy consumption;

Fig. 9 is the mean size of hot file the is saved percentage lab diagram that affects on energy;

Figure 10 is the lab diagram that affects of hot cold file number comparison energy consumption;

Figure 11 is the lab diagram that affects that hot cold file number comparison energy is saved percentage.

Detailed description of the invention

The present invention puts into different disk regions for the existing technology of utilizing Data classification by different dataThe deficiency in energy consumption is saved in territory, introduced to new data and season dsc data concept, anticipation green data has been proposedClassification policy. Anticipation green data classification policy is exactly that rate of people logging in is greater than hot threshold value by the rate of people logging in of file and threshold value comparisonBe exactly dsc data, it is exactly cold data that rate of people logging in is less than cold threshold value, and the data between cold and hot threshold value are carried out anticipation, whenCertain time period, when hot, just migrated to thermal region, otherwise is put it into cool region. If there is new data to carry out write operation, andThere are not these data in former cloud storage system, the data in these data and former cloud storage system are carried out to correlation analysis, find withThe data that its degree of correlation is the highest, put into new data and the disk of legacy data same type.

As shown in Figure 1, the concrete steps of the inventive method are:

Step (1): to cold data, dsc data, season dsc data and new data define, concrete data classification methodAs follows:

1) cold data: the average access number of operations of these data is less than cold threshold value in long-term process operation data;

2) dsc data: the average access number of operations of these data is greater than hot threshold value in long-term process operation data;

3) season dsc data: in long-term operating process, these data only have certain section or a few sections of time operations on average timeNumber is greater than hot threshold value, and data temperature presents the situation of fluctuation up and down, and data average access is grasped in whole process operation dataMake number of times between cold and hot threshold value;

4) new data: be illustrated in data not in cloud storage system, these data yet do not exist and grasped accordinglyThe number of times of doing.

The reasonability that theory analysis new data temperature is judged.

The direct acting factor of data temperature is the operated number of times of data, relevant to people's behavial factor. If not yetThere is the great accident of generation, people's behavial factor and too large change generally can not occur to the behavior favourite hobby of data attribute, soHere we think that the property value of data can show the temperature value of data indirectly. So for new data, IConcentrate at legacy data the temperature finding with the data of the data attribute value correlation maximum of these data, just can determine new dataTemperature.

Step (2): the definition to anticipation green data classification policy model:

Definition F={f₁,...,f_i,...,f_m, the set of F representation file, f_i＝{m_i,p_i, wherein p_iRepresent file f_iCurrentThe array that every month, rate of people logging in formed of the first two years on date, m_iRepresent file f_iProperty value, wherein attribute comprises: fileThe cryptographic Hash of keyword, the cryptographic Hash of file name, the cryptographic Hash of file content. p_ijRepresent array p_iIn j element,ap_iRepresent p_iThe mean value of array, it is the average access rate of visit data,th_cExpression is judged to beThe threshold value of cold data, th_hRepresent to be judged to be the threshold value of dsc data.

1) the average access frequency ap of judgement visit data_iWhether be 0, no, these data are legacy data, go to 2); Be,These data are new data, go to 4).

2) by the average access frequency values ap of legacy data_iCompare with cold and hot threshold value, if average access frequency is greater thanEqual hot threshold value th_h, these data are dsc data, and these data are put into Thermomagnetic disc; Data between cold and hot threshold value are considered as to seasonJoint dsc data, will be less than or equal to cold threshold value th_cData be considered as cold data, by cold data and season dsc data put into cold diskIn; If these data are dsc data in season, after putting into cold disk, go to 3).

3) by neutral net to season dsc data predict, the average access rate of predicting its next month, judges under itThe temperature of individual month. If the average access rate of its next month is more than or equal to hot threshold value th_h, these data are put into Thermomagnetic disc, noPut into cold disk.

4) by the property value m of the legacy data in the property value of new data and cold and hot disk_iCarry out correlation analysis, find phaseThe legacy data of pass degree maximum, puts into new data and the disk of legacy data same type.

Data classification strategy false code is as follows:

Input: P_i，m_i，th_c，rh_h

foreachtimewindowTdo

foreachfile

ifp_i！=’‘

${\mathrm{ap}}_i=\frac{1}{24}{\mathrm{\Σ}}_{j=1}^{24}{p}_{\mathrm{ij}}$

ifap_i≤th_c

Corresponding document is put into cold node

elselfap_i≥th_h

Corresponding document is put into thermal center point

else

Adopt the neural network prediction rate of people logging in y of next month_i

ify_i≥th_h

Corresponding document is put into thermal center point

else

Corresponding document is put into cold node

end

end

elseifp_i==′′andp_i！=’‘

Judge the temperature of these data by new data temperature judge algorithm

If this file is cold data, just file i is put into cold node, otherwise put

Enter thermal center point.

end

end

end

Step (3): the structure of energy consumption model:

Make popular file and the number ratio of non-popular file beMake the request number ratio of popular file and non-popular fileFor η, S'_hS_c'＝k,0＜k＜1，S'_hRepresent the mean size of hot demand file, S_c' represent the mean size of cold demand file,Ordert^hRepresent hyperdisk transfer rate, unit is Mb/s, t^cRepresent disk transfer rate at a slow speed, unit isMb/s，|D_h|/|D_c|＝γ，D_h＝{d₁,...,d_h,...,d_e}，D_hRepresent fast rotational disc-pack, D_c＝{d_f,...,d_c,...,d_n}，D_cRepresent spinning disk set at a slow speed.

In the cold and hot disk situation of differentiation, total power consumption is:

Wherein R_h＝{r₁,...r_h,...,r_b}，R_hThe set of representative request dsc data, t_ceshiRepresent in experimentation realThe time of testing, p^hRepresent the positive energy consumption of hyperdisk, unit is J/Mb, i^hRepresent the desirable moment energy consumption of hyperdisk, unit is J/s。

In the time not distinguishing cold and hot disk, total power consumption is:

The energy that can save taking the temperature of specified data by Data classification as:

Be below analysis and the proof thereof of Energy saving theory:

e_hotThe energy consumption of dsc data place disk while representing to distinguish cold and hot disk, e_coldCold number while representing to distinguish cold and hot diskAccording to the energy consumption of place disk, e '_hotThe energy consumption of dsc data place disk while representing not distinguish cold and hot disk, e '_coldRepresent not distinguishThe energy consumption of cold data place disk when cold and hot disk, wherein e '_cold＝e’_hot＝e_hot. Wherein R_c＝{r_p,...r_c,...,r_x}，R_cThe set of the cold data of representative request, p^c(J/Mb) represent the positive energy consumption of low speed disk, i^c(J/s) while representing that low speed disk is desirableCarve energy consumption, S_cRepresent the size of C cold demand file.

e_total＝e_hot+e_cold,e’_total＝e’_hot+e’_cold,e’_hot＝e_hot，

$e_{\mathrm{cold}}=\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\Σ}}}{S}_c*p^c+i^c*\left(\right|D_c|*t_{\mathrm{xeshi}}-\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\Σ}}}{S}_c/t^c),$

$e_{\mathrm{cold}}^{,}=\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\Σ}}}{S}_c*p^h+i^h*\left(\right|D_c|*t_{\mathrm{ceshi}}-\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\Σ}}}{S}_c/t^h),$

Due to t^h＞t^c? $\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\&Sigma;}}}{S}_c/t^h<\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\&Sigma;}}}{S}_c/t^c,$

? $\left|D_c\right|*t_{\mathrm{ceshi}}-\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\&Sigma;}}}{S}_c/t^c<\left|D_c\right|*t_{\mathrm{ceshi}}-\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\&Sigma;}}}{S}_c/t^h,$

I again^h＞i^c, $i^c*\left(\right|D_c|*t_{\mathrm{ceshi}}-\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\&Sigma;}}}{S}_c/t^c)<i^c*\left(\right|D_c|*t_{\mathrm{ceshi}}-\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\&Sigma;}}}{S}_c/t^h)$

Because p^h＞p^c? $\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\&Sigma;}}}{S}_c*p^c<\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\&Sigma;}}}{S}_c*p^h$

Again $i^c*\left(\right|D_c|*t_{\mathrm{ceshi}}-\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\&Sigma;}}}{S}_c/t^c)<i^h*\left(\right|D_c|*t_{\mathrm{ceshi}}-\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\&Sigma;}}}{S}_c/t^h)$

?

$\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\&Sigma;}}}{S}_c*p^c+i^c*\left(\right|D_c|*t_{\mathrm{ceshi}}-\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\&Sigma;}}}{S}_c/t^c)<\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\&Sigma;}}}{S}_c*p^h+i^h*\left(\right|D_c|*t_{\mathrm{ceshi}}-\underset{c=1}{\overset{\left|R_c\right|}{\mathrm{\&Sigma;}}}{S}_c/t^h)$

E_cold＜e'_cold，

E so_total＜e_t'_otal, e > 0 so energy save percentage must be greater than 0.

Known by based on anticipation green data classification policy by above-mentioned proof, dissimilar data are navigated toIn different disk regions, there is consumption reduction effect.

Below in conjunction with accompanying drawing subordinate list and embodiment, the present invention is described further.

The present embodiment is realized based on cold and hot disk array simulator, and the relevant parameter of disk is in table 1.

Table 1: disk relevant parameter

Wherein have two class parameters directly to have influence on the final data of the present embodiment, it comprises: workload feature and disk driveThe characteristic of moving device. Wherein have a large amount of parameter influence workload features, we determine five key characteristics (table 2):

(1) number of files is because the sum of file directly determines to distribute in a parallel disk array the negative of disk arrayCarry, be set to 5000, each disk can be held in the situation of about 312 files 16 disc driver battle arraysRow. The quantity of the file on each disk is that the situation of the imitation real world based on real determines.

(2) total request number is because total request number directly affects the work times of storage system within the testing time,And then affect the energy consumption of storage system. We are 10000 by total request number of times setting value.

(3) proportion of cold and hot request is because the proportion of cold and hot file access directly affects the reading times in cold and hot region, shadowRung the energy consumption of whole storage system, still the value of cold and hot request ratio is set as to 6:46.5:3.57:37.5:2.58:2。

(4) percentage of the coverage of the whole file system of coverage of file system is defined as file system filesThe workload of request of access. The coverage that we are provided with system is 100%, this means all literary compositions in file systemPart is crossed at least one times in the access of parallel disk array system.

(5) hot cold number of disks can effectively be saved energy consumption than the ratio of reasonably setting hot cold number of disks, according to itFront formula, we set hot cold disk than being 3:1.

Table 2: the related data that experiment is used is described

Describe	Default value (value)
		Total number of files	500
Request sum	10000
		File system coverage	100%
The cold request number ratio of heat	8:2（6:46.5:3.57:37.5:2.58:2）
		The cold number of disks ratio of heat	1:3（3/13,4/12,5/11,6/10,7/9,8/8,9/7,10/6）
The mean size of hot file	45（1520253035404550）M
		The ratio of the cold number of files of heat	2.8（0.81.31.82.32.83.33.834.34.85.35.86.3）
Disk sum	（121620242832）
		Testing time	3*24 hour

The ratio of cold and hot disk transfer rate

1:3

The present embodiment is respectively by number of disks, and heat is asked number, number of disks ratio, hot file mean size, hot cold literary compositionThe energy consumption of the change modeling test data of experiment classification of the ratio of part number changes and the variation of non-classified energy consumption and energy-conservation hundredThe variation (Fig. 2-Figure 11) of proportion by subtraction.

Embodiment result shows: this method is for all rising in the situations such as large-scale storage systems and large document storage systemArrived good consumption reduction performance, consumption reduction percentage fluctuates between 0.1 to 0.2. This anticipation Data classification strategy well willData file is reasonably classified, and has substantially avoided may going out the unreasonable classification in the situation that by rationally effectively classificationExisting hot file is at cool region, and energy resource consumption and the file of cold file in thermal region reads the situation that time delay increases.

Should be understood that: above-described embodiment is just to explanation of the present invention, instead of limitation of the present invention, anyDo not exceed the innovation and creation within the scope of connotation of the present invention, within all falling into protection scope of the present invention.

Claims (1)

1. the consumption-reducing method based on anticipation green data classification policy in cloud storage system, it is characterized in that the method comprise withLower step:

Step (1): to cold data, dsc data, season dsc data, and new data defines, and specifically defines as follows:

1) cold data: the average access number of operations of these data is less than cold threshold value in long-term process operation data;

2) dsc data: the average access number of operations of these data is greater than hot threshold value in long-term process operation data;

3) season dsc data: in long-term operating process these data only have certain section or a few sections of time operation average times largeIn hot threshold value, data temperature presents the situation of fluctuation up and down, and data average access operation time in whole process operation dataNumber is between cold and hot threshold value;

4) new data: be illustrated in data not in cloud storage system, these data do not exist operated yet accordinglyNumber of times;

Step (2): the definition to anticipation green data classification policy model:

Definition F={f₁,...,f_i,...,f_m, the set of F representation file, f_i＝{m_i,p_i, wherein p_iRepresent file f_iCurrent dateThe first two years every month rate of people logging in form array, m_iRepresent file f_iProperty value, wherein attribute comprises: the key of fileThe cryptographic Hash of word, the cryptographic Hash of file name, the cryptographic Hash of file content; p_ijRepresent array p_iIn j element, ap_iGenerationTable p_iThe mean value of array, it is the average access rate of visit data, ${\mathrm{ap}}_i=\frac{1}{24}\underset{j=1}{\overset{24}{\&Sigma;}}{p}_{ij};$

1) the average access frequency ap of judgement visit data_iWhether be 0, no, these data are legacy data, go to 2); ShouldData are new data, go to 4);

2) by the average access frequency values ap of legacy data_iCompare with cold and hot threshold value, if average access frequency is more than or equal to heatThreshold value th_h, these data are dsc data, and these data are put into Thermomagnetic disc; Data between cold and hot threshold value are considered as to hot number in seasonAccording to, will be less than or equal to cold threshold value th_cData be considered as cold data, by cold data and season dsc data put into cold disk; IfThese data are dsc data in season, after putting into cold disk, go to 3);

3) by neutral net to season dsc data predict, the average access rate of predicting its next month, judges its next monthTemperature; If the average access rate of its next month is more than or equal to hot threshold value th_h, these data are put into Thermomagnetic disc, otherwise putEnter cold disk;

4) property value of the legacy data in the property value of new data and cold and hot disk is carried out to correlation analysis, find the degree of correlationLarge legacy data, puts into new data and the disk of legacy data same type;

Step (3): the structure of energy consumption model:

Make popular file and the number ratio of non-popular file beMaking popular file and the request number ratio of non-popular file is η,S'_h/S′_c＝k,0<k<1，S'_hRepresent the mean size of hot demand file, S '_cRepresent the mean size of cold demand file, ordert^hRepresent hyperdisk transfer rate, unit is Mb/s, t^cRepresent disk transfer rate at a slow speed, unit is Mb/s，|D_h|/|D_c|＝γ，D_h＝{d₁,...,d_h,...,d_e}，D_hRepresent fast rotational disc-pack, D_c＝{d_f,...,d_c,...,d_n}，D_cRepresent spinning disk set at a slow speed;

In the cold and hot disk situation of differentiation, total power consumption is:

Wherein R_h＝{r₁,...r_h,...,r_b}，R_hThe set of representative request dsc data, t_ceshiRepresent to test in experimentation timeBetween, p^hRepresent the positive energy consumption of hyperdisk, unit is J/Mb, i^hRepresent the desirable moment energy consumption of hyperdisk, unit is J/s;

In the time not distinguishing cold and hot disk, total power consumption is:

The energy that can save taking the temperature of specified data by Data classification as: