CN105653591A - Hierarchical storage and migration method of industrial real-time data - Google Patents

Hierarchical storage and migration method of industrial real-time data Download PDF

Info

Publication number
CN105653591A
CN105653591A CN201510969294.3A CN201510969294A CN105653591A CN 105653591 A CN105653591 A CN 105653591A CN 201510969294 A CN201510969294 A CN 201510969294A CN 105653591 A CN105653591 A CN 105653591A
Authority
CN
China
Prior art keywords
data object
data
migration
value
time
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201510969294.3A
Other languages
Chinese (zh)
Other versions
CN105653591B (en
Inventor
徐星
陈鹏
叶莹
王天林
宋丽娜
庄严
周玄昊
俞翔
韩冰
王挺
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
ZHEJIANG ZHOUSHAN TO CONTROL INTELLIGENT EQUIPMENT TECHNOLOGY CO., LTD.
Original Assignee
ZHEJIANG SUPCON RESEARCH Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ZHEJIANG SUPCON RESEARCH Co Ltd filed Critical ZHEJIANG SUPCON RESEARCH Co Ltd
Priority to CN201510969294.3A priority Critical patent/CN105653591B/en
Publication of CN105653591A publication Critical patent/CN105653591A/en
Application granted granted Critical
Publication of CN105653591B publication Critical patent/CN105653591B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • G06F16/214Database migration support
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention provides a hierarchical storage and migration method and system of industrial real-time data. The method comprises the following steps: monitoring a hierarchical storage system in fixed time, and triggering data migration calculation after the storage volume use ratio of advanced storage equipment achieves a first preset threshold value; during migration, firstly, carrying out value assessment on each data object in the storage equipment to obtain the value of each data object and sort the data object according to the value; and according to sorting, setting a migration strategy, picking up the data object to be migrated, and forming and migrating a migration queue. The method and the system set the migration strategy according to different storage equipment priorities and the value sorting of the current data objects, wherein value calculation considers a time factor, number of access users, other data object situations associated with the data object, the access contrast ratio of different storage equipment and the own size of the data object, and the time factor, the number of access users, other data object situations associated with the data object, the access contrast ratio of different storage equipment and the own size of the data object affect data object migration efficiency. The method improves data value judgment accuracy.

Description

A kind of industrial real-time data classification storage and moving method
Technical field
The present invention relates to data access and stores processor technical field, in particular to a kind of industrial real-time data classification storage and moving method.
Background technology
Along with the expansion of industrial system scale and the development of automatization information technology, the application of industrial automation system mass data causes surging of the concurrent visit capacity of distributed document system, and file read-write pressure becomes the system bottleneck that must need greatly to consider that file I/O brings. Meanwhile, in process control, the real-time of data is required higher by a lot of application. Consider that different storage device performance is different with cost, and data access has Time and place locality, it is thus desirable to carry out classification storage, making the data trend being often accessed in being stored in high-performance equipment, the data placement infrequently read and write in the nearest access time is to low-performance equipment. Moreover, it is contemplated that data also exist periodic Changing Pattern, the temperature of data access is change, and in mass memory system, the data of suitable vast scale are static does not move, and high-performance storing device is limited, therefore carries out data migration based on classification memory technology.
With the fast development of the solid state hard discs such as SSD and applying in every field, carry out, in conjunction with solid state hard disc, the emphasis that dynamic data attemper has become current and following storage aspect research. Solid state hard disc has more significantly relative merits compared with conventional hard, can the better performance of optimization system and energy consumption, it is possible to as the fast disk medium of dynamic data attemper medium. But due to expensive, it is thus desirable to comprehensive considering various effects obtains compromise in performance, cost and energy consumption.
It is not special filing frequently in backup environment that traditional classification storage the earliest is mainly used in access. But the performance difference considering equipment is different, if the equipment that performance difference is big and performance difference is little adopts identical triggering condition to be unfavorable for the extensibility of system.Realize the unified management of dynamic data attemper equipment file, meta data block, meta data server module, target data server module are set respectively, meta data server module is provided with system administration and file migration decision-making module, artificial acquisition moves candidate's file, file is divided into upgrade queue and degradation queue, sends migration instruction by migration scheduling controller and move. For source data server and target data server, setting data service module and migration execution module respectively. The major defect of this technology is, it does not have the method for a system and concrete judgement file migration trigger point, and the artificial file migration ratio that proposes carries out file migration, and evaluation that cannot be comprehensively full and accurate affect all factors of data value judgement.
Above-mentioned mass data hierarchical memory technology mainly according to the performance of storing device be worth different by data placement on different devices, and carry out data migration in the suitable time. But classification and migration strategy (data value decision method) are not fully excavated various measurement index by these classification storage meanss, owing to data placement and data migration strategy directly determine the overall performance of whole classification storage system, more perfect migration strategy and data classification store laying method and are urgently suggested.
Summary of the invention
It is an object of the invention to provide a kind of industrial real-time data classification storage and moving method, store and migrating technology does not fully carry out data value judgement and affects data and store and the problem of migration performance to solve existing mass data hierarchical.
For achieving the above object, the present invention provides a kind of industrial real-time data classification storage and moving method, comprises data classification storage and two parts are moved in data classification, and wherein, data classification storage comprises the following steps:
I: carry out data being worth evaluation;
II: placed or migrate in suitable level according to data value;
Data classification migration comprises the following steps:
S1: regular monitoring classification storage system, when, after the first threshold value that the storage capacity rate of utilization of high priority storing device reaches default, trigger data migration calculates, and enters step S2;
S2: carry out each data object in storing device being worth evaluation, obtain the value of each data object, according to the size of described value, each corresponding data object is sorted;
S3: be default Second Threshold according to accounting, selects data object lower for the sorting out value stored in high priority storing device, composition migration queue, and the data object in described migration queue is migrated to low priority storing device;
S4: be the 3rd default threshold value according to accounting, the data object address kept current in data object address lower for the remaining sorting out value stored in high priority storing device after step S3 execution and buffer memory is compared, address such as any one data object wherein be kept at as described in buffer memory, then this data object is migrated to low priority storing device, otherwise the internal memory address of this data object is preserved in the buffer, the rest may be inferred, if in buffer memory, data object number of addresses is N after completing all comparisonsbIf, Nb��Nh, then this migration work stops, if Nb>Nh, then value corresponding to the data object address preserved in buffer memory is by sorting successively to little greatly, rejects successively, until remaining data object address number is N from the maximum data object address of valueh, this migration work stops, wherein, and NhFor the preset upper limit of data object number of addresses in buffer memory;
S5: find and be worth as the data object of maximum value described in current buffer memory, and by valuable for institute in the low priority storing device data object being greater than described maximum value according to value by high to Low order form migration queue also shifting to high priority storing device.
Goodly, in described S2, adopt the method for moving window, the value that each moment in this moving window calculates sought weighted mean, is specially:
If the width of given window is N, when being respectively V for the current recently data object that N time calculates is worth in this window1��V2����VNTime, then the value calculation formula of current data object is as follows:
V c = Σ i = 1 N λ i V i .
Goodly, in described S2, when carrying out data object being worth evaluation, according to following formula, value is calculated:
V=w1T+w2C+w3N+w4CT+w5/S
Wherein, T is time factor, and C is access number of users factor, and N is the value factor of the associated data object with notebook data object, and CT is the contrast gradient factor of different storage device, and S is the size factor of data object self, w1��w2��w3��w4And w5It is respectively the weight of each corresponding factor.
Goodly, the acquisition methods of described time factor T is:
The moment accepted the interview that acquisition data Object Creation is all after starting: t1��t2��tn, n is positive integer;
Calculate the time span T at interval between each access1��T2��Tn-1, then:
Ti=ti+1-tiI=1,2 ..., n-1
Calculate T:
T = Σ i = 1 n - 1 α i T i ,
Wherein, ��i, i=1,2 ..., n-1 is one group of weighted value given in advance, and meetsAnd ��1�ܦ�2��...�ܦ�n-1��
Goodly, to any one data object, its associated data object is defined as follows:
Setting-up time length threshold is Tth, any t0Time data object obj1Accessed, then at t0+TthIn time interval, data object obj2Also accessed, then think data object obj1And obj2It is associated.
Goodly, data object obj1The acquisition methods of value factor N of associated data object as follows:
Find and data object obj1Set of data objects �� (the obj being associated1);
Find �� (obj1) in the value of all data objects;
To with data object obj1It is as follows that the value of all data objects being associated carries out summation:
N = Σ o b j ∈ Φ ( obj 1 ) V o b j
VobjFor the value record of data object obj.
Goodly, by data object, from being established to, current time interval is divided into m section, then the contrast gradient factor CT of different storage device calculates according to following formula:
C T = Σ i = 1 m β i ( δ w × FW i + δ r × FR i )
Wherein, FWiAnd FRiRepresent the read-write frequency of data object within i-th period, ��iRepresent the weighting weight of i-th period, and ��1<��2<��<��m, ��rIt is the reading contrast gradient between two different storage device, ��wIt is write contrast gradient between two different storage device.
Goodly, for different storing device A and B, if it reads contrast gradient is ��r, writing contrast gradient is ��w, then have:
&delta; r = R A / R B ( R B R A ) 2 + 1 ,
&delta; w = W A / W B ( W B W A ) 2 + 1 ,
Wherein, RA��RBRepresent respectively lasting reading on the equipment of A and B two kinds of different performances according to time speed, WA��WBFor the speed continued when writing data of correspondence.
Goodly, described first threshold value is 80%, and Second Threshold is the 10%, three threshold value is 10%.
Present invention also offers a kind of industrial real-time data classification storage and migration system, comprising:
Classification storage system, comprise the storing device that some the priority for storing data object are different, also comprise buffer memory, wherein, buffer memory is used for preserving the address being worth lower data object in the storing device of high priority, and these data objects carry out dynamic change with transition process, when occurring data by high to Low migration, if the data object address that any one is chosen has preserved in the buffer, then this data object is migrated to low priority storing device;
It is worth and judges management device, for the data object obtained in real time in described classification storage system, and calculate the value of data object;
Data placement planning management device, is worth, from described, the value judging management device for obtaining, and selects data object to be migrated to form migration queue according to the result being worth, and forms data placement plan and migration strategy;
Migration engine controller, orders to application server proxy for obtaining described data placement plan and migration strategy send migration, and described application server proxy is registered;
Application server proxy, for registering to migration engine controller during initialize, and receives described migration order and moves/move back module to forward to corresponding data;
Module is moved/moved back to data, is located between the different storing device of every two priority respectively, for carrying out the migration of data according to described migration order or move back, and migration results feeds back to described migration engine controller.
Goodly, described migration engine controller comprises:
Data monitoring module, for monitoring and record the update status of data object and value variation and feed back this update status to described values and judge management device, and the I/O access situation of system is monitored;
And data management module, for regularly inquiring about described data placement planning management device, to carry out the renewal of data information, and send described migration order and receive described migration results.
Industrial real-time data classification provided by the invention stores and migration system and method achieves following technique effect:
(1) the industrial automation system data classification storage architecture that the present invention adopts, it is proposed that a set of perfect reasonably a whole set of physical structure of migration and the system of logical organization.
(2) the present invention adopts the data value decision method for magnanimity industrial real-time data storage demand, the method adopts and is worth target function, introduce one group of weight parameter to comprising the time, ask number of users, with the relational degree of other data, the value of associated data, the I/O of different storage device accesses contrast gradient, the influence factors such as the size of data object own carry out quantitative analysis, and adopt the method for moving window, carry out data value dynamic judging fully, it is to increase the accuracy that data value judges.
(3) the present invention adopts data dynamic migration strategy, namely in traditional data migration mechanism, buffer zone is increased, as high-performance equipment is worth the region undetermined before moving compared with lower part data object, if this data object is worth still lower in second time value judges, then can trigger migration event, meanwhile, the maximum value that in buffer zone, data object is worth also is the threshold value upwards moved as low-performance equipment data object. Adopt this mechanism, it is possible to be worth by the data object on high-performance equipment dynamically and assess, the repeatedly migration of effective suppression data object between high low-performance equipment.
Accompanying drawing explanation
Fig. 1 is that industrial real-time data classification provided by the invention stores and migration system configuration diagram;
Fig. 2 is industrial real-time data classification storage provided by the invention and moving method schema.
Embodiment
For the present invention is described better, hereby with a preferred embodiment, and coordinate accompanying drawing that the present invention does detailed explanation, specific as follows:
Industrial real-time data classification storage provided by the invention and migration system are applied in industrial data storage system general at present, and this storage system is used for storage industry magnanimity real time data, it is possible to by SAN network or IP transmitted data on network to carry out data storage.
Specifically, as shown in Figure 1, the industrial real-time data classification that the present embodiment provides stores and migration system, comprise: (it comprises the different storing device of some priority to classification storage system 10, such as Fig. 1, in the present embodiment, classification storage system comprises first device 11, secondary equipment 12 and three grades of equipment 13, wherein, first device 11 is compared with secondary equipment 12 equipment, belong to higher priority devices, and secondary equipment 12 is compared with three grades of equipment 13, belong to higher priority devices), it is worth and judges management device 20, data placement planning management device 30, migration engine controller 40, module 60 is moved/moved back to application server proxy 50 and data. wherein, in the present embodiment, data move/move back that module 60 comprises between first device 11 and secondary equipment 12 first moves/move back module 61, and the 2nd move/move back module 62 between secondary equipment 12 and three grades of equipment 13.
Wherein, classification storage system 10, also comprises buffer memory. Buffer memory in the present embodiment is high-performance buffer memory, such as high-speed cache. Wherein, buffer memory is used for preserving the address being worth lower data object in the storing device of high priority, and these data objects carry out dynamic change with transition process, belong to category undetermined. When occurring data by high to Low migration, if any one data object address chosen has preserved in the buffer, then this data object is migrated to low priority storing device.
In the present embodiment, high priority storing device refers to high performance storing device, such as: solid state hard disc; And low priority storing device refers to the storing device of lower performance, relatively high priority storing device, such as: sas hard disk or sata hard disk. Certainly, the type needing unrestricted choice high priority storing device and low priority storing device that those skilled in the art can store according to data, as long as the performance that the storing device meeting different priorities reads and writes data difference to some extent, thus different data object can be had influence on and store efficiency. Thus the inventive method and system can be applied to the situation needing arbitrarily to optimize based on the difference of this kind of storing device and data object.
When this industrial real-time data classification stores and moves system works, judge that management device 20 carries out obtaining the data object in described classification storage system in real time by value, and calculate the value of data object, and value information is sent to data placement planning management device 30; Data placement planning management device 30 is after value-capture information, and after the quality being worth assessment method being analyzed and weighed according to the result being worth, select data object to be migrated to form migration queue, form data placement plan simultaneously and move strategy to be supplied to migration engine controller 40; Migration engine controller 40 sends migration order to application server proxy according to its content after obtaining data placement plan and migration strategy, its also when initialize application server agency 50 register; Application server proxy 50 receives to move to order from migration engine controller 40 moves/moves back module to forward to corresponding data; Module 60 is moved/moved back to data, carries out the migration of data according to migration order corresponding with it or moves back, and migration results feeds back to migration engine controller.
Wherein, migration engine controller 40 specifically comprises:
Data monitoring module, for monitoring and record the update status of data object and value variation and feed back this update status to described values and judge management device, and the I/O access situation of system is monitored;
And data management module, for regularly inquiring about described data placement planning management device, to carry out the renewal of data information, and send described migration order and receive described migration results.
Industrial real-time data classification storage provided by the invention and moving method, comprise data classification storage and two parts are moved in data classification, and wherein, data classification storage comprises the following steps:
I: carry out data being worth evaluation;
II: placed or migrate to according to data value in suitable level.
Specifically, those skilled in the art are according to the data value obtained in step I, the classification carrying out data according to classification storage architecture general at present stores, as it being placed in suitable level respectively according to the matching degree of the size of data value and the storing device of different priorities, such as first device 11, secondary equipment 12 and three grades of equipment 13. This kind of storage mode can take into account performance of storage system and economy, and can fully take into account the property of value of data object.
As shown in Figure 2, data classification moving method comprises the following steps, and data migration occurs between the different storing device of two performances. Not losing generality, below for transition process between first device 11 and secondary equipment 12 equipment, wherein first device 11 is high priority storing device:
S1: regular monitoring classification storage system, when, after the first threshold value that the storage capacity rate of utilization of advanced storage equipment reaches default, trigger data migration calculates, and enters step S2;
S2: carry out each data object in storing device being worth evaluation, obtain the value of each data object, according to the size of described value, each corresponding data object is sorted;
S3: be default Second Threshold according to accounting, selects data object lower for the sorting out value stored in high priority storing device 11, composition migration queue, and the data object in migration queue is migrated to low priority storing device;
S4: be the 3rd default threshold value according to accounting, the data object address kept current in data object address lower for the remaining sorting out value stored in high priority storing device after step S3 execution and buffer memory is compared, address such as any one data object wherein be kept at as described in buffer memory, then this data object is migrated to low priority storing device, otherwise the internal memory address of this data object is preserved in the buffer, the rest may be inferred, if in buffer memory, data object number of addresses is N after completing all comparisonsbIf, Nb��Nh, then this migration work stops, if Nb>Nh, then value corresponding to the data object address preserved in buffer memory is by sorting successively to little greatly, rejects successively, until remaining data object address number is N from the maximum data object address of valueh, this migration work stops, wherein, and NhFor the preset upper limit of data object number of addresses in buffer memory;
S5: find and be worth as the data object of maximum value described in current buffer memory, and by valuable for institute in the low-performance equipment data object being greater than described maximum value according to value by high to Low order form migration queue also shifting to high-performance equipment.
Also it is, when each system is moved, data object in the storing device of high priority is worth by after high to Low sequence by it, will sequence be that the data object Direct Transfer of default Second Threshold is to low priority storing device from the per-cent accounting for the current capacity of this storing device of minimum value side.And to remaining data object, again the data object that the per-cent accounting for this storing device current data amount from minimum value side in residue sequence is the 3rd threshold value is considered again. The standard considered is whether the address seeing those data objects is existing data address in current cache, also be exactly before data move in whether by those Data object placements in the buffer, in this way, then show that those data objects are worth lower, it is in the 3rd threshold range when at least second time is moved, and need to be adjusted in low priority storing device by high priority storing device by data object corresponding for its address. And this is in the data object not having corresponding address in the 3rd threshold range in buffer memory, then by its address stored in buffer memory, to treat follow-up migration as a reference. And value the maximum of data object corresponding to the address of buffer memory, then the reference moved is carried out as the data object of low priority storing device store.
Further, in above-mentioned S2, adopt the method for moving window, the value that each moment in this moving window calculates sought weighted mean, is specially:
If the width of given window is N, when being respectively V for the current recently data object that N time calculates is worth in this window1��V2����VNTime, then the value calculation formula of current data object is as follows:
V c = &Sigma; i = 1 N &lambda; i V i
Namely Vc passes to data placement planning management device as the value of final data object and follow-up data move/move back module for during data dynamic migration with reference to and use.
Wherein, in above-mentioned S2, when carrying out data object being worth evaluation, according to following formula, value is calculated:
V=w1T+w2C+w3N+w4CT+w5/S
Wherein, T is time factor, and C is access number of users factor, and N is the value factor of the associated data object with notebook data object, and CT is the contrast gradient factor of different storage device, and S is the size factor of data object self, w1��w2��w3��w4And w5It is respectively the weight of each corresponding factor. Specifically:
1) acquisition methods of time factor T is:
First, the moment accepted the interview that acquisition data Object Creation is all after starting: t1��t2��tn, n is positive integer;
Then, calculate the time span T at interval between each orientation1��T2��Tn-1, then:
Ti=ti+1-tiI=1,2 ..., n-1
Finally, time factor T is calculated:
T = &Sigma; i = 1 n - 1 &alpha; i T i
Wherein, ��i, i=1,2 ..., n-1 is one group of weighted value given in advance, and meetsAnd ��1�ܦ�2��...�ܦ�n-1. Due to record data, from being created to, its access time response of current time may have occurred change, therefore, with bigger weight, nearest several time spans is counted mean value calculation so that the result T obtained more meets current practical situation.
2) relational degree factor analysis between data object, it is necessary to find all data objects being associated with notebook data object. To any one data object, its associated data object is defined as follows:
Setting-up time length threshold is Tth, any t0Time data object obj1Accessed, then at t0+TthIn time interval, data object obj2Also accessed, then think data object obj1And obj2It is associated.
Then data object obj1The acquisition methods of value factor N of associated data object as follows:
First, find and any one data object obj1Set of data objects �� (the obj being associated1);
Secondly, find �� (obj1) in the value of all data objects;
Finally, to data object obj1It is as follows that the value of all data objects being associated carries out summation:
N = &Sigma; o b j &Element; &Phi; ( obj 1 ) V o b j
VobjFor the value record of data object obj in migration engine controller.
3) value accessing number of users C then can directly obtain from migration engine controller.
4) the I/O access contrast gradient method of calculation of different storage device are as follows, and owing to both making it is same equipment, the speed of read-write is not identical yet, therefore need read-write contrast gradient separately to be considered:
For different storing device A and B, if it reads contrast gradient is ��r, writing contrast gradient is ��w, then have:
&delta; r = R A / R B ( R B R A ) 2 + 1 ,
&delta; w = W A / W B ( W B W A ) 2 + 1 ,
Wherein, RA��RBRepresent respectively lasting reading on the equipment of A and B two kinds of different performances according to time speed, WA��WBFor the speed continued when writing data of correspondence.
And the I/O access contrast gradient of different storage device is also relevant with I/O access frequency, the recent access frequency the closer to current time enters I/O with relatively authority restatement and accesses contrast gradient calculating. Therefore, by data object, from being established to, current time interval is divided into m section to the present embodiment, then the contrast gradient factor CT of different storage device calculates according to following formula:
C T = &Sigma; i = 1 m &beta; i ( &delta; w &times; FW i + &delta; r &times; FR i )
Wherein, FWiAnd FRiRepresent the read-write frequency of data object within i-th period, ��iRepresent the weighting weight of i-th period, and ��1<��2<��<��m, ��rIt is the reading contrast gradient between two different storage device, ��wIt is write contrast gradient between two different storage device.
5) value of data object size factor S can directly obtain from migration engine controller.
Preferably, the first threshold value in the present embodiment is 80%, and Second Threshold is the 10%, three threshold value is 10%. Certainly, in other classification storage system, it is possible to size by the first threshold value, Second Threshold and the 3rd threshold value is set to other suitable values as required.
The above; being only the specific embodiment of the present invention, but protection scope of the present invention is not limited thereto, the technician of any this area is in the technical scope that the present invention discloses; distortion the present invention done or replacement, all should be encompassed within protection scope of the present invention. Therefore, protection scope of the present invention should be as the criterion with the protection domain of described claim.

Claims (11)

1. an industrial real-time data classification storage and moving method, it is characterised in that, two parts are moved in protection data classification storage and data classification, and wherein, data classification storage comprises the following steps:
I: carry out data being worth evaluation;
II: placed or migrate in suitable level according to data value;
Data classification migration comprises the following steps:
S1: regular monitoring classification storage system, when, after the first threshold value that the storage capacity rate of utilization of high priority storing device reaches default, trigger data migration calculates, and enters step S2;
S2: carry out each data object in storing device being worth evaluation, obtain the value of each data object, according to the size of described value, each corresponding data object is sorted;
S3: be default Second Threshold according to accounting, selects data object lower for the sorting out value stored in high priority storing device, composition migration queue, and the data object in described migration queue is migrated to low priority storing device;
S4: be the 3rd default threshold value according to accounting, the data object address kept current in data object address lower for the remaining sorting out value stored in high priority storing device after step S3 execution and buffer memory is compared, address such as any one data object wherein be kept at as described in buffer memory, then this data object is migrated to low priority storing device, otherwise the internal memory address of this data object is preserved in the buffer, the rest may be inferred, if in buffer memory, data object number of addresses is N after completing all comparisonsbIf, Nb��Nh, then this migration work stops, if Nb>Nh, then value corresponding to the data object address preserved in buffer memory is by sorting successively to little greatly, rejects successively, until remaining data object address number is N from the maximum data object address of valueh, this migration work stops, wherein, and NhFor the preset upper limit of data object number of addresses in buffer memory;
S5: find and be worth as the data object of maximum value described in current buffer memory, and by valuable for institute in the low priority storing device data object being greater than described maximum value according to value by high to Low order form migration queue also shifting to high priority storing device.
2. industrial real-time data classification storage according to claim 1 and moving method, it is characterised in that, in described S2, adopt the method for moving window, the value that each moment in this moving window calculates is sought weighted mean, is specially:
If the width of given window is N, when being respectively V for the current recently data object that N time calculates is worth in this window1��V2����VNTime, then the value calculation formula of current data object is as follows:
V c = &Sigma; i = 1 N &lambda; i V i .
3. industrial real-time data classification storage according to claim 1 and moving method, it is characterised in that, in described S2, when carrying out data object being worth evaluation, according to following formula, value is calculated:
V=w1T+w2C+w3N+w4CT+w5/S
Wherein, T is time factor, and C is access number of users factor, and N is the value factor of the associated data object with notebook data object, and CT is the contrast gradient factor of different storage device, and S is the size factor of data object self, w1��w2��w3��w4And w5It is respectively the weight of each corresponding factor.
4. industrial real-time data classification storage according to claim 3 and moving method, it is characterised in that, the acquisition methods of described time factor T is:
The moment accepted the interview that acquisition data Object Creation is all after starting: t1��t2��tn, n is positive integer;
Calculate the time span T at interval between each access1��T2��Tn-1, then:
Ti=ti+1-tiI=1,2 ..., n-1
Calculate T:
T = &Sigma; i = 1 n - 1 &alpha; i T i ,
Wherein, ��i, i=1,2 ..., n-1 is one group of weighted value given in advance, and meetsAnd ��1�ܦ�2��...�ܦ�n-1��
5. industrial real-time data classification storage according to claim 3 and moving method, it is characterised in that, to any one data object, its associated data object is defined as follows:
Setting-up time length threshold is Tth, any t0Time data object obj1Accessed, then at t0+TthIn time interval, data object obj2Also accessed, then think data object obj1And obj2It is associated.
6. industrial real-time data classification storage according to claim 3 or 5 and moving method, it is characterised in that, data object obj1The acquisition methods of value factor N of associated data object as follows:
Find and data object obj1Set of data objects �� (the obj being associated1);
Find �� (obj1) in the value of all data objects;
To with data object obj1It is as follows that the value of all data objects being associated carries out summation:
N = &Sigma; o b j &Element; &Phi; ( obj 1 ) V o b j ;
VobjFor the value record of data object obj.
7. industrial real-time data classification storage according to claim 3 and moving method, it is characterised in that, by data object, from being established to, current time interval is divided into m section, then the contrast gradient factor CT of different storage device calculates according to following formula:
C T = &Sigma; i = 1 m &beta; i ( &delta; w &times; FW i + &delta; r &times; FR i ) ,
Wherein, FWiAnd FRiRepresent the read-write frequency of data object within i-th period, ��iRepresent the weighting weight of i-th period, and ��1<��2<��<��m, ��rIt is the reading contrast gradient between two different storage device, ��wIt is write contrast gradient between two different storage device.
8. industrial real-time data classification storage according to claim 3 or 7 and moving method, it is characterised in that, for different storing device A and B, if it reads contrast gradient is ��r, writing contrast gradient is ��w, then have:
&delta; r = R A / R B ( R B R A ) 2 + 1 ,
&delta; w = W A / W B ( W B W A ) 2 + 1 ,
Wherein, RA��RBRepresent respectively lasting reading on the equipment of A and B two kinds of different performances according to time speed, WA��WBFor the speed continued when writing data of correspondence.
9. industrial real-time data classification storage according to claim 1 and moving method, it is characterised in that, described first threshold value is 80%, and Second Threshold is the 10%, three threshold value is 10%.
10. an industrial real-time data classification stores and migration system, it is characterised in that, comprising:
Classification storage system, comprise the storing device that some the priority for storing data object are different, also comprise buffer memory, wherein, buffer memory is used for preserving the address being worth lower data object in the storing device of high priority, and these data objects carry out dynamic change with transition process, when occurring data by high to Low migration, if the data object address that any one is chosen has preserved in the buffer, then this data object is migrated to low priority storing device;
It is worth and judges management device, for the data object obtained in real time in described classification storage system, and calculate the value of data object;
Data placement planning management device, is worth, from described, the value judging management device for obtaining, and selects data object to be migrated to form migration queue according to the result being worth, and forms data placement plan and migration strategy;
Migration engine controller, orders to application server proxy for obtaining described data placement plan and migration strategy send migration, and described application server proxy is registered;
Application server proxy, for registering to migration engine controller during initialize, and receives described migration order and moves/move back module to forward to corresponding data;
Module is moved/moved back to data, is located between the different storing device of every two priority respectively, for carrying out the migration of data according to described migration order or move back, and migration results feeds back to described migration engine controller.
11. industrial real-time data classifications according to claim 10 store and migration system, it is characterised in that, described migration engine controller comprises:
Data monitoring module, for monitoring and record the update status of data object and value variation and feed back this update status to described values and judge management device, and the I/O access situation of system is monitored;
And data management module, for regularly inquiring about described data placement planning management device, to carry out the renewal of data information, and send described migration order and receive described migration results.
CN201510969294.3A 2015-12-22 2015-12-22 A kind of industrial real-time data classification storage and moving method Active CN105653591B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510969294.3A CN105653591B (en) 2015-12-22 2015-12-22 A kind of industrial real-time data classification storage and moving method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510969294.3A CN105653591B (en) 2015-12-22 2015-12-22 A kind of industrial real-time data classification storage and moving method

Publications (2)

Publication Number Publication Date
CN105653591A true CN105653591A (en) 2016-06-08
CN105653591B CN105653591B (en) 2019-02-05

Family

ID=56477595

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510969294.3A Active CN105653591B (en) 2015-12-22 2015-12-22 A kind of industrial real-time data classification storage and moving method

Country Status (1)

Country Link
CN (1) CN105653591B (en)

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106293509A (en) * 2016-07-12 2017-01-04 乐视控股(北京)有限公司 Date storage method and system
CN106406253A (en) * 2015-07-30 2017-02-15 西门子公司 System and method for control and/or analytics of an industrial process
CN106990924A (en) * 2017-04-07 2017-07-28 广东浪潮大数据研究有限公司 A kind of data migration method and system
CN108052278A (en) * 2017-10-09 2018-05-18 清华大学 The storage controlling method and storage system of electron microscopic data
CN108491165A (en) * 2018-03-27 2018-09-04 中国农业银行股份有限公司 A kind of data migration method and system for being classified storage
CN108509147A (en) * 2017-02-28 2018-09-07 慧与发展有限责任合伙企业 Data block migration
CN108874311A (en) * 2018-05-29 2018-11-23 北京盛和大地数据科技有限公司 Merge the data migration method and device in storage system
CN108920344A (en) * 2018-05-11 2018-11-30 南京南瑞继保电气有限公司 A kind of storage method and equipment, computer readable storage medium
CN109582223A (en) * 2018-10-31 2019-04-05 华为技术有限公司 A kind of method and device of internal storage data migration
CN110941513A (en) * 2019-11-22 2020-03-31 浪潮电子信息产业股份有限公司 Data reconstruction method and related device
CN111367469A (en) * 2020-02-16 2020-07-03 苏州浪潮智能科技有限公司 Layered storage data migration method and system
CN112261097A (en) * 2020-10-15 2021-01-22 科大讯飞股份有限公司 Object positioning method for distributed storage system and electronic equipment
CN112685454A (en) * 2021-03-10 2021-04-20 江苏金恒信息科技股份有限公司 Industrial data hierarchical storage system and method and industrial data hierarchical query method
CN112860188A (en) * 2021-02-09 2021-05-28 山东英信计算机技术有限公司 Data migration method, system, device and medium
CN113342781A (en) * 2021-06-29 2021-09-03 深圳前海微众银行股份有限公司 Data migration method, device, equipment and storage medium
CN113741819A (en) * 2021-09-15 2021-12-03 第四范式(北京)技术有限公司 Method and device for hierarchical storage of data
CN114611572A (en) * 2022-01-28 2022-06-10 北京工商大学 Data hierarchical storage algorithm based on improved RBF neural network
CN115062061A (en) * 2022-06-27 2022-09-16 艾象科技(深圳)股份有限公司 Commodity ERP management system and ERP management method
CN117193656A (en) * 2023-02-27 2023-12-08 自然资源部信息中心 Data hierarchical storage migration flow method

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109918448A (en) * 2019-03-06 2019-06-21 电子科技大学 A kind of cloud storage data classification method based on user behavior

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060015529A1 (en) * 2004-07-15 2006-01-19 Hitachi, Ltd. Method and apparatus of hierarchical storage management based on data value
CN101101563A (en) * 2007-07-23 2008-01-09 清华大学 Migration management based on massive data classified memory system
US20080154988A1 (en) * 2005-06-10 2008-06-26 Fujitsu Limited Hsm control program and method
WO2013097119A1 (en) * 2011-12-28 2013-07-04 华为技术有限公司 Method and device for realizing multilevel storage in file system
CN104598489A (en) * 2013-10-31 2015-05-06 大连易维立方技术有限公司 Method for updating book information based on crawler strategy of specialized search engine

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060015529A1 (en) * 2004-07-15 2006-01-19 Hitachi, Ltd. Method and apparatus of hierarchical storage management based on data value
US20080154988A1 (en) * 2005-06-10 2008-06-26 Fujitsu Limited Hsm control program and method
CN101101563A (en) * 2007-07-23 2008-01-09 清华大学 Migration management based on massive data classified memory system
WO2013097119A1 (en) * 2011-12-28 2013-07-04 华为技术有限公司 Method and device for realizing multilevel storage in file system
CN104598489A (en) * 2013-10-31 2015-05-06 大连易维立方技术有限公司 Method for updating book information based on crawler strategy of specialized search engine

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
吕帅 等: "海量信息分级存储数据迁移策略研究", 《计算机工程与科学》 *
李俊杰: "面向海量存储系统的分层存储技术研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 *

Cited By (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106406253A (en) * 2015-07-30 2017-02-15 西门子公司 System and method for control and/or analytics of an industrial process
US10303128B2 (en) 2015-07-30 2019-05-28 Siemens Aktiengesellschaft System and method for control and/or analytics of an industrial process
CN106406253B (en) * 2015-07-30 2019-12-20 西门子公司 System and method for controlling and/or analyzing an industrial process
CN106293509A (en) * 2016-07-12 2017-01-04 乐视控股(北京)有限公司 Date storage method and system
CN108509147A (en) * 2017-02-28 2018-09-07 慧与发展有限责任合伙企业 Data block migration
CN106990924A (en) * 2017-04-07 2017-07-28 广东浪潮大数据研究有限公司 A kind of data migration method and system
CN108052278A (en) * 2017-10-09 2018-05-18 清华大学 The storage controlling method and storage system of electron microscopic data
CN108491165A (en) * 2018-03-27 2018-09-04 中国农业银行股份有限公司 A kind of data migration method and system for being classified storage
CN108920344A (en) * 2018-05-11 2018-11-30 南京南瑞继保电气有限公司 A kind of storage method and equipment, computer readable storage medium
CN108920344B (en) * 2018-05-11 2021-11-12 南京南瑞继保电气有限公司 Storage method and device and computer readable storage medium
CN108874311A (en) * 2018-05-29 2018-11-23 北京盛和大地数据科技有限公司 Merge the data migration method and device in storage system
CN109582223A (en) * 2018-10-31 2019-04-05 华为技术有限公司 A kind of method and device of internal storage data migration
US11809732B2 (en) 2018-10-31 2023-11-07 Huawei Technologies Co., Ltd. Method and apparatus of data migration based on use algorithm
US11435931B2 (en) 2018-10-31 2022-09-06 Huawei Technologies Co., Ltd. Memory data migration method and apparatus where cold data is migrated to shared storage prior to storing in destination storage
CN110941513A (en) * 2019-11-22 2020-03-31 浪潮电子信息产业股份有限公司 Data reconstruction method and related device
CN110941513B (en) * 2019-11-22 2022-03-22 浪潮电子信息产业股份有限公司 Data reconstruction method and related device
CN111367469B (en) * 2020-02-16 2022-07-08 苏州浪潮智能科技有限公司 Method and system for migrating layered storage data
CN111367469A (en) * 2020-02-16 2020-07-03 苏州浪潮智能科技有限公司 Layered storage data migration method and system
CN112261097A (en) * 2020-10-15 2021-01-22 科大讯飞股份有限公司 Object positioning method for distributed storage system and electronic equipment
CN112261097B (en) * 2020-10-15 2023-11-24 科大讯飞股份有限公司 Object positioning method for distributed storage system and electronic equipment
CN112860188A (en) * 2021-02-09 2021-05-28 山东英信计算机技术有限公司 Data migration method, system, device and medium
CN112685454A (en) * 2021-03-10 2021-04-20 江苏金恒信息科技股份有限公司 Industrial data hierarchical storage system and method and industrial data hierarchical query method
CN113342781A (en) * 2021-06-29 2021-09-03 深圳前海微众银行股份有限公司 Data migration method, device, equipment and storage medium
CN113741819A (en) * 2021-09-15 2021-12-03 第四范式(北京)技术有限公司 Method and device for hierarchical storage of data
CN114611572A (en) * 2022-01-28 2022-06-10 北京工商大学 Data hierarchical storage algorithm based on improved RBF neural network
CN114611572B (en) * 2022-01-28 2024-05-14 北京工商大学 Data hierarchical storage algorithm based on improved RBF neural network
CN115062061A (en) * 2022-06-27 2022-09-16 艾象科技(深圳)股份有限公司 Commodity ERP management system and ERP management method
CN117193656A (en) * 2023-02-27 2023-12-08 自然资源部信息中心 Data hierarchical storage migration flow method

Also Published As

Publication number Publication date
CN105653591B (en) 2019-02-05

Similar Documents

Publication Publication Date Title
CN105653591A (en) Hierarchical storage and migration method of industrial real-time data
CN101105772B (en) Method for controlling computer readable memory and manager data unit device
US8793427B2 (en) Remote memory for virtual machines
US8176233B1 (en) Using non-volatile memory resources to enable a virtual buffer pool for a database application
CN106886485B (en) System capacity analysis and prediction method and device
US10140034B2 (en) Solid-state drive assignment based on solid-state drive write endurance
US9971548B1 (en) Storage resource management employing performance analytics
US9612758B1 (en) Performing a pre-warm-up procedure via intelligently forecasting as to when a host computer will access certain host data
Sethi et al. RecShard: statistical feature-based memory optimization for industry-scale neural recommendation
US20190361811A1 (en) Data processing apparatus and prefetch method
CN107122126B (en) Data migration method, device and system
CA2442188A1 (en) Methods and mechanisms for proactive memory management
CN102857560B (en) A kind of cloud towards multi-service application stores data distribution method
CN111367469B (en) Method and system for migrating layered storage data
Puttaswamy et al. Frugal storage for cloud file systems
US10997080B1 (en) Method and system for address table cache management based on correlation metric of first logical address and second logical address, wherein the correlation metric is incremented and decremented based on receive order of the first logical address and the second logical address
US20140244892A1 (en) Assigning a weighting to host quality of service indicators
CN106233265A (en) Access frequency hierarchical structure is used for evicting from the selection of target
CN104699424A (en) Page hot degree based heterogeneous memory management method
CN104268099A (en) Management data reading and writing method and device
CN104111898A (en) Hybrid storage system based on multidimensional data similarity and data management method
CN107665219A (en) A kind of blog management method and device
CN108829343B (en) Cache optimization method based on artificial intelligence
CN103020077A (en) Method for managing memory of real-time database of power system
Thomasian et al. Disk scheduling policies with lookahead

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20190904

Address after: 316000 No. 3 307-6, No. 10, West Xingzhou Avenue, Yancang Street, Dinghai District, Zhoushan City, Zhejiang Province

Patentee after: ZHEJIANG ZHOUSHAN TO CONTROL INTELLIGENT EQUIPMENT TECHNOLOGY CO., LTD.

Address before: 310053 Hangzhou Province, Binjiang District Province, No. six and No. 309 Road, the center of science and Technology Park (high tech Zone) ()

Patentee before: Zhejiang Supcon Research Co., Ltd.