CN106294671A - A kind of data migration method and device - Google Patents
A kind of data migration method and device Download PDFInfo
- Publication number
- CN106294671A CN106294671A CN201610636095.5A CN201610636095A CN106294671A CN 106294671 A CN106294671 A CN 106294671A CN 201610636095 A CN201610636095 A CN 201610636095A CN 106294671 A CN106294671 A CN 106294671A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- strategy
- storage strategy
- reading frequency
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/119—Details of migration of file systems
Abstract
The embodiment of the invention discloses a kind of data migration method and device, be applied to name node, the method includes: obtain the first storage strategy and reading frequency of purpose data；Determine the purpose reading frequency scope of described first storage strategy, and judge that described reading frequency is whether in the range of described purpose reading frequency；If it has not, determine the second storage strategy according to described reading frequency, it is described second storage strategy by described first storage strategy modification, and according to purpose data described in described second storage policy migration.The technical scheme that the application embodiment of the present invention provides, improves the resource utilization of HDFS.
Description
Technical field
The present invention relates to technical field of data processing, particularly to a kind of data migration method and device.
Background technology
HDFS (Hadoop Distributed File System, the distributed file system of big data structure) is used for depositing
Storage data, including: a NameNode (name node) and at least one DataNode (back end).Wherein, NameNode
Mainly it is responsible for the NameSpace of HDFS, HDFS configuration information and the duplication etc. of memory block；DataNode is file storage
Elementary cell, memory block is stored in local file system by it, preserves the metadata of memory block, the most periodically will be all
Memory block information is sent to NameNode, and a memory block is the copy of data.It addition, DataNode is divided into again disk layer
Node and filing node layer, disk layer node has higher computing capability；Filing node layer has higher memory density, relatively
Low computing capability.
Summary of the invention
The embodiment of the invention discloses a kind of data migration method and device, to improve the resource utilization of HDFS.
For reaching above-mentioned purpose, the embodiment of the invention discloses a kind of data migration method, be applied to name node, described
Method includes:
Obtain the first storage strategy and reading frequency of purpose data；
Determine the purpose reading frequency scope of described first storage strategy, and judge that whether described reading frequency is at described mesh
Reading frequency in the range of；
If it has not, determine the second storage strategy according to described reading frequency, it is described by described first storage strategy modification
Second storage strategy, and according to purpose data described in described second storage policy migration.
In one embodiment of the invention, the first storage strategy and reading frequency of described acquisition purpose data, including:
Obtain the first storage strategy of purpose data, and according to default storage strategy and the corresponding pass between the data cycle
System, determines the first data cycle that described first storage strategy is corresponding；
Obtain the reading times of described purpose data；
According to described first data cycle and described reading times, determine the reading frequency of described purpose data.
In one embodiment of the invention, described the second storage strategy is determined according to described reading frequency, by described
One storage strategy modification is described second storage strategy, and stores purpose data described in policy migration according to described second, including:
Determine the second storage strategy according to described reading frequency, be the second storage plan by described first storage strategy modification
Slightly, and the transition state of described purpose data is set to state to be migrated；
When current time reaches Preset Time, according to described second storage strategy, it is described to be migrated to transition state
The purpose data of state migrate.
In one embodiment of the invention, described according to described second storage policy migration described in purpose data, including:
When described first storage strategy is temperature data store strategy, when the second storage strategy is cold data store strategy, delete
Except being stored in the memory block of purpose data described in disk layer node.
In one embodiment of the invention, described according to described second storage policy migration described in purpose data, including:
When described first storage strategy is cold data store strategy, when the second storage strategy is temperature data store strategy, will
It is stored in the memory block of purpose data described in filing node layer and is copied to disk layer node.
For reaching above-mentioned purpose, the embodiment of the invention also discloses a kind of data migration device, be applied to name node, institute
State device to include:
Acquiring unit, for obtaining the first storage strategy and reading frequency of purpose data；
Judging unit, for determining the purpose reading frequency scope of described first storage strategy, and judges described reading frequently
Whether rate is in the range of described purpose reading frequency；
Migration units, in the case of described judging unit judged result is no, determines according to described reading frequency
Second storage strategy, is described second storage strategy by described first storage strategy modification, and according to described second storage strategy
Migrate described purpose data.
In one embodiment of the invention, described acquiring unit, specifically for:
Obtain the first storage strategy of purpose data, and according to default storage strategy and the corresponding pass between the data cycle
System, determines the first data cycle that described first storage strategy is corresponding；Obtain the reading times of described purpose data；According to described
First data cycle and described reading times, determine the reading frequency of described purpose data.
In one embodiment of the invention, described device also includes:
Unit is set, for determining the second storage strategy according to described reading frequency, by described first storage strategy modification
It is the second storage strategy, and the transition state of described purpose data is set to state to be migrated；
Described migration units, is additionally operable to when current time reaches Preset Time, according to described second storage strategy, to moving
Shifting state is that the purpose data of described state to be migrated migrate.
In one embodiment of the invention, described migration units, specifically for:
When described first storage strategy is temperature data store strategy, when the second storage strategy is cold data store strategy, delete
Except being stored in the memory block of purpose data described in disk layer node.
In one embodiment of the invention, described migration units, specifically for:
When described first storage strategy is cold data store strategy, when the second storage strategy is temperature data store strategy, will
It is stored in the memory block of purpose data described in filing node layer and is copied to disk layer node.
Embodiments providing a kind of data migration method and device, name node can obtain purpose number in real time
According to reading frequency, when reading frequency is in the range of the purpose reading frequency of purpose data, determine second according to reading frequency
Storage strategy, is the second storage strategy by the storage strategy modification of purpose data, finally, according to the second storage policy migration purpose
Data.Obtain the reading frequency of purpose data the most in real time, update the storage strategy of purpose data, and move target in real time
Data, improve the resource utilization of HDFS.
Accompanying drawing explanation
In order to be illustrated more clearly that the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
In having technology to describe, the required accompanying drawing used is briefly described, it should be apparent that, the accompanying drawing in describing below is only this
Some embodiments of invention, for those of ordinary skill in the art, on the premise of not paying creative work, it is also possible to
Other accompanying drawing is obtained according to these accompanying drawings.
The schematic flow sheet of a kind of data migration method that Fig. 1 provides for the embodiment of the present invention；
The structural representation of a kind of HDFS that Fig. 2 provides for the embodiment of the present invention；
The structural representation of the another kind of HDFS that Fig. 3 (a) provides for the embodiment of the present invention；
The structural representation of the another kind of HDFS that Fig. 3 (b) provides for the embodiment of the present invention；
The structural representation of the another kind of HDFS that Fig. 3 (c) provides for the embodiment of the present invention；
The structural representation of a kind of data migration device that Fig. 4 provides for the embodiment of the present invention.
Detailed description of the invention
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Describe, it is clear that described embodiment is only a part of embodiment of the present invention rather than whole embodiments wholely.Based on
Embodiment in the present invention, it is every other that those of ordinary skill in the art are obtained under not making creative work premise
Embodiment, broadly falls into the scope of protection of the invention.
In HDFS, the data of storage have 3 kinds, are respectively as follows: dsc data, temperature data and cold data.Wherein, dsc data is interviewed
The frequency (being properly termed as again reading frequency) asked is the highest, and generally every day is accessed at least one times；Temperature data are accessed for frequency
Rate is accessed several times the most weekly；It is the lowest that cold data are accessed for number of times, and generally some months or 1 year is accessed several times.
For the data of this 3 type, HFDS is provided with 3 kinds of storage strategies, is respectively as follows: dsc data and stores data tactful, warm storage
Tactful and cold data store strategy.If to a data D application dsc data storage strategy, then all copies of these data D need
All store in disk layer node；If to data D application temperature data store strategy, then a copy of data D is stored in disk
In node layer, other copies of data D need to be stored in filing node layer；If data D are applied cold data store strategy, then
All copies of data D are required for being stored in filing node layer.
Below by specific embodiment, the present invention is described in detail.
With reference to the schematic flow sheet of a kind of data migration method that Fig. 1, Fig. 1 provide for the embodiment of the present invention, the method should
NameNode in HDFS, refers to Fig. 2, HDFS and includes: NameNode100, more than one saves as disk layer
Put the DataNode of 210 and more than one is as the DataNode filing node layer 220.The method includes:
S101: obtain the first storage strategy and reading frequency of purpose data；
Wherein, storage strategy includes: dsc data stores data store strategy tactful, warm and cold data store strategy.
NameNode100 can be each data configuration storage strategy, and according to each data of storage policy store of configuration.
As: storage one data D are presently required1, then NameNode100 can be data D1Distribution dsc data storage strategy, by data D1
All copies be stored in disk layer node 210, store data D1Afterwards, along with data D1It is accessed for frequency, namely reads
Take the change of frequency, adjust data D1Storage strategy, and according to the storage strategy after adjusting, migrate data D that stored1's
Copy.
The storage strategy of data is relevant with reading frequency, therefore when determining purpose data the need of migration, and Ke Yixian
Obtain the first storage strategy and reading frequency of purpose data.
In actual application, carry out storing and reading data by NameNode100.In one embodiment of the invention,
NameNode100 directly can be calculated according to the reading times of data the reading frequency of these data, and store data
Reading frequency, so, when being necessary to determine whether move target data, NameNode100 can directly obtain purpose data
Reading times.
In another embodiment of the present invention, NameNode100 can only record the reading times of data, it addition,
NameNode100 can also store the data cycle of each storage strategy corresponding.Here, reading times is corresponding with the time；Number
Can be a value determined according to the cycle, such as 7 days, it is also possible to be a span, such as 7 days～15 days.When it needs to be determined that be
During no move target data, NameNode100 can obtain the first storage strategy that purpose data are corresponding, then obtains the first storage
The first data cycle that strategy is corresponding, it is possible to according to the time that reading times is corresponding, it is thus achieved that the first number of distance current time
According to the reading times of purpose data in the cycle, according to the reading frequency of below equation acquisition purpose data:
It is assumed that purpose digital independent number of times is respectively as follows: [2015.10.1 10:00,1 time], [2015.10.1 11:00,1
Secondary], [2015.10.1 12:00,1 time], [2015.10.2 10:00,1 time], [2015.10.2 11:00,1 time], if currently
Moment is 2015.10.2 12:00, and the first data cycle was 1 day, then can obtain the first data cycle of distance current time
The reading times of interior purpose data is: [2015.10.1 12:00,1 time], [2015.10.2 10:00,1 time], [2015.10.2
11:00,1 time], the most totally 3 times, now determine that the reading frequency of purpose data is: 3 times/1 day=3 times/day.
In one preferably implementation, what above-mentioned reading times and data cycle can be corresponding is stored in an information
In table, refer to table 1, a kind of information table that table 1 provides for the embodiment of the present invention.
Table 1
| Data | The data cycle | Reading times | Storage strategy |
| Data1 | Age < 7 day | 20 | Hot |
| Data2 | 7 days < Age < January | 5 | Warm |
| Data3 | < Age < March in January | 5 | Cold |
| … | … | … | … |
Wherein, Hot is dsc data storage strategy, and Warm is temperature data store strategy, and Cold is cold data store strategy.This
Sample, NameNode100 directly can obtain data cycle and the reading times of purpose data, and then determine from above-mentioned information table
The reading frequency of purpose data.
S102: determine the purpose reading frequency scope of the first storage strategy, and judge that whether above-mentioned reading frequency is in purpose
In the range of reading frequency, if it has not, perform S103；
NameNode100 can also arrange the reading frequency scope of each storage strategy, e.g., can arrange: the reading of Hot
Take frequency range for [2 times/day, ∞), the reading frequency scope of Warm be [2 times/week, 14 times/all), the reading frequency model of Cold
Enclose for plan [0,8 time/week).When determining purpose data the need of migration, can first determine that the reading frequency of purpose data is
The no purpose reading frequency scope corresponding at the first storage strategy: if the reading frequency of purpose data is in purpose reading frequency scope
In, then may indicate that the storage strategy of purpose data is without updating, it is not necessary to purpose data are migrated；If purpose data
Reading frequency not in the range of purpose reading frequency, then may indicate that the storage strategy of purpose data needs amendment, purpose data
Need to migrate, namely perform S103.
S103: determine the second storage strategy according to above-mentioned reading frequency, is the second storage plan by the first storage strategy modification
Slightly, and according to second policy migration purpose data are stored.
When determining that purpose data need to migrate, it may be determined that the reading frequency of purpose data is in the reading of which kind of storage strategy
Take in frequency range, and then determine the second storage strategy, such as: the reading frequency of purpose data in the range of the reading frequency of Hot,
Then may determine that the second storage strategy is Hot.
After determining the second storage strategy, it is possible to storing strategy modification by the first of purpose data is the second storage plan
Slightly, and then store policy migration purpose data according to second, it is, the DataNode that the copy of move target data is the most corresponding
In.
In one embodiment of the invention, HDFS needs to process substantial amounts of data, if processing peak period pair in data
Data migrate, and will affect the data processing speed of HDFS, and the efficiency of Data Migration is the lowest.Based on this feelings
Condition, can be each data configuration transition state, need the transition state of data migrated could be arranged to state to be migrated (as
" Doing "), it is not necessary to the transition state of the data of migration could be arranged to transition state (such as " Done ")；It addition,
Can also pre-set a time in NameNode100, when arriving this Preset Time, NameNode100 scans each number
According to transition state, it is thus achieved that transition state is the storage strategy of the data of Doing, and according to obtain storage strategy data are entered
Row migrates.After migration terminates, the transition state migrating successful data can be set to Done, it is to avoid data are continuous
Migration, take the calculating resource of HDFS, affect the data processing speed of HDFS.
Here, Preset Time can be: morning 00:00, morning, the data of the HDFS such as 03:00 processed the non-peak moment.
It addition, when carrying out Data Migration, it is to avoid in all of Data Migration to part disk layer node or partly will return
In shelves node layer, can be by each DataNode (the disk layer node migrating to correspondence of the copy load balancing of purpose data
Or filing node layer) in, it is, each DataNode of correspondence stores the copy of purpose data.
In one embodiment of the invention, if the first storage strategy is Warm, a pair of current purpose data is shown
This F1Being stored in disk layer node 210, other all of copies are stored in filing node layer 220, if now determining second
Storage strategy is Cold, can be by copy F1Directly mobile to filing in node layer 220；Further, since in filing node layer 220
Store the copy of purpose data, if also by copy F1Directly in mobile filing node layer 220, it will take the calculating of HDFS
Resource and bandwidth, and also the memory space of filing node layer 220 can be taken, therefore can be by copy F1At disk layer node
Delete in 210；
If the first storage strategy is Cold, show that all copies of current purpose data are stored in filing node layer 220
In, if now determining, the second storage strategy is Warm, for avoiding data moving process takies the calculating resource of HDFS and bandwidth,
The copy of purpose data can be replicated in disk layer node 210.
It is noted that in the embodiment of the present invention, Data Migration may include that the duplication of data, the deletion of data and
Data mobile etc..Here, the movement of data can be: directly data is moved from storage position, source to purpose storage position；
Can also be: shear data in storage position, source to replicate these data etc. in purpose storage position.
Below in conjunction with the transition process of example explanation data, referring to Fig. 3 (a), shown in Fig. 3 (a), HDFS includes one
Individual NameNode100 and 6 DataNode, in these 6 DataNode, 3 as disk layer node 210,3 conduct filing layer
Node 220, the storage strategy of data Data1 is Hot, its 3 copy Data1 ', Data1 " and Data1 " ' it is respectively stored in 3
In disk layer node 210.
After Data1 stores a period of time, the reading frequency of Data1 reduces, and the reading frequency of Data1 is in the reading of Warm
In frequency range, then need to be stored strategy Hot and be revised as Warm, and its transition state is set to Doing, when currently
When moment reaches Preset Time, can be by Data1 " and Data1 " ' migrate to load balancing file in node layer 220, meanwhile,
The transition state of Data1 is set to Done, refers to Fig. 3 (b)；
After a period of time, if the reading frequency of Data1 reduces again, its reading frequency is in the range of the reading frequency of Cold
, then need the storage strategy Warm of Data1 is revised as Cold, and its transition state be set to Doing, work as current time
When reaching Preset Time, Data1 ' directly can be deleted, Data1 " and Data1 " ' without change, meanwhile, moving Data1
Shifting state is set to Done, refers to Fig. 3 (c)；
After a period of time, if the reading frequency of Data1 improves, and its reading frequency is in the reading frequency scope of Warm
In, then need the storage strategy Cold of Data1 is revised as Warm, and its transition state is set to Doing, when current
When quarter reaches Preset Time, can be according to Data1 " or Data1 " ' in disk layer node 210, directly replicate a Data1 ',
Data1 " and Data1 " ' without change, meanwhile, the transition state of Data1 is set to Done, refers to Fig. 3 (b)；
After a period of time, if the reading frequency of Data1 improves again, and its reading frequency is at the reading frequency model of Hot
In enclosing, then need the storage strategy Warm of Data1 is revised as Hot, and its transition state is set to Doing, when currently
When moment reaches Preset Time, can be by Data1 " and Data1 " ' migrate to load balancing in disk layer node 210, meanwhile,
The transition state of Data1 is set to Done, refers to Fig. 3 (a).
Embodiments providing a kind of data migration method, name node can obtain the reading of purpose data in real time
Take frequency, when reading frequency is in the range of the purpose reading frequency of purpose data, determine the second storage plan according to reading frequency
Slightly, it is the second storage strategy by the storage strategy modification of purpose data, finally, according to the second storage policy migration purpose data.
Obtain the reading frequency of purpose data the most in real time, update the storage strategy of purpose data, and move target data in real time,
Improve the resource utilization of HDFS.
With reference to the structural representation of a kind of data migration device that Fig. 4, Fig. 4 provide for the embodiment of the present invention, this device should
NameNode in HDFS, refers to Fig. 2, HDFS and includes: NameNode100, more than one saves as disk layer
Put the DataNode of 210 and more than one is as the DataNode filing node layer 220.This device includes:
Acquiring unit 401, for obtaining the first storage strategy and reading frequency of purpose data；
Judging unit 402, for determining the purpose reading frequency scope of described first storage strategy, and judges described reading
Whether frequency is in the range of described purpose reading frequency；
Migration units 403, in the case of described judging unit 402 judged result is no, reads frequency according to described
Rate determines the second storage strategy, is described second storage strategy by described first storage strategy modification, and deposits according to described second
Purpose data described in storage policy migration.
In one embodiment of the invention, described acquiring unit 401, specifically for:
Obtain the first storage strategy of purpose data, and according to default storage strategy and the corresponding pass between the data cycle
System, determines the first data cycle that described first storage strategy is corresponding；Obtain the reading times of described purpose data；According to described
First data cycle and described reading times, determine the reading frequency of described purpose data.
In one embodiment of the invention, described data migration device can also include:
Unit (not shown in Fig. 4) is set, for determining the second storage strategy according to described reading frequency, by described first
Storage strategy modification is the second storage strategy, and the transition state of described purpose data is set to state to be migrated；
In this case, described migration units (not shown in Fig. 4), it is additionally operable to when current time reaches Preset Time,
According to described second storage strategy, it is that the purpose data of described state to be migrated migrate to transition state.
In one embodiment of the invention, described migration units 403, specifically for:
When described first storage strategy is temperature data store strategy, when the second storage strategy is cold data store strategy, delete
Except being stored in the memory block of purpose data described in disk layer node.
In one embodiment of the invention, described migration units 403, specifically for:
When described first storage strategy is cold data store strategy, when the second storage strategy is temperature data store strategy, will
It is stored in the memory block of purpose data described in filing node layer and is copied to disk layer node.
Embodiments providing a kind of data migration device, name node can obtain the reading of purpose data in real time
Take frequency, when reading frequency is in the range of the purpose reading frequency of purpose data, determine the second storage plan according to reading frequency
Slightly, it is the second storage strategy by the storage strategy modification of purpose data, finally, according to the second storage policy migration purpose data.
Obtain the reading frequency of purpose data the most in real time, update the storage strategy of purpose data, and move target data in real time,
Improve the resource utilization of HDFS.
For device embodiment, owing to it is substantially similar to embodiment of the method, so describe is fairly simple, relevant
Part sees the part of embodiment of the method and illustrates.
It should be noted that in this article, the relational terms of such as first and second or the like is used merely to a reality
Body or operation separate with another entity or operating space, and deposit between not necessarily requiring or imply these entities or operating
Relation or order in any this reality.And, term " includes ", " comprising " or its any other variant are intended to
Comprising of nonexcludability, so that include that the process of a series of key element, method, article or equipment not only include that those are wanted
Element, but also include other key elements being not expressly set out, or also include for this process, method, article or equipment
Intrinsic key element.In the case of there is no more restriction, statement " including ... " key element limited, it is not excluded that
Including process, method, article or the equipment of described key element there is also other identical element.
One of ordinary skill in the art will appreciate that all or part of step realizing in said method embodiment is can
Completing instructing relevant hardware by program, described program can be stored in computer read/write memory medium,
The storage medium obtained designated herein, such as: ROM/RAM, magnetic disc, CD etc..
The foregoing is only presently preferred embodiments of the present invention, be not intended to limit protection scope of the present invention.All
Any modification, equivalent substitution and improvement etc. made within the spirit and principles in the present invention, are all contained in protection scope of the present invention
In.
Claims (10)
1. a data migration method, it is characterised in that being applied to name node, described method includes:
Obtain the first storage strategy and reading frequency of purpose data；
Determine the purpose reading frequency scope of described first storage strategy, and judge whether described reading frequency is read in described purpose
Take in frequency range；
If it has not, determine the second storage strategy according to described reading frequency, it is described second by described first storage strategy modification
Storage strategy, and according to purpose data described in described second storage policy migration.
Method the most according to claim 1, it is characterised in that the first storage strategy and reading of described acquisition purpose data
Frequency, including:
Obtain the first storage strategy of purpose data, and according to default storage strategy and the corresponding relation between the data cycle, really
The first data cycle that fixed described first storage strategy is corresponding；
Obtain the reading times of described purpose data；
According to described first data cycle and described reading times, determine the reading frequency of described purpose data.
Method the most according to claim 1, it is characterised in that described determine the second storage plan according to described reading frequency
Slightly, it is described second storage strategy by described first storage strategy modification, and according to mesh described in described second storage policy migration
Data, including:
Determine the second storage strategy according to described reading frequency, be the second storage strategy by described first storage strategy modification, and
The transition state of described purpose data is set to state to be migrated；
When current time reaches Preset Time, according to described second storage strategy, it is described state to be migrated to transition state
Purpose data migrate.
Method the most according to claim 1, it is characterised in that described according to purpose described in described second storage policy migration
Data, including:
When described first storage strategy is temperature data store strategy, and when the second storage strategy is cold data store strategy, deletion is deposited
Storage memory block of purpose data described in disk layer node.
Method the most according to claim 1, it is characterised in that described according to purpose described in described second storage policy migration
Data, including:
When described first storage strategy is cold data store strategy, when the second storage strategy is temperature data store strategy, will storage
Described in filing node layer, the memory block of purpose data is copied to disk layer node.
6. a data migration device, it is characterised in that being applied to name node, described device includes:
Acquiring unit, for obtaining the first storage strategy and reading frequency of purpose data；
Judging unit, for determining the purpose reading frequency scope of described first storage strategy, and judges that described reading frequency is
No in the range of described purpose reading frequency；
Migration units, in the case of described judging unit judged result is no, determines second according to described reading frequency
Storage strategy, is described second storage strategy by described first storage strategy modification, and according to described second storage policy migration
Described purpose data.
Device the most according to claim 6, it is characterised in that described acquiring unit, specifically for:
Obtain the first storage strategy of purpose data, and according to default storage strategy and the corresponding relation between the data cycle, really
The first data cycle that fixed described first storage strategy is corresponding；Obtain the reading times of described purpose data；According to described first
Data cycle and described reading times, determine the reading frequency of described purpose data.
Device the most according to claim 6, it is characterised in that described device also includes:
Unit is set, for determining the second storage strategy according to described reading frequency, is the by described first storage strategy modification
Two storage strategies, and the transition state of described purpose data is set to state to be migrated；
Described migration units, is additionally operable to when current time reaches Preset Time, according to described second storage strategy, to migrating shape
State is that the purpose data of described state to be migrated migrate.
Device the most according to claim 6, it is characterised in that described migration units, specifically for:
When described first storage strategy is temperature data store strategy, and when the second storage strategy is cold data store strategy, deletion is deposited
Storage memory block of purpose data described in disk layer node.
Device the most according to claim 6, it is characterised in that described migration units, specifically for:
When described first storage strategy is cold data store strategy, when the second storage strategy is temperature data store strategy, will storage
Described in filing node layer, the memory block of purpose data is copied to disk layer node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610636095.5A CN106294671A (en) | 2016-08-03 | 2016-08-03 | A kind of data migration method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610636095.5A CN106294671A (en) | 2016-08-03 | 2016-08-03 | A kind of data migration method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106294671A true CN106294671A (en) | 2017-01-04 |
Family
ID=57665452
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610636095.5A Pending CN106294671A (en) | 2016-08-03 | 2016-08-03 | A kind of data migration method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106294671A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108776690A (en) * | 2018-06-05 | 2018-11-09 | 上海孚典智能科技有限公司 | The method of HDFS Distribution and Centralization blended data storage systems based on separated layer handling |
| CN109144791A (en) * | 2018-09-30 | 2019-01-04 | 北京金山云网络技术有限公司 | Data conversion storage method, apparatus and data management server |
| CN109614039A (en) * | 2018-11-26 | 2019-04-12 | 新华三大数据技术有限公司 | Data migration method and device |
| WO2021056243A1 (en) * | 2019-09-25 | 2021-04-01 | 华为技术有限公司 | Method and device for managing data in partition table, management node, and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102156738A (en) * | 2011-04-13 | 2011-08-17 | 成都市华为赛门铁克科技有限公司 | Method for processing data blocks, and data block storage equipment and system |
| US20120303585A1 (en) * | 2011-05-23 | 2012-11-29 | Boris Zuckerman | Propagating a snapshot attribute in a distributed file system |
| CN103064914A (en) * | 2012-12-20 | 2013-04-24 | 曙光信息产业(北京)有限公司 | Data processing system and method |
| CN103106152A (en) * | 2012-12-13 | 2013-05-15 | 深圳先进技术研究院 | Data scheduling method based on gradation storage medium |
| CN103914516A (en) * | 2014-02-25 | 2014-07-09 | 深圳市中博科创信息技术有限公司 | Method and system for layer-management of storage system |
| US20140325151A1 (en) * | 2013-04-25 | 2014-10-30 | Xerox Corporation | Method and system for dynamically managing big data in hierarchical cloud storage classes to improve data storing and processing cost efficiency |
| CN104462240A (en) * | 2014-11-18 | 2015-03-25 | 浪潮(北京)电子信息产业有限公司 | Method and system for realizing hierarchical storage and management in cloud storage |
| CN104598540A (en) * | 2014-12-31 | 2015-05-06 | 国家电网公司 | Timing data migration device and using method thereof |
| CN105138476A (en) * | 2015-08-26 | 2015-12-09 | 广东创我科技发展有限公司 | Data storage method and system based on hadoop heterogeneous storage |
-
2016
- 2016-08-03 CN CN201610636095.5A patent/CN106294671A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102156738A (en) * | 2011-04-13 | 2011-08-17 | 成都市华为赛门铁克科技有限公司 | Method for processing data blocks, and data block storage equipment and system |
| US20120303585A1 (en) * | 2011-05-23 | 2012-11-29 | Boris Zuckerman | Propagating a snapshot attribute in a distributed file system |
| CN103106152A (en) * | 2012-12-13 | 2013-05-15 | 深圳先进技术研究院 | Data scheduling method based on gradation storage medium |
| CN103064914A (en) * | 2012-12-20 | 2013-04-24 | 曙光信息产业(北京)有限公司 | Data processing system and method |
| US20140325151A1 (en) * | 2013-04-25 | 2014-10-30 | Xerox Corporation | Method and system for dynamically managing big data in hierarchical cloud storage classes to improve data storing and processing cost efficiency |
| CN103914516A (en) * | 2014-02-25 | 2014-07-09 | 深圳市中博科创信息技术有限公司 | Method and system for layer-management of storage system |
| CN104462240A (en) * | 2014-11-18 | 2015-03-25 | 浪潮(北京)电子信息产业有限公司 | Method and system for realizing hierarchical storage and management in cloud storage |
| CN104598540A (en) * | 2014-12-31 | 2015-05-06 | 国家电网公司 | Timing data migration device and using method thereof |
| CN105138476A (en) * | 2015-08-26 | 2015-12-09 | 广东创我科技发展有限公司 | Data storage method and system based on hadoop heterogeneous storage |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108776690A (en) * | 2018-06-05 | 2018-11-09 | 上海孚典智能科技有限公司 | The method of HDFS Distribution and Centralization blended data storage systems based on separated layer handling |
| CN108776690B (en) * | 2018-06-05 | 2020-07-07 | 上海孚典智能科技有限公司 | Method for HDFS distributed and centralized mixed data storage system based on hierarchical governance |
| CN109144791A (en) * | 2018-09-30 | 2019-01-04 | 北京金山云网络技术有限公司 | Data conversion storage method, apparatus and data management server |
| CN109144791B (en) * | 2018-09-30 | 2020-12-22 | 北京金山云网络技术有限公司 | Data unloading method and device and data management server |
| CN109614039A (en) * | 2018-11-26 | 2019-04-12 | 新华三大数据技术有限公司 | Data migration method and device |
| CN109614039B (en) * | 2018-11-26 | 2022-03-22 | 新华三大数据技术有限公司 | Data migration method and device |
| WO2021056243A1 (en) * | 2019-09-25 | 2021-04-01 | 华为技术有限公司 | Method and device for managing data in partition table, management node, and storage medium |
| US11768856B2 (en) | 2019-09-25 | 2023-09-26 | Huawei Cloud Computing Technologies Co., Ltd. | Method, apparatus, and management node for managing data in partition table, and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3014487B1 (en) | Grouping of objects in a distributed storage system based on journals and placement policies | |
| CN104298681B (en) | A kind of date storage method and device | |
| CN101777016B (en) | Snapshot storage and data recovery method of continuous data protection system | |
| CN105528367B (en) | Storage and near real-time querying method based on open source big data to time sensitive data | |
| CN106294671A (en) | A kind of data migration method and device | |
| CN102055650B (en) | Load balance method and system and management server | |
| CN103176754A (en) | Reading and storing method for massive amounts of small files | |
| CN104657459A (en) | Massive data storage method based on file granularity | |
| CN103473335B (en) | A kind of hot spot data detection method and device | |
| CN110162528A (en) | Magnanimity big data search method and system | |
| CN107798130A (en) | A kind of Snapshot Method of distributed storage | |
| CN105183839A (en) | Hadoop-based storage optimizing method for small file hierachical indexing | |
| CN106201355A (en) | Data processing method and device and storage system | |
| CN105574217B (en) | The method of data synchronization and device of distributed relation database | |
| CN104866521B (en) | Handle the device and method of the inquiry in the database with mixing memory | |
| CN106716409A (en) | Method and system for adaptively building and updating column store database from row store database based on query demands | |
| CN103366015A (en) | OLAP (on-line analytical processing) data storage and query method based on Hadoop | |
| CN106156283A (en) | Isomery Hadoop based on data temperature and joint behavior stores method | |
| CN104869140B (en) | The method of the data storage of multi-cluster system and control multi-cluster system | |
| CN106445677A (en) | Load balancing method and device | |
| CN102253985B (en) | File system data management method and system | |
| CN103176750B (en) | Based on mobile Internet data storage system and the method thereof of staggered time subregion | |
| CN103049574B (en) | Realize key assignments file system and the method for file dynamic copies | |
| CN109767274B (en) | Method and system for carrying out associated storage on massive invoice data | |
| CN105677761A (en) | Data sharding method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 310052 Binjiang District Changhe Road, Zhejiang, China, No. 466, No. Applicant after: NEW H3C TECHNOLOGIES Co.,Ltd. Address before: 310053 Hangzhou science and Technology Industrial Park, high tech Industrial Development Zone, Zhejiang Province, No. six and road, No. 310 Applicant before: HANGZHOU H3C TECHNOLOGIES Co.,Ltd. |
|
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170104 |