CN104598453B - Data migration method based on data buffering - Google Patents
Data migration method based on data buffering Download PDFInfo
- Publication number
- CN104598453B CN104598453B CN201310527273.7A CN201310527273A CN104598453B CN 104598453 B CN104598453 B CN 104598453B CN 201310527273 A CN201310527273 A CN 201310527273A CN 104598453 B CN104598453 B CN 104598453B
- Authority
- CN
- China
- Prior art keywords
- data
- buffer stopper
- file
- value
- migrated
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/119—Details of migration of file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention proposes the data migration method based on data buffering.Methods described includes：（A1）Data Migration target device performs Data Migration initialization operation；（A2）Create data and read in thread and data write line journey；（A3）The data read in thread and perform data reading operation to read data to be migrated from outside movable storage device, and by the data write-in shared data buffering area to be migrated read out；（A4）The data write-in thread performs data write operation to read out the data to be migrated from the shared data buffering area, and the data to be migrated read out is write in the storage device in the Data Migration target device.Data migration method disclosed in this invention based on data buffering can realize the fast transferring of data and have wide applicability.
Description
Technical field
The present invention relates to data migration method, more particularly, to the data migration method based on data buffering.
Background technology
At present, increasingly extensive and different field the class of business applied with cyber-net becomes increasingly abundant,
By the Data Migration in movable storage device into computer to carry out storage and subsequent treatment (such as by camera, mobile phone, U
Data copy in disk is into computer) become more and more important.
In existing technical scheme, usually using operating system（Such as Windows operating system）Own resource is clear
The file copy function look in device realizes the migration of data.
However, there are the following problems for existing technical scheme：（1）It is difficult to meet there is strict limitation to the time of Data Migration
Application；（2）Due to using the specific resource browser in specific operating system, therefore can only be directed to specific file system
Form（Such as FAT file formats）The file resource of storage carries out migration operation, so as to limit being applicable for Data Migration function
Property（Such as the file resource progress migration operation for the file format do not supported using operating system can not be directed to）.
Accordingly, there exist following demand：The fast transferring that data can be realized and the base with wide applicability are provided
In the data migration method of data buffering.
The content of the invention
In order to solve the problems of above-mentioned prior art, the present invention, which proposes, can realize that the quick of data moves
Move and there is the data migration method based on data buffering of wide applicability.
The purpose of the present invention is achieved through the following technical solutions：
A kind of data migration method based on data buffering, the data migration method based on data buffering include following
Step：
（A1）Data Migration target device performs Data Migration initialization operation；
（A2）Create data and read in thread and data write line journey；
（A3）The data read in thread and perform data reading operation with to be migrated from the reading of outside movable storage device
Data, and by the data write-in shared data buffering area to be migrated read out；
（A4）The data write-in thread performs data write operation to read out institute from the shared data buffering area
Data to be migrated are stated, and the storage that the data to be migrated read out are write in the Data Migration target device fills
In putting.
In scheme disclosed above, it is preferable that the step (A1) further comprises:The shared data is set to delay
The size of the number of the buffer stopper rushed in area and each buffer stopper, and the value of the flag bit of each buffer stopper is arranged to
“Invalid”。
In scheme disclosed above, it is preferable that the step (A1) further comprises:If the outside movement is deposited
Storage equipment does not support the file system format of FAT forms, then the Data Migration target device obtains institute by the interface of correlation
State the file directory content of outside movable storage device and file index is established in internal memory.
In scheme disclosed above, it is preferable that the step（A3）Further comprise：
（B1）The data, which read in thread the value of its flag bit is distributed in the shared data buffering area, is
The buffer stopper of " Invalid " is as currently used buffer stopper and based on the file directory or described of the outside movable storage device
The file index established in internal memory selects a conduct currently file to be migrated, and then from described in file to be migrated
Outside movable storage device reads in the data of the predefined size of the current file to be migrated and the data of reading is written into institute
State in currently used buffer stopper, the value of the flag bit of the currently used buffer stopper is arranged to " Valid head " therewith；
（B2）The data read in thread continuation value of one its flag bit of distribution in the shared data buffering area
The new buffer stopper of " Invalid " is then read in as new currently used buffer stopper from the outside movable storage device
The data of reading are simultaneously written to described new currently used slow by the follow-up data of the predefined size of the current file to be migrated
Rush in block, the value of the flag bit of the new currently used buffer stopper is arranged to " Valid " therewith, wherein, if described work as
The follow-up data deficiency predefined size of preceding file to be migrated, then after reading in the currently actual size of file to be migrated
Continuous data；
（B3）Repeat step（B2）, finished or without it until the data of the current file to be migrated have been read
The value of flag bit is the buffer stopper of " Invalid ", wherein, if the data currently read in are the last of current file to be migrated
Remaining data, then the value of the flag bit of currently used buffer stopper is arranged to " Valid end ", and if its no mark
The value of will position is the buffer stopper of " Invalid ", then goes to step（C1）；
（B4）Repeat step（B1）To continue with next file to be migrated, and if all files to be migrated
It is disposed, then the data read in Thread Termination.
In scheme disclosed above, it is preferable that methods described further comprises：（C1）The shared data is intercepted to delay
The I/O completion statuses of each buffer stopper rushed in area, and the value of the flag bit for the buffer stopper for having completed I/O operation is arranged to
" Invalid ", and then pass to step（B2）.
In scheme disclosed above, it is preferable that the step（A4）Further comprise：
（D1）The data write-in thread is examined in each buffer stopper in the shared data buffering area according to order
Flag bit value, and if the value of the flag bit of the buffer stopper of current check is " Valid " or " Valid end ", then turn
To step（D3）, and if the value of the flag bit of the buffer stopper of current check is that " Valid head ", then go to step（D2）；
（D2）The Data Migration target device calling system function creates new file in its storage device, and will be current
The value of its flag bit checked is " data in Valid head " buffer stopper are write in created new file, and this is delayed
The value for rushing the flag bit of block is arranged to " Waiting "；
（D3）By the value of its flag bit of current check be " Valid " or " data in Valid end " buffer stopper are write
Enter in the corresponding file in the storage device of the Data Migration target device, and the value of the flag bit of the buffer stopper is set
For " Waiting ", and if the value of the flag bit of the buffer stopper of current check is " Valid end ", then by the buffer stopper
In the corresponding file that all writes in the storage devices of the Data Migration target devices of data in after close the phase
The file answered；
（D4）The data write thread repeat step（D1）, until each buffer stopper in the shared data buffering area
The value of flag bit be " Invalid " or " Waiting ", or All Files migration terminates.
In scheme disclosed above, it is preferable that the data in buffer stopper are write into the number using asynchronous I/O pattern
According in the corresponding file in the storage device of migration target device.
Data migration method disclosed in this invention based on data buffering has advantages below：（1）Due to Data Migration
Target device can obtain the file directory content of outside movable storage device by the interface of correlation and text is established in internal memory
Part indexes, therefore the file resource that can be directed to the file format do not supported using operating system carries out migration operation, so as to have
There is wide applicability；（2）Due to having used shared data buffering area in the reading of data and ablation process and using different
Walk I/O patterns to write the data in buffer stopper in the corresponding file in the storage device of Data Migration target device, therefore subtract
The expense of the data copy in data migration process is lacked, so as to realize the faster migration of data.
Brief description of the drawings
With reference to accompanying drawing, technical characteristic of the invention and advantage will be more fully understood by those skilled in the art, wherein：
Fig. 1 is the flow chart of the data migration method according to an embodiment of the invention based on data buffering.
Embodiment
Fig. 1 is the flow chart of the data migration method according to an embodiment of the invention based on data buffering.Such as Fig. 1 institutes
Show, the data migration method disclosed in this invention based on data buffering comprises the following steps：（A1）Data Migration target device
（Such as computer）Perform Data Migration initialization operation；（A2）Create data and read in thread and data write line journey；（A3）Institute
State data and read in thread execution data reading operation to read data to be migrated from outside movable storage device, and will read out
Data write-in shared data buffering area to be migrated in；（A4）Data write-in thread performs data write operation with from institute
State and the data to be migrated are read out in shared data buffering area, and by described in the data write-in to be migrated read out
Storage device in Data Migration target device（Such as disk）In.
Preferably, in the data migration method disclosed in this invention based on data buffering, the step (A1) enters one
Step includes:The number of buffer stopper in the shared data buffering area is set（Such as 128）And size (the example of each buffer stopper
Such as 128KB), and the value of the flag bit of each buffer stopper is arranged to " Invalid ".
Preferably, in the data migration method disclosed in this invention based on data buffering, the step (A1) enters one
Step includes:If the outside movable storage device does not support the file system format of FAT forms, the Data Migration target
Equipment obtains the file directory content of the outside movable storage device by the interface of correlation and file rope is established in internal memory
Draw.
Preferably, in the data migration method disclosed in this invention based on data buffering, the step（A3）Enter one
Step includes：（B1）The data, which read in thread the value of its flag bit is distributed in the shared data buffering area, is
The buffer stopper of " Invalid " is as currently used buffer stopper and based on the file directory of the outside movable storage device（Institute
In the case of stating the file system format that outside movable storage device supports FAT forms）Or the file established in internal memory
Index（In the case where the outside movable storage device does not support the file system format of FAT forms）Select file to be migrated
In the current file to be migrated of a conduct, and then read in the current text to be migrated from the outside movable storage device
The predefined size of part（Such as 128KB）Data and the data of reading are written in the currently used buffer stopper, therewith will
The value of the flag bit of the currently used buffer stopper is arranged to " Valid head "；（B2）The data are read in thread and continued in institute
The value for stating one its flag bit of distribution in shared data buffering area is the new buffer stopper of " Invalid " as newly currently used
Buffer stopper, and the predefined size of the current file to be migrated is then read in from the outside movable storage device（Such as
128KB）Follow-up data and the data of reading are written in the new currently used buffer stopper, described new is worked as therewith
The value of the preceding flag bit using buffer stopper is arranged to " Valid ", wherein, if the follow-up data of the current file to be migrated is not
The foot predefined size（Such as 128KB）, then the follow-up data of the actual size of the current file to be migrated is read in（It is i.e. all
The remaining data of current file to be migrated）；（B3）Repeat step（B2）, until the data of the current file to be migrated have been read
Take finish or without its flag bit value be " Invalid " buffer stopper, wherein, if the data currently read in be ought
The last remaining data of preceding file to be migrated, then be arranged to the value of the flag bit of currently used buffer stopper " Valid end ",
And if the value without its flag bit is the buffer stopper of " Invalid ", then go to step（C1）；（B4）Repeat step（B1）
To continue with next file to be migrated, and finished if all files to be migrated are processed, then the data are read in
Thread Termination.
Preferably, the data migration method disclosed in this invention based on data buffering further comprises：（C1）Intercept institute
State the I/O of each buffer stopper in shared data buffering area（Input/output）Completion status, and I/O operation will have been completed（I.e.
The corresponding file that its read-out data cached has all been write in the storage device of the Data Migration target device
In）The value of flag bit of buffer stopper be arranged to " Invalid ", and then pass to step（B2）.
Preferably, in the data migration method disclosed in this invention based on data buffering, the step（A4）Enter one
Step includes：（D1）The data write-in thread is examined in each buffer stopper in the shared data buffering area according to order
The value of flag bit, and if the value of the flag bit of the buffer stopper of current check is " Valid " or " Valid end ", then go to
Step（D3）, and if the value of the flag bit of the buffer stopper of current check is that " Valid head ", then go to step（D2）；
（D2）The Data Migration target device calling system function（Such as the system function of Windows operating systems
CreateFile）In its storage device（Such as disk）Middle establishment new file（Exemplarily, the new file with it is current to be migrated
File is of the same name）, and by the value of its flag bit of current check by " the data write-in in Valid head " buffer stopper is created
New file in, and the value of the flag bit of the buffer stopper is arranged to " Waiting "；（D3）By its flag bit of current check
It is worth for " Valid " or " data in Valid end " buffer stopper writes the storage device of the Data Migration target device（Example
Such as disk）In corresponding file in, and the value of the flag bit of the buffer stopper is arranged to " Waiting ", and if current
The value of the flag bit of the buffer stopper of inspection is " Valid end ", then the data in the buffer stopper all to be write into the data
Corresponding file is closed after migrating in the corresponding file in the storage device of target device；（D4）The data write-in
Thread repeat step（D1）, up to the value of the flag bit of each buffer stopper in the shared data buffering area is
" Invalid " or " Waiting ", or All Files migration terminate.
Preferably, will using asynchronous I/O pattern in the data migration method disclosed in this invention based on data buffering
Data in buffer stopper write the storage device of the Data Migration target device（Such as disk）In corresponding file in.
Therefore the data migration method disclosed in this invention based on data buffering has following advantages：（1）Due to
Data Migration target device can be obtained the file directory content of outside movable storage device by the interface of correlation and in internal memory
In establish file index, therefore the file resource that can be directed to the file format do not supported using operating system carry out migration behaviour
Make, so as to have wide applicability；（2）Due to having used shared data buffering area simultaneously in the reading of data and ablation process
And the corresponding file for being write the data in buffer stopper in the storage device of Data Migration target device using asynchronous I/O pattern
In, therefore reduce the expense of the data copy in data migration process, so as to realize the faster migration of data.
Although the present invention is described by above-mentioned preferred embodiment, its way of realization is not limited to
Above-mentioned embodiment.It should be realized that：In the case where not departing from spirit and scope of the present invention, those skilled in the art can be with
Different change and modifications is made to the present invention.
Claims (5)
1. a kind of data migration method based on data buffering, the data migration method based on data buffering includes following step
Suddenly：
（A1）Data Migration target device performs Data Migration initialization operation；
（A2）Create data and read in thread and data write line journey；
（A3）The data read in thread and perform data reading operation to read data to be migrated from outside movable storage device,
And by the data write-in shared data buffering area to be migrated read out；
（A4）The data write-in thread performs data write operation to read out described treat from the shared data buffering area
The data of migration, and the storage device that the data to be migrated read out are write in the Data Migration target device
In,
The step (A1) further comprises：If the outside movable storage device does not support the file system lattice of FAT forms
The file directory content of formula, then the interface acquisition outside movable storage device that the Data Migration target device passes through correlation
And file index is established in internal memory.
2. the data migration method according to claim 1 based on data buffering, it is characterised in that the step (A1) is entered
One step includes:The number of buffer stopper in the shared data buffering area and the size of each buffer stopper are set, and will be each
The value of the flag bit of buffer stopper is arranged to " Invalid ".
3. the data migration method according to claim 1 based on data buffering, it is characterised in that the step（A3）Enter
One step includes：
（B1）The value that the data reading thread distributes its flag bit in the shared data buffering area is " Invalid "
Buffer stopper as currently used buffer stopper and based on the file directory or described in internal memory of the outside movable storage device
The file index of foundation selects a conduct currently file to be migrated, and then from described outside mobile in file to be migrated
Storage device, which reads in the data of the predefined size of the current file to be migrated and is written to the data of reading, described currently to be made
With in buffer stopper, the value of the flag bit of the currently used buffer stopper is arranged to " Valid head " therewith；
（B2）The data read in thread continuation value of one its flag bit of distribution in the shared data buffering area
The new buffer stopper of " Invalid " is then read in as new currently used buffer stopper from the outside movable storage device
The data of reading are simultaneously written to described new currently used slow by the follow-up data of the predefined size of the current file to be migrated
Rush in block, the value of the flag bit of the new currently used buffer stopper is arranged to " Valid " therewith, wherein, if described work as
The follow-up data deficiency predefined size of preceding file to be migrated, then after reading in the currently actual size of file to be migrated
Continuous data；
（B3）Repeat step（B2）, finished or without its mark until the data of the current file to be migrated have been read
The value of position is the buffer stopper of " Invalid ", wherein, if the data currently read in are the last residues of current file to be migrated
Data, then the value of the flag bit of currently used buffer stopper is arranged to " Valid end ", and if without its flag bit
Value be " Invalid " buffer stopper, then go to step（C1）Intercept each buffer stopper in the shared data buffering area
I/O completion statuses, and the value of the flag bit for the buffer stopper for having completed I/O operation is arranged to " Invalid ", and then turn
To step（B2）；
（B4）Repeat step（B1）To continue with next file to be migrated, and if all files to be migrated are processed
Finish, then the data read in Thread Termination.
4. the data migration method according to claim 3 based on data buffering, it is characterised in that the step（A4）Enter
One step includes：
（D1）The data write-in thread is examined in the mark of each buffer stopper in the shared data buffering area according to order
The value of will position, and if the value of the flag bit of the buffer stopper of current check for " Valid " or " Valid end ", then goes to step
Suddenly（D3）, and if the value of the flag bit of the buffer stopper of current check is that " Valid head ", then go to step（D2）；
（D2）The Data Migration target device calling system function creates new file in its storage device, and by current check
Its flag bit value for " data in Valid head " buffer stopper are write in created new file, and by the buffer stopper
The value of flag bit be arranged to " Waiting "；
（D3）By the value of its flag bit of current check is " Valid " or " data in Valid end " buffer stopper write institute
State in the corresponding file in the storage device of Data Migration target device, and the value of the flag bit of the buffer stopper is arranged to
" Waiting ", and if the value of the flag bit of the buffer stopper of current check is " Valid end ", then by the buffer stopper
The corresponding file that all writes in the storage devices of the Data Migration target devices of data in after close it is described corresponding
File；
（D4）The data write thread repeat step（D1）, until the mark of each buffer stopper in the shared data buffering area
The value of will position is " Invalid " or " Waiting ", or All Files migration terminates.
5. the data migration method according to claim 4 based on data buffering, it is characterised in that use asynchronous I/O mould
Formula writes the data in buffer stopper in the corresponding file in the storage device of the Data Migration target device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310527273.7A CN104598453B (en) | 2013-10-31 | 2013-10-31 | Data migration method based on data buffering |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310527273.7A CN104598453B (en) | 2013-10-31 | 2013-10-31 | Data migration method based on data buffering |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104598453A CN104598453A (en) | 2015-05-06 |
| CN104598453B true CN104598453B (en) | 2018-02-27 |
Family
ID=53124252
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310527273.7A Active CN104598453B (en) | 2013-10-31 | 2013-10-31 | Data migration method based on data buffering |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104598453B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105429783A (en) * | 2015-10-30 | 2016-03-23 | 上海帝联信息科技股份有限公司 | Edge node server, log file processing method and log file processing system |
| CN105718507A (en) * | 2016-01-06 | 2016-06-29 | 杭州数梦工场科技有限公司 | Data migration method and device |
| CN109388342A (en) * | 2018-08-31 | 2019-02-26 | 深圳市众鸿科技股份有限公司 | The multifile system recognition methods of movable storage device and system, car-mounted terminal |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101188544A (en) * | 2007-12-04 | 2008-05-28 | 浙江大学 | File transfer method for distributed file server based on buffer |
| CN101441655A (en) * | 2008-12-24 | 2009-05-27 | 深圳市迅雷网络技术有限公司 | Method and apparatus for reading data |
| CN102999537A (en) * | 2011-09-19 | 2013-03-27 | 阿里巴巴集团控股有限公司 | System and method for data migration |
| CN103309820A (en) * | 2013-06-28 | 2013-09-18 | 曙光信息产业(北京)有限公司 | Implementation method for disk array cache |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7099875B2 (en) * | 1999-06-29 | 2006-08-29 | Emc Corporation | Method and apparatus for making independent data copies in a data processing system |
-
2013
- 2013-10-31 CN CN201310527273.7A patent/CN104598453B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101188544A (en) * | 2007-12-04 | 2008-05-28 | 浙江大学 | File transfer method for distributed file server based on buffer |
| CN101441655A (en) * | 2008-12-24 | 2009-05-27 | 深圳市迅雷网络技术有限公司 | Method and apparatus for reading data |
| CN102999537A (en) * | 2011-09-19 | 2013-03-27 | 阿里巴巴集团控股有限公司 | System and method for data migration |
| CN103309820A (en) * | 2013-06-28 | 2013-09-18 | 曙光信息产业(北京)有限公司 | Implementation method for disk array cache |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104598453A (en) | 2015-05-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106790599B (en) | A kind of symbiosis virtual machine communication method based on multicore without lock buffer circle | |
| US7739475B2 (en) | System and method for updating dirty data of designated raw device | |
| CN108228647B (en) | Method and apparatus for data copying | |
| CN108959117B (en) | H2D write operation acceleration method and device, computer equipment and storage medium | |
| CN103793475B (en) | A kind of method of Distributed File System Data migration | |
| CN108469979A (en) | A kind of method for page jump, device, server and storage medium | |
| JP2017117204A (en) | Processor, reconfigurable circuit control method, and program | |
| US10430040B2 (en) | Method and an apparatus for providing a multitasking view | |
| CN104598453B (en) | Data migration method based on data buffering | |
| CN108108324A (en) | A kind of PCIE extended methods and device | |
| JP2015087837A (en) | Computer system and method for controlling data arrangement | |
| CN103324532A (en) | Dynamic migration method and system of virtual machine | |
| WO2016082227A1 (en) | Data storage method and apparatus | |
| CN105740330A (en) | Data paging display method and apparatus | |
| CN104985939A (en) | Laser marking machine control method and laser marking machine | |
| CN104461929B (en) | Distributed data cache method based on blocker | |
| KR102166185B1 (en) | Data processing apparatus for pipeline execution acceleration and method thereof | |
| CN106383667B (en) | The method and apparatus that LUN is migrated in service operation | |
| CN107391272A (en) | The method and terminal of multi-process data interaction | |
| CN205986931U (en) | Switch based on NVMeSSD | |
| CN106446114A (en) | Data migration method and device | |
| CN102708075A (en) | Secure digital (SD) memory card hardware control device and control method | |
| CN103139294A (en) | Method and system for local resource mapping based on application virtualization | |
| US20140258347A1 (en) | Grouping files for optimized file operations | |
| CN111309424A (en) | Page restoration method and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |