CN107145409A - A kind of method of file multichannel backup - Google Patents
A kind of method of file multichannel backup Download PDFInfo
- Publication number
- CN107145409A CN107145409A CN201710397842.9A CN201710397842A CN107145409A CN 107145409 A CN107145409 A CN 107145409A CN 201710397842 A CN201710397842 A CN 201710397842A CN 107145409 A CN107145409 A CN 107145409A
- Authority
- CN
- China
- Prior art keywords
- backup
- file
- scanner
- job
- wafer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
- G06F11/1453—Management of the data involved in backup or backup restore using de-duplication of the data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Storage Device Security (AREA)
- Techniques For Improving Reliability Of Storages (AREA)
Abstract
A kind of method of file multichannel backup, the method worked by multi-threaded parallel makes full use of disk I/O, and whole backup flow is divided into following part by significant increase file backup speed：Scanner, wafer breaker and back-up job person, each part are operated on independent thread, are communicated between all parts by message queue, and backup every time can generate a scanner, a wafer breaker, the back-up job person of one or more.The method for a kind of file multichannel backup that the present invention is provided, with advantages below：Play CPU performance, play magnetic disc i/o performance and with good stability.
Description
Technical field
The present invention relates to the method in microcomputer data processing field, more particularly to a kind of backup of file multichannel.
Background technology
File backup refers to be replicated the data in file system, when occurring disaster or maloperation, Ke Yifang
Just the valid data of recovery system are to ensure the normal operation of system and in time.
An important index is backup rate for file backup, and this is directly connected to backup job to be taken
Time window.And due to needing to take substantial amounts of system resource (such as in backup procedure：Disk I/O, CPU time), this meeting pair
Operation system affects.Therefore shortening this time window just seems necessary, and shortening time window needs
Lift the speed of backup.
For file backup, current existing solution is operated by single thread, and back up it is each
Individual step is serially carried out.
This working method has a disadvantage that：
1. the serialization of backup flow causes mutually to wait between all parts entirely backed up, next step is carried out necessary
Last completion is waited to carry out.
2. single thread, which carries out Backup Data, can not make full use of system resource, cause backup rate and be not up to most
It is excellent.
The content of the invention
The present invention is exactly in view of the deficienciess of the prior art, a kind of method backed up there is provided file multichannel.
To solve the above problems, the technical solution used in the present invention is as follows：
A kind of method of file multichannel backup, the method worked by multi-threaded parallel makes full use of disk I/O, greatly
File backup speed is lifted, whole backup flow is divided into following part：Scanner, wafer breaker and back-up job person,
Each part is operated on independent thread, is communicated between all parts by message queue, and backup every time can give birth to
Into a scanner, a wafer breaker, the back-up job person of one or more；
The function of all parts is as follows：
(1) scanners；
The Backup options scanning file system set according to user obtains the detail file list to be backed up, and by this list
It is sent to wafer breaker；
(2) wafer breakers；
The listed files received from scanner is evenly distributed to each back-up job by wafer breaker by internal algorithm
Person；
(3) back-up job persons；
It is according to the listed files received that file is standby that back-up job person sets up a passage being connected with storage server
Part arrives storage server, and according to Backup options, back-up job person can be to text before the file that will be backed up is sent to storage server
Part is compressed, encrypted, data de-duplication operations.
Compared with prior art, implementation result of the invention is as follows by the present invention：
The method for a kind of file multichannel backup that the present invention is provided, with advantages below：
1. play CPU performance.
By the way that backup flow is divided into each part for operating in separate threads, by slicing files list, backup is repeated
Computational load to the multinuclear such as data deletion and compression gives full play to CPU performance.Simultaneously because design each back-up job meticulously
The logic of person is separate, need not be synchronized each other.
2. play magnetic disc i/o performance.
For RAID storages are by scanning/generation listed files and read file load to multiple disks, disk is given full play to
I/O performances.
3. with good stability.
Have benefited from the system backup set form well-designed and each backup thread Backup Data it is relatively independent, i.e.,
It is destroyed the Backup Data of some passage, the Backup Data of other passages still has method recovery, therefore whole standby system
With good stability.
Embodiment
Illustrate present disclosure below in conjunction with specific embodiments.
A kind of method of file multichannel backup, the method worked by multi-threaded parallel makes full use of disk I/O, greatly
File backup speed is lifted, whole backup flow is divided into following part：Scanner, wafer breaker and back-up job person,
Each part is operated on independent thread, is communicated between all parts by message queue, and backup every time can give birth to
Into a scanner, a wafer breaker, the back-up job person of one or more.
The function of all parts is as follows.
(1) scanners.
The Backup options scanning file system set according to user obtains the detail file list to be backed up, and by this list
It is sent to wafer breaker.
(2) wafer breakers.
The listed files received from scanner is evenly distributed to each back-up job by wafer breaker by internal algorithm
Person.
(3) back-up job persons.
It is according to the listed files received that file is standby that back-up job person sets up a passage being connected with storage server
Part arrives storage server, and according to Backup options, back-up job person can be to text before the file that will be backed up is sent to storage server
Part is compressed, encrypted, data de-duplication operations.
Compared with prior art, implementation result of the invention is as follows by the present invention：
The method for a kind of file multichannel backup that the present invention is provided, with advantages below：
1. play CPU performance.
By the way that backup flow is divided into each part for operating in separate threads, by slicing files list, backup is repeated
Computational load to the multinuclear such as data deletion and compression gives full play to CPU performance.Simultaneously because design each back-up job meticulously
The logic of person is separate, need not be synchronized each other.
2. play magnetic disc i/o performance.
For RAID storages are by scanning/generation listed files and read file load to multiple disks, disk is given full play to
I/O performances.
3. with good stability.
Have benefited from the system backup set form well-designed and each backup thread Backup Data it is relatively independent, i.e.,
It is destroyed the Backup Data of some passage, the Backup Data of other passages still has method recovery, therefore whole standby system
With good stability.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all essences in the present invention
Any modifications, equivalent substitutions and improvements made within refreshing and principle etc., should be included in the scope of the protection.
Claims (1)
1. a kind of method of file multichannel backup, it is characterised in that：The method worked by multi-threaded parallel, makes full use of magnetic
Whole backup flow is divided into following part by disk IO, significant increase file backup speed：Scanner, wafer breaker and standby
Part worker, each part is operated on independent thread, is communicated between all parts by message queue, every time
Backup can generate a scanner, a wafer breaker, the back-up job person of one or more；
The function of all parts is as follows：
(1) scanners；
The Backup options scanning file system set according to user obtains the detail file list to be backed up, and this list is sent
To wafer breaker；
(2) wafer breakers；
The listed files received from scanner is evenly distributed to each back-up job person by wafer breaker by internal algorithm；
(3) back-up job persons；
Back-up job person sets up a passage being connected with storage server and is arrived file backup according to the listed files received
Storage server, according to Backup options, back-up job person can enter to file before the file that will be backed up is sent to storage server
Row compression, encryption, data de-duplication operations.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710397842.9A CN107145409A (en) | 2017-05-31 | 2017-05-31 | A kind of method of file multichannel backup |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710397842.9A CN107145409A (en) | 2017-05-31 | 2017-05-31 | A kind of method of file multichannel backup |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107145409A true CN107145409A (en) | 2017-09-08 |
Family
ID=59779918
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710397842.9A Withdrawn CN107145409A (en) | 2017-05-31 | 2017-05-31 | A kind of method of file multichannel backup |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107145409A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109344001A (en) * | 2018-08-29 | 2019-02-15 | 广东微云科技股份有限公司 | A kind of concurrent backup method and system of virtual machine file |
| CN111880967A (en) * | 2020-07-28 | 2020-11-03 | 平安科技（深圳）有限公司 | File backup method, device, medium and electronic equipment in cloud scene |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105045681A (en) * | 2015-07-10 | 2015-11-11 | 上海爱数软件有限公司 | Oracle multichannel parallel backup and recovery method |
| CN105183585A (en) * | 2015-08-27 | 2015-12-23 | 北京金山安全软件有限公司 | Data backup method and device |
| CN105373453A (en) * | 2015-12-15 | 2016-03-02 | 中国农业银行股份有限公司 | Data backup method and system |
| US20160077925A1 (en) * | 2014-09-16 | 2016-03-17 | Actifio, Inc. | Multi-threaded smart copy |
| US20170083407A1 (en) * | 2012-12-19 | 2017-03-23 | Emc Corporation | Multi stream deduplicated backup of collaboration server data |
-
2017
- 2017-05-31 CN CN201710397842.9A patent/CN107145409A/en not_active Withdrawn
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170083407A1 (en) * | 2012-12-19 | 2017-03-23 | Emc Corporation | Multi stream deduplicated backup of collaboration server data |
| US20160077925A1 (en) * | 2014-09-16 | 2016-03-17 | Actifio, Inc. | Multi-threaded smart copy |
| CN105045681A (en) * | 2015-07-10 | 2015-11-11 | 上海爱数软件有限公司 | Oracle multichannel parallel backup and recovery method |
| CN105183585A (en) * | 2015-08-27 | 2015-12-23 | 北京金山安全软件有限公司 | Data backup method and device |
| CN105373453A (en) * | 2015-12-15 | 2016-03-02 | 中国农业银行股份有限公司 | Data backup method and system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109344001A (en) * | 2018-08-29 | 2019-02-15 | 广东微云科技股份有限公司 | A kind of concurrent backup method and system of virtual machine file |
| CN109344001B (en) * | 2018-08-29 | 2022-01-11 | 广东微云科技股份有限公司 | Concurrent backup method and system for virtual machine files |
| CN111880967A (en) * | 2020-07-28 | 2020-11-03 | 平安科技（深圳）有限公司 | File backup method, device, medium and electronic equipment in cloud scene |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11256558B1 (en) | Prioritized data rebuilding in a dispersed storage network based on consistency requirements | |
| US7567188B1 (en) | Policy based tiered data deduplication strategy | |
| Cidon et al. | Tiered replication: A cost-effective alternative to full cluster geo-replication | |
| US9292371B1 (en) | Systems and methods for preventing failures of nodes in clusters | |
| WO2016036875A1 (en) | Wide spreading data storage architecture | |
| US7496785B2 (en) | Enclosure-based raid parity assist | |
| WO2015045262A1 (en) | Log analysis system, fault cause analysis system, log analysis method, and recording medium which stores program | |
| US6883110B1 (en) | System and method for providing a data backup of a server on client systems in a network | |
| CN109871369B (en) | Database switching method, system, medium and device | |
| CN105120003A (en) | Method for realizing data backup in cloud environment | |
| CN110727508A (en) | Task scheduling system and scheduling method | |
| US8332678B1 (en) | Power save mode operation for continuous data protection | |
| CN107145409A (en) | A kind of method of file multichannel backup | |
| Feng et al. | Magicube: High reliability and low redundancy storage architecture for cloud computing | |
| CN103713854B (en) | Implementing dynamic banding of self encrypting drive | |
| CN101266535A (en) | System for using a virtual tape encryption format | |
| US11023324B2 (en) | Harmonization of failure domains in a distributed system | |
| Zhao et al. | Research and implementation of data storage backup | |
| CN112152798B (en) | Distributed ciphertext sharing key management method and system based on encrypted data deduplication | |
| CN105573860A (en) | STD (Suspend to disk) management method and STD management system used in computer system | |
| Kulkarni et al. | Efficient storage utilization using erasure codes in openstack cloud | |
| CN105138417B (en) | A kind of asynchronous replication method based on zfs send | |
| CN106445729A (en) | Backup virtualization-based method | |
| JP2017142605A (en) | Backup restoration system and restoration method | |
| Chen et al. | Dynamic replication scheduling for cloud datacenters based on workload statistics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20170908 |
|
| WW01 | Invention patent application withdrawn after publication |