CN106648967A - File scanning method and system - Google Patents
File scanning method and system Download PDFInfo
- Publication number
- CN106648967A CN106648967A CN201610899145.9A CN201610899145A CN106648967A CN 106648967 A CN106648967 A CN 106648967A CN 201610899145 A CN201610899145 A CN 201610899145A CN 106648967 A CN106648967 A CN 106648967A
- Authority
- CN
- China
- Prior art keywords
- file
- backed
- scan
- catalogue
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
- G06F11/1451—Management of the data involved in backup or backup restore by selection of backup contents
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a file scanning method and system. The file scanning method comprises the following steps of: obtaining a pre-set backup rule; according to the backup rule, scanning a file needing to be backed up, and furthermore, sending a file list needing to be backed up; and, receiving the file list, and backing up the file needing to be backed up. By means of the file scanning method and system provided by the invention, the pre-set backup rule is obtained; then, according to the backup rule, the file needing to be backed up is scanned; furthermore, the file list needing to be backed up is sent; finally, the file list is received; furthermore, the file needing to be backed up is backed up; and thus, a file in a production system can be transmitted into an offline system correctly and effectively.
Description
Technical field
The present invention relates to computer realm, it particularly relates to a kind of method and system of scanning file.
Background technology
Filing system is to provide in the data duplication in production system to off-line system, and this Backup Data is carried out
The system of management, wherein, production system and off-line system can be dawn parallel memory system, or third party's file system
System.Need a file scan module in the filing system to provide the function of scanning file, so which filing system determines
A little files need backup.But, existing scanning file function is simultaneously unsatisfactory for the requirement of filing system, it is impossible to which realizing will production system
File in system is correctly efficiently transferred in off-line system.
For the problem in correlation technique, effective solution is not yet proposed at present.
The content of the invention
For the problem in correlation technique, the present invention proposes a kind of method and system of scanning file, can be by production system
File in system is correctly efficiently transferred in off-line system.
The technical scheme is that what is be achieved in that：
According to an aspect of the invention, there is provided a kind of method of scanning file.
The method of the scanning file includes：The backup rules that acquisition pre-sets；According to backup rules, scan and be backed up
File, and by the listed files being backed up send；Listed files is received, and the file being backed up is backed up.
According to one embodiment of present invention, according to backup rules, scanning the file being backed up includes：According to backup rule
Then, scan task is generated, and scan task is added in worker thread；According to scan task, worker thread is carried out to catalogue
Scanning, and process the corresponding directory entry of catalogue.
According to one embodiment of present invention, listed files includes：Fileinfo, directory information.
According to one embodiment of present invention, worker thread is scanned to catalogue, and processes the corresponding directory entry of catalogue
Including：In the case where directory entry is file, worker thread sends out the fileinfo in the corresponding directory entry of the file being backed up
Deliver to corresponding control module；In the case where directory entry is catalogue, worker thread is allocated scan task to being scanned into
Journey, and scan task is added to into the head of the queue of scan queue, and by the catalogue in the corresponding directory entry of the file being backed up
Information is sent to corresponding control module.
According to one embodiment of present invention, the listed files being backed up is sent includes：By the listed files being backed up
It is transmitted in batches.
According to one embodiment of present invention, receiving listed files includes：Listed files is received in batches, and in every batch
After receiving listed files success, control module feedback Ack information.
According to a further aspect in the invention, there is provided a kind of system of scanning file.
The system of the scanning file includes：
Acquisition module, for obtaining the backup rules for pre-setting；
Scanning sending module, for according to backup rules, scanning the file being backed up, and the file being backed up is arranged
Table sends；
Backup module, for receiving listed files, and the file being backed up is backed up.
According to one embodiment of present invention, scanning sending module includes：
Add module, for according to backup rules, generating scan task, and scan task is added in worker thread；
Scan module, for according to scan task, worker thread to be scanned to catalogue, and processes the corresponding catalogue of catalogue
.
According to one embodiment of present invention, listed files includes：Fileinfo, directory information.
According to one embodiment of present invention, scan module includes：
File scan module, in the case of being file in directory entry, worker thread is corresponding by the file being backed up
Fileinfo in directory entry is sent to corresponding control module；
Directory scan module, for directory entry be catalogue in the case of, worker thread by scan task be allocated to
Scan procedure, and scan task is added to into the head of the queue of scan queue, and by the corresponding directory entry of the file being backed up
Directory information send to corresponding control module.
The present invention Advantageous Effects be：
The present invention, subsequently according to backup rules, scans the file being backed up by obtaining the backup rules for pre-setting,
And the listed files being backed up is sent, listed files is finally received, and the file being backed up is backed up such that it is able to
File in production system is correctly efficiently transferred in off-line system.
Description of the drawings
In order to be illustrated more clearly that the embodiment of the present invention or technical scheme of the prior art, below will be to institute in embodiment
The accompanying drawing that needs are used is briefly described, it should be apparent that, drawings in the following description are only some enforcements of the present invention
Example, for those of ordinary skill in the art, on the premise of not paying creative work, can be being obtained according to these accompanying drawings
Obtain other accompanying drawings.
Fig. 1 is the flow chart of the method for scanning file according to embodiments of the present invention；
Fig. 2 is the schematic diagram of filing system according to embodiments of the present invention；
Fig. 3 is scanning engine thread structure figure according to embodiments of the present invention；
Fig. 4 is the schematic diagram of the interaction of control module according to embodiments of the present invention and file scan module；
Fig. 5 is the flow chart of the task of file scan module processing and control module according to embodiments of the present invention；
Fig. 6 is the block diagram of the system of scanning file according to embodiments of the present invention.
Specific embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation is described, it is clear that described embodiment is only a part of embodiment of the invention, rather than the embodiment of whole.It is based on
Embodiment in the present invention, the every other embodiment that those of ordinary skill in the art are obtained belongs to present invention protection
Scope.
A kind of embodiments in accordance with the present invention, there is provided method of scanning file.
As illustrated, the method for scanning file according to embodiments of the present invention includes：
Step S101, obtains the backup rules for pre-setting；
Step S103, according to backup rules, scans the file being backed up, and the listed files being backed up is sent；
Step S105, receives listed files, and the file being backed up is backed up.
The present invention, subsequently according to backup rules, scans the file being backed up by obtaining the backup rules for pre-setting,
And the listed files being backed up is sent, listed files is finally received, and the file being backed up is backed up such that it is able to
File in production system is correctly efficiently transferred in off-line system.
According to one embodiment of present invention, according to backup rules, scanning the file being backed up includes：According to backup rule
Then, scan task is generated, and scan task is added in worker thread；According to scan task, worker thread is carried out to catalogue
Scanning, and process the corresponding directory entry of catalogue.
According to one embodiment of present invention, listed files includes：Fileinfo, directory information.
According to one embodiment of present invention, worker thread is scanned to catalogue, and processes the corresponding directory entry of catalogue
Including：In the case where directory entry is file, worker thread sends out the fileinfo in the corresponding directory entry of the file being backed up
Deliver to corresponding control module；In the case where directory entry is catalogue, worker thread is allocated scan task to being scanned into
Journey, and scan task is added to into the head of the queue of scan queue, and by the catalogue in the corresponding directory entry of the file being backed up
Information is sent to corresponding control module.
According to one embodiment of present invention, the listed files being backed up is sent includes：By the listed files being backed up
It is transmitted in batches.
According to one embodiment of present invention, receiving listed files includes：Listed files is received in batches, and in every batch
After receiving listed files success, control module feedback Ack information.
The present invention will be described in detail by specific embodiment below.
As shown in Fig. 2 the filing system includes：OMgs modules, oBKe modules, oScan modules, oScan modules, above-mentioned mould
The major function of block is as follows：
OMgs modules, for being responsible for adding to revise and the operation such as looking into for backup rules, while the increase for being responsible for all kinds of nodes is deleted
Deng operation, meanwhile, backup rules are sent to into oBKe modules；
OBKe modules, the oBKe modules are control module, as the mster-control centre of the filing system, for receiving oMgs
The backup rules that module sends, and the file that oScan modules send is received, while the operation of backup file is decomposed into concurrently
Task dispatching operation；
OScan modules, the module is file scan module, in being arranged on index server cluster oPara, for being responsible for root
The file to be backed up is scanned according to backup rules, and these listed files are sent to into oBke modules；
OBks modules, for being responsible for receiving the file of backup is needed, and is stored in off-line system.
Additionally, as shown in figure 3, when backup policy perform the time then, oBke modules to oScan modules send message,
OScan modules generate a task, be delivered to local pending queues, wherein, the Pending queues be used for preserve it is all still
The write order for processing response is not returned from disk.Meanwhile, control thread is responsible for obtaining scan task from local pending queues,
Distribution task structure, in being delivered to worker thread queue.
Worker thread is responsible for being scanned catalogue (readdir) and processing catalogue (dentry) item, if dentry items
Be file then call the file process function of registration that fileinfo is added the buffering area buf of correspondence oBke (cross-thread is not altogether
Enjoy), then judge whether the buf is full, it is then to send asynchronous message (the message total amount in control transmission)；If dentry items are
Catalogue, the catalogue for calling registration processes function distribution task, is added to scan queue head of the queue.Each worker thread synchronization is only located
Scan task under the corresponding catalogue of one backup rules of reason, while be up to 5 scanning threads work simultaneously, and other sweeps
Retouch task to wait in pending queues.
After the completion of whole scan task is processed, network call back function delivers a task to queue is completed, and completing thread will
As a result oBke modules are notified.
In addition, Fig. 4 shows the interaction of oBke modules and oScan modules, it is specific as follows：
First stage
Step S11, oBke modules need the task of scanning to send to oScan modules by all of；
Step S12, oScan modules receive above-mentioned scan task, and all tasks are added in worker thread queue, and will
The state of above-mentioned all tasks is set to " in scanning "；
Second stage
Step S21, oScan modules send the 1st batch file list to oBke modules, if oScan modules do not receive oBke
The ACK (Acknowledgement confirms character) of module feedback, then be delivered to transmit queue again；
Step S22, oBke modules receive the 1st batch file list, and the 1st batch file list are stored, rear feeding back ACK；
Step S23, repeat step above-mentioned steps S21-S22, until oScan modules by the n-th batch file list send to
OBke modules, meanwhile, mark is incidentally completed in the n-th batch file list
Step S24, oBke modules receive the n-th batch file list, and by the n-th batch file list storage, and by the task
State be arranged to " completing ", and synchronous these files of preparation；
Phase III
OScan resume modules are completed, and the task is deleted from queue, and clear up the internal memory of occupancy.
Additionally, Fig. 5 shows the flow chart of the task of oScan resume module oBke modules, idiographic flow is as follows：
Step 51, oScan module start to process scan tasks；
Step 52, oScan modules check whether available free worker thread, if not having, return to step S51, if having, is held
Row step S51；
Step 53, oScan modules take out a task from local pending queues, select the active line of a free time
Journey, the task is mounted in the thread, meanwhile, a task is taken out from local pending queues, select an idle line
Journey, and be mounted in the idle thread, while arranging the thread state for busy (busy).
Embodiments in accordance with the present invention, additionally provide a kind of system of scanning file.
As shown in fig. 6, the system of scanning file according to embodiments of the present invention includes：
Acquisition module 61, for obtaining the backup rules for pre-setting；
Scanning sending module 62, for according to backup rules, scanning the file being backed up, and by the file being backed up
List sends；
Backup module 63, for receiving listed files, and the file being backed up is backed up.
According to one embodiment of present invention, scanning sending module 62 includes：
Add module (not shown), for according to backup rules, generating scan task, and is added to work by scan task
In thread；
Scan module (not shown), for according to scan task, worker thread to be scanned to catalogue, and processes catalogue pair
The directory entry answered.
According to one embodiment of present invention, listed files includes：Fileinfo, directory information.
According to one embodiment of present invention, scan module includes：
File scan module (not shown), in the case of being file in directory entry, worker thread is by the text being backed up
Fileinfo in the corresponding directory entry of part is sent to corresponding control module；
Directory scan module (not shown), in the case of being catalogue in directory entry, worker thread enters scan task
Row distributes to scan procedure, and scan task is added to into the head of the queue of scan queue, and the file being backed up is corresponding
Directory information in directory entry is sent to corresponding control module.
In sum, by means of the above-mentioned technical proposal of the present invention, by obtaining the backup rules for pre-setting, subsequent root
According to backup rules, the file being backed up is scanned, and the listed files being backed up is sent, finally receive listed files, and
The file being backed up is backed up such that it is able to which the file in production system is correctly efficiently transferred in off-line system.
Meanwhile, after every batch receives listed files success, control module feedback Ack information, it is to avoid file error of transmission or transmission
Situations such as failure.
Presently preferred embodiments of the present invention is the foregoing is only, not to limit the present invention, all essences in the present invention
Within god and principle, any modification, equivalent substitution and improvements made etc. should be included within the scope of the present invention.
Claims (10)
1. a kind of method of scanning file, it is characterised in that include：
The backup rules that acquisition pre-sets；
According to the backup rules, the file being backed up is scanned, and the listed files being backed up is sent；
The listed files is received, and the file being backed up is backed up.
2. method according to claim 1, it is characterised in that according to the backup rules, scan the file being backed up
Including：
According to backup rules, scan task is generated, and scan task is added in worker thread；
According to the scan task, the worker thread is scanned to catalogue, and processes the corresponding directory entry of the catalogue.
3. method according to claim 2, it is characterised in that the listed files includes：Fileinfo, directory information.
4. method according to claim 3, it is characterised in that the worker thread is scanned to catalogue, and processes institute
Stating the corresponding directory entry of catalogue includes：
In the case where the directory entry is file, the worker thread is by the corresponding directory entry of the file being backed up
Fileinfo is sent to corresponding control module；
In the case where the directory entry is catalogue, the worker thread is allocated the scan task to scan procedure,
And the scan task is added to into the head of the queue of scan queue, and by the catalogue in the corresponding directory entry of the file being backed up
Information is sent to corresponding control module.
5. method according to claim 4, it is characterised in that sending the listed files being backed up includes：
The listed files being backed up is transmitted in batches.
6. method according to claim 5, it is characterised in that receiving the listed files includes：File is received in batches
List, and after every batch receives listed files success, the control module feeds back Ack information.
7. a kind of system of scanning file, it is characterised in that include：
Acquisition module, for obtaining the backup rules for pre-setting；
Scanning sending module, for according to the backup rules, scanning the file being backed up, and the file being backed up is arranged
Table sends；
Backup module, for receiving the listed files, and the file being backed up is backed up.
8. system according to claim 1, it is characterised in that scanning sending module includes：
Add module, for according to backup rules, generating scan task, and scan task is added in worker thread；
Scan module, for according to the scan task, the worker thread to be scanned to catalogue, and processes the catalogue pair
The directory entry answered.
9. method according to claim 8, it is characterised in that the listed files includes：Fileinfo, directory information.
10. method according to claim 8, it is characterised in that scan module includes：
File scan module, in the case of being file in the directory entry, the worker thread is by the text being backed up
Fileinfo in the corresponding directory entry of part is sent to corresponding control module；
Directory scan module, in the case of being catalogue in the directory entry, the worker thread enters the scan task
Row distributes to scan procedure, and the scan task is added to into the head of the queue of scan queue, and by the file pair being backed up
Directory information in the directory entry answered is sent to corresponding control module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610899145.9A CN106648967A (en) | 2016-10-14 | 2016-10-14 | File scanning method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610899145.9A CN106648967A (en) | 2016-10-14 | 2016-10-14 | File scanning method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106648967A true CN106648967A (en) | 2017-05-10 |
Family
ID=58856532
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610899145.9A Pending CN106648967A (en) | 2016-10-14 | 2016-10-14 | File scanning method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106648967A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108804253A (en) * | 2017-05-02 | 2018-11-13 | 中国科学院高能物理研究所 | A kind of concurrent job backup method for mass data backup |
| CN109542858A (en) * | 2018-10-11 | 2019-03-29 | 平安科技（深圳）有限公司 | Data archiving method, device based on distributed memory system, computer equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060209362A1 (en) * | 2005-03-15 | 2006-09-21 | Kabushiki Kaisha Toshiba | Scanner system and scanned data storing method |
| CN102495880A (en) * | 2011-12-05 | 2012-06-13 | 北京新媒传信科技有限公司 | Method and device for synchronizing file |
| CN103020186A (en) * | 2012-11-30 | 2013-04-03 | 广东欧珀移动通信有限公司 | File searching method, device and equipment based on embedded device |
| CN103036956A (en) * | 2012-11-30 | 2013-04-10 | 航天恒星科技有限公司 | Filing system and implement method of distributed configured massive data |
| CN105744343A (en) * | 2016-01-26 | 2016-07-06 | 珠海迈科智能科技股份有限公司 | Method for rapidly scanning media files and set top box |
| CN106572280A (en) * | 2016-10-12 | 2017-04-19 | 曙光信息产业(北京)有限公司 | Method and system for scanning file |
-
2016
- 2016-10-14 CN CN201610899145.9A patent/CN106648967A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060209362A1 (en) * | 2005-03-15 | 2006-09-21 | Kabushiki Kaisha Toshiba | Scanner system and scanned data storing method |
| CN102495880A (en) * | 2011-12-05 | 2012-06-13 | 北京新媒传信科技有限公司 | Method and device for synchronizing file |
| CN103020186A (en) * | 2012-11-30 | 2013-04-03 | 广东欧珀移动通信有限公司 | File searching method, device and equipment based on embedded device |
| CN103036956A (en) * | 2012-11-30 | 2013-04-10 | 航天恒星科技有限公司 | Filing system and implement method of distributed configured massive data |
| CN105744343A (en) * | 2016-01-26 | 2016-07-06 | 珠海迈科智能科技股份有限公司 | Method for rapidly scanning media files and set top box |
| CN106572280A (en) * | 2016-10-12 | 2017-04-19 | 曙光信息产业(北京)有限公司 | Method and system for scanning file |
Non-Patent Citations (1)
| Title |
|---|
| 董赞强: "《基于网络编码的数据通信技术研究》", 31 December 2015 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108804253A (en) * | 2017-05-02 | 2018-11-13 | 中国科学院高能物理研究所 | A kind of concurrent job backup method for mass data backup |
| CN108804253B (en) * | 2017-05-02 | 2021-08-06 | 中国科学院高能物理研究所 | Parallel operation backup method for mass data backup |
| CN109542858A (en) * | 2018-10-11 | 2019-03-29 | 平安科技（深圳）有限公司 | Data archiving method, device based on distributed memory system, computer equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8171474B2 (en) | System and method for managing, scheduling, controlling and monitoring execution of jobs by a job scheduler utilizing a publish/subscription interface | |
| CN101227379B (en) | Method and system for implementation of data synchronization | |
| CN105610972A (en) | Clustered task dispatching system | |
| CN106170016A (en) | A kind of method and system processing high concurrent data requests | |
| CN1846202A (en) | Mail server based application record synchronization | |
| CN107105009A (en) | Job scheduling method and device based on Kubernetes system docking workflow engines | |
| CN101383839A (en) | Data distribution system based on data server and implementation method | |
| CN111274052A (en) | Data distribution method, server, and computer-readable storage medium | |
| CN101179553A (en) | Efficient order-preserving delivery method and device for concurrent messages | |
| CN105959420A (en) | Multi-client metadata synchronous update method and device | |
| CN106648967A (en) | File scanning method and system | |
| CN107562803A (en) | Data supply system and method, terminal | |
| CN103248636A (en) | Offline download system and method | |
| CN110570158A (en) | power grid standing book electronic transfer method and system and computer readable storage medium | |
| CN113342764A (en) | Data synchronization method and device among different cloud servers | |
| CN106572280B (en) | Scan the method and system of file | |
| CN112711522B (en) | Cloud testing method and system based on docker and electronic equipment | |
| CN112099930A (en) | Quantum computer cluster distributed queue scheduling method | |
| CN102342077A (en) | Method and system for the real time synthesis of interactions relating to a user | |
| CN111709696A (en) | Method and device for generating mail list based on SOA (service oriented architecture) | |
| CN111475333A (en) | Openstack-based database backup method and device | |
| CN110928911A (en) | System, method and device for processing checking request and computer readable storage medium | |
| US20090164523A1 (en) | System and method for sending messages | |
| US20090106781A1 (en) | Remote call handling methods and systems | |
| CN207264367U (en) | A kind of parallel printing system based on controller |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170510 |