CN106648967A - File scanning method and system - Google Patents

File scanning method and system Download PDF

Info

Publication number
CN106648967A
CN106648967A CN201610899145.9A CN201610899145A CN106648967A CN 106648967 A CN106648967 A CN 106648967A CN 201610899145 A CN201610899145 A CN 201610899145A CN 106648967 A CN106648967 A CN 106648967A
Authority
CN
China
Prior art keywords
file
backed
scan
catalogue
module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201610899145.9A
Other languages
Chinese (zh)
Inventor
龚超
罗广俊
张新凤
王慧
李国君
罗广骏
王灯殿
石静
张记华
宋频轶
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Dawning Information Industry Beijing Co Ltd
Original Assignee
Dawning Information Industry Beijing Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dawning Information Industry Beijing Co Ltd filed Critical Dawning Information Industry Beijing Co Ltd
Priority to CN201610899145.9A priority Critical patent/CN106648967A/en
Publication of CN106648967A publication Critical patent/CN106648967A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14Error detection or correction of the data by redundancy in operation
    • G06F11/1402Saving, restoring, recovering or retrying
    • G06F11/1446Point-in-time backing up or restoration of persistent data
    • G06F11/1448Management of the data involved in backup or backup restore
    • G06F11/1451Management of the data involved in backup or backup restore by selection of backup contents

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention provides a file scanning method and system. The file scanning method comprises the following steps of: obtaining a pre-set backup rule; according to the backup rule, scanning a file needing to be backed up, and furthermore, sending a file list needing to be backed up; and, receiving the file list, and backing up the file needing to be backed up. By means of the file scanning method and system provided by the invention, the pre-set backup rule is obtained; then, according to the backup rule, the file needing to be backed up is scanned; furthermore, the file list needing to be backed up is sent; finally, the file list is received; furthermore, the file needing to be backed up is backed up; and thus, a file in a production system can be transmitted into an offline system correctly and effectively.

Description

A kind of method and system of scanning file
Technical field
The present invention relates to computer realm, it particularly relates to a kind of method and system of scanning file.
Background technology
Filing system is to provide in the data duplication in production system to off-line system, and this Backup Data is carried out The system of management, wherein, production system and off-line system can be dawn parallel memory system, or third party's file system System.Need a file scan module in the filing system to provide the function of scanning file, so which filing system determines A little files need backup.But, existing scanning file function is simultaneously unsatisfactory for the requirement of filing system, it is impossible to which realizing will production system File in system is correctly efficiently transferred in off-line system.
For the problem in correlation technique, effective solution is not yet proposed at present.
The content of the invention
For the problem in correlation technique, the present invention proposes a kind of method and system of scanning file, can be by production system File in system is correctly efficiently transferred in off-line system.
The technical scheme is that what is be achieved in that:
According to an aspect of the invention, there is provided a kind of method of scanning file.
The method of the scanning file includes:The backup rules that acquisition pre-sets;According to backup rules, scan and be backed up File, and by the listed files being backed up send;Listed files is received, and the file being backed up is backed up.
According to one embodiment of present invention, according to backup rules, scanning the file being backed up includes:According to backup rule Then, scan task is generated, and scan task is added in worker thread;According to scan task, worker thread is carried out to catalogue Scanning, and process the corresponding directory entry of catalogue.
According to one embodiment of present invention, listed files includes:Fileinfo, directory information.
According to one embodiment of present invention, worker thread is scanned to catalogue, and processes the corresponding directory entry of catalogue Including:In the case where directory entry is file, worker thread sends out the fileinfo in the corresponding directory entry of the file being backed up Deliver to corresponding control module;In the case where directory entry is catalogue, worker thread is allocated scan task to being scanned into Journey, and scan task is added to into the head of the queue of scan queue, and by the catalogue in the corresponding directory entry of the file being backed up Information is sent to corresponding control module.
According to one embodiment of present invention, the listed files being backed up is sent includes:By the listed files being backed up It is transmitted in batches.
According to one embodiment of present invention, receiving listed files includes:Listed files is received in batches, and in every batch After receiving listed files success, control module feedback Ack information.
According to a further aspect in the invention, there is provided a kind of system of scanning file.
The system of the scanning file includes:
Acquisition module, for obtaining the backup rules for pre-setting;
Scanning sending module, for according to backup rules, scanning the file being backed up, and the file being backed up is arranged Table sends;
Backup module, for receiving listed files, and the file being backed up is backed up.
According to one embodiment of present invention, scanning sending module includes:
Add module, for according to backup rules, generating scan task, and scan task is added in worker thread;
Scan module, for according to scan task, worker thread to be scanned to catalogue, and processes the corresponding catalogue of catalogue .
According to one embodiment of present invention, listed files includes:Fileinfo, directory information.
According to one embodiment of present invention, scan module includes:
File scan module, in the case of being file in directory entry, worker thread is corresponding by the file being backed up Fileinfo in directory entry is sent to corresponding control module;
Directory scan module, for directory entry be catalogue in the case of, worker thread by scan task be allocated to Scan procedure, and scan task is added to into the head of the queue of scan queue, and by the corresponding directory entry of the file being backed up Directory information send to corresponding control module.
The present invention Advantageous Effects be:
The present invention, subsequently according to backup rules, scans the file being backed up by obtaining the backup rules for pre-setting, And the listed files being backed up is sent, listed files is finally received, and the file being backed up is backed up such that it is able to File in production system is correctly efficiently transferred in off-line system.
Description of the drawings
In order to be illustrated more clearly that the embodiment of the present invention or technical scheme of the prior art, below will be to institute in embodiment The accompanying drawing that needs are used is briefly described, it should be apparent that, drawings in the following description are only some enforcements of the present invention Example, for those of ordinary skill in the art, on the premise of not paying creative work, can be being obtained according to these accompanying drawings Obtain other accompanying drawings.
Fig. 1 is the flow chart of the method for scanning file according to embodiments of the present invention;
Fig. 2 is the schematic diagram of filing system according to embodiments of the present invention;
Fig. 3 is scanning engine thread structure figure according to embodiments of the present invention;
Fig. 4 is the schematic diagram of the interaction of control module according to embodiments of the present invention and file scan module;
Fig. 5 is the flow chart of the task of file scan module processing and control module according to embodiments of the present invention;
Fig. 6 is the block diagram of the system of scanning file according to embodiments of the present invention.
Specific embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete Site preparation is described, it is clear that described embodiment is only a part of embodiment of the invention, rather than the embodiment of whole.It is based on Embodiment in the present invention, the every other embodiment that those of ordinary skill in the art are obtained belongs to present invention protection Scope.
A kind of embodiments in accordance with the present invention, there is provided method of scanning file.
As illustrated, the method for scanning file according to embodiments of the present invention includes:
Step S101, obtains the backup rules for pre-setting;
Step S103, according to backup rules, scans the file being backed up, and the listed files being backed up is sent;
Step S105, receives listed files, and the file being backed up is backed up.
The present invention, subsequently according to backup rules, scans the file being backed up by obtaining the backup rules for pre-setting, And the listed files being backed up is sent, listed files is finally received, and the file being backed up is backed up such that it is able to File in production system is correctly efficiently transferred in off-line system.
According to one embodiment of present invention, according to backup rules, scanning the file being backed up includes:According to backup rule Then, scan task is generated, and scan task is added in worker thread;According to scan task, worker thread is carried out to catalogue Scanning, and process the corresponding directory entry of catalogue.
According to one embodiment of present invention, listed files includes:Fileinfo, directory information.
According to one embodiment of present invention, worker thread is scanned to catalogue, and processes the corresponding directory entry of catalogue Including:In the case where directory entry is file, worker thread sends out the fileinfo in the corresponding directory entry of the file being backed up Deliver to corresponding control module;In the case where directory entry is catalogue, worker thread is allocated scan task to being scanned into Journey, and scan task is added to into the head of the queue of scan queue, and by the catalogue in the corresponding directory entry of the file being backed up Information is sent to corresponding control module.
According to one embodiment of present invention, the listed files being backed up is sent includes:By the listed files being backed up It is transmitted in batches.
According to one embodiment of present invention, receiving listed files includes:Listed files is received in batches, and in every batch After receiving listed files success, control module feedback Ack information.
The present invention will be described in detail by specific embodiment below.
As shown in Fig. 2 the filing system includes:OMgs modules, oBKe modules, oScan modules, oScan modules, above-mentioned mould The major function of block is as follows:
OMgs modules, for being responsible for adding to revise and the operation such as looking into for backup rules, while the increase for being responsible for all kinds of nodes is deleted Deng operation, meanwhile, backup rules are sent to into oBKe modules;
OBKe modules, the oBKe modules are control module, as the mster-control centre of the filing system, for receiving oMgs The backup rules that module sends, and the file that oScan modules send is received, while the operation of backup file is decomposed into concurrently Task dispatching operation;
OScan modules, the module is file scan module, in being arranged on index server cluster oPara, for being responsible for root The file to be backed up is scanned according to backup rules, and these listed files are sent to into oBke modules;
OBks modules, for being responsible for receiving the file of backup is needed, and is stored in off-line system.
Additionally, as shown in figure 3, when backup policy perform the time then, oBke modules to oScan modules send message, OScan modules generate a task, be delivered to local pending queues, wherein, the Pending queues be used for preserve it is all still The write order for processing response is not returned from disk.Meanwhile, control thread is responsible for obtaining scan task from local pending queues, Distribution task structure, in being delivered to worker thread queue.
Worker thread is responsible for being scanned catalogue (readdir) and processing catalogue (dentry) item, if dentry items Be file then call the file process function of registration that fileinfo is added the buffering area buf of correspondence oBke (cross-thread is not altogether Enjoy), then judge whether the buf is full, it is then to send asynchronous message (the message total amount in control transmission);If dentry items are Catalogue, the catalogue for calling registration processes function distribution task, is added to scan queue head of the queue.Each worker thread synchronization is only located Scan task under the corresponding catalogue of one backup rules of reason, while be up to 5 scanning threads work simultaneously, and other sweeps Retouch task to wait in pending queues.
After the completion of whole scan task is processed, network call back function delivers a task to queue is completed, and completing thread will As a result oBke modules are notified.
In addition, Fig. 4 shows the interaction of oBke modules and oScan modules, it is specific as follows:
First stage
Step S11, oBke modules need the task of scanning to send to oScan modules by all of;
Step S12, oScan modules receive above-mentioned scan task, and all tasks are added in worker thread queue, and will The state of above-mentioned all tasks is set to " in scanning ";
Second stage
Step S21, oScan modules send the 1st batch file list to oBke modules, if oScan modules do not receive oBke The ACK (Acknowledgement confirms character) of module feedback, then be delivered to transmit queue again;
Step S22, oBke modules receive the 1st batch file list, and the 1st batch file list are stored, rear feeding back ACK;
Step S23, repeat step above-mentioned steps S21-S22, until oScan modules by the n-th batch file list send to OBke modules, meanwhile, mark is incidentally completed in the n-th batch file list
Step S24, oBke modules receive the n-th batch file list, and by the n-th batch file list storage, and by the task State be arranged to " completing ", and synchronous these files of preparation;
Phase III
OScan resume modules are completed, and the task is deleted from queue, and clear up the internal memory of occupancy.
Additionally, Fig. 5 shows the flow chart of the task of oScan resume module oBke modules, idiographic flow is as follows:
Step 51, oScan module start to process scan tasks;
Step 52, oScan modules check whether available free worker thread, if not having, return to step S51, if having, is held Row step S51;
Step 53, oScan modules take out a task from local pending queues, select the active line of a free time Journey, the task is mounted in the thread, meanwhile, a task is taken out from local pending queues, select an idle line Journey, and be mounted in the idle thread, while arranging the thread state for busy (busy).
Embodiments in accordance with the present invention, additionally provide a kind of system of scanning file.
As shown in fig. 6, the system of scanning file according to embodiments of the present invention includes:
Acquisition module 61, for obtaining the backup rules for pre-setting;
Scanning sending module 62, for according to backup rules, scanning the file being backed up, and by the file being backed up List sends;
Backup module 63, for receiving listed files, and the file being backed up is backed up.
According to one embodiment of present invention, scanning sending module 62 includes:
Add module (not shown), for according to backup rules, generating scan task, and is added to work by scan task In thread;
Scan module (not shown), for according to scan task, worker thread to be scanned to catalogue, and processes catalogue pair The directory entry answered.
According to one embodiment of present invention, listed files includes:Fileinfo, directory information.
According to one embodiment of present invention, scan module includes:
File scan module (not shown), in the case of being file in directory entry, worker thread is by the text being backed up Fileinfo in the corresponding directory entry of part is sent to corresponding control module;
Directory scan module (not shown), in the case of being catalogue in directory entry, worker thread enters scan task Row distributes to scan procedure, and scan task is added to into the head of the queue of scan queue, and the file being backed up is corresponding Directory information in directory entry is sent to corresponding control module.
In sum, by means of the above-mentioned technical proposal of the present invention, by obtaining the backup rules for pre-setting, subsequent root According to backup rules, the file being backed up is scanned, and the listed files being backed up is sent, finally receive listed files, and The file being backed up is backed up such that it is able to which the file in production system is correctly efficiently transferred in off-line system. Meanwhile, after every batch receives listed files success, control module feedback Ack information, it is to avoid file error of transmission or transmission Situations such as failure.
Presently preferred embodiments of the present invention is the foregoing is only, not to limit the present invention, all essences in the present invention Within god and principle, any modification, equivalent substitution and improvements made etc. should be included within the scope of the present invention.

Claims (10)

1. a kind of method of scanning file, it is characterised in that include:
The backup rules that acquisition pre-sets;
According to the backup rules, the file being backed up is scanned, and the listed files being backed up is sent;
The listed files is received, and the file being backed up is backed up.
2. method according to claim 1, it is characterised in that according to the backup rules, scan the file being backed up Including:
According to backup rules, scan task is generated, and scan task is added in worker thread;
According to the scan task, the worker thread is scanned to catalogue, and processes the corresponding directory entry of the catalogue.
3. method according to claim 2, it is characterised in that the listed files includes:Fileinfo, directory information.
4. method according to claim 3, it is characterised in that the worker thread is scanned to catalogue, and processes institute Stating the corresponding directory entry of catalogue includes:
In the case where the directory entry is file, the worker thread is by the corresponding directory entry of the file being backed up Fileinfo is sent to corresponding control module;
In the case where the directory entry is catalogue, the worker thread is allocated the scan task to scan procedure, And the scan task is added to into the head of the queue of scan queue, and by the catalogue in the corresponding directory entry of the file being backed up Information is sent to corresponding control module.
5. method according to claim 4, it is characterised in that sending the listed files being backed up includes:
The listed files being backed up is transmitted in batches.
6. method according to claim 5, it is characterised in that receiving the listed files includes:File is received in batches List, and after every batch receives listed files success, the control module feeds back Ack information.
7. a kind of system of scanning file, it is characterised in that include:
Acquisition module, for obtaining the backup rules for pre-setting;
Scanning sending module, for according to the backup rules, scanning the file being backed up, and the file being backed up is arranged Table sends;
Backup module, for receiving the listed files, and the file being backed up is backed up.
8. system according to claim 1, it is characterised in that scanning sending module includes:
Add module, for according to backup rules, generating scan task, and scan task is added in worker thread;
Scan module, for according to the scan task, the worker thread to be scanned to catalogue, and processes the catalogue pair The directory entry answered.
9. method according to claim 8, it is characterised in that the listed files includes:Fileinfo, directory information.
10. method according to claim 8, it is characterised in that scan module includes:
File scan module, in the case of being file in the directory entry, the worker thread is by the text being backed up Fileinfo in the corresponding directory entry of part is sent to corresponding control module;
Directory scan module, in the case of being catalogue in the directory entry, the worker thread enters the scan task Row distributes to scan procedure, and the scan task is added to into the head of the queue of scan queue, and by the file pair being backed up Directory information in the directory entry answered is sent to corresponding control module.
CN201610899145.9A 2016-10-14 2016-10-14 File scanning method and system Pending CN106648967A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610899145.9A CN106648967A (en) 2016-10-14 2016-10-14 File scanning method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610899145.9A CN106648967A (en) 2016-10-14 2016-10-14 File scanning method and system

Publications (1)

Publication Number Publication Date
CN106648967A true CN106648967A (en) 2017-05-10

Family

ID=58856532

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610899145.9A Pending CN106648967A (en) 2016-10-14 2016-10-14 File scanning method and system

Country Status (1)

Country Link
CN (1) CN106648967A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108804253A (en) * 2017-05-02 2018-11-13 中国科学院高能物理研究所 A kind of concurrent job backup method for mass data backup
CN109542858A (en) * 2018-10-11 2019-03-29 平安科技(深圳)有限公司 Data archiving method, device based on distributed memory system, computer equipment

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060209362A1 (en) * 2005-03-15 2006-09-21 Kabushiki Kaisha Toshiba Scanner system and scanned data storing method
CN102495880A (en) * 2011-12-05 2012-06-13 北京新媒传信科技有限公司 Method and device for synchronizing file
CN103020186A (en) * 2012-11-30 2013-04-03 广东欧珀移动通信有限公司 File searching method, device and equipment based on embedded device
CN103036956A (en) * 2012-11-30 2013-04-10 航天恒星科技有限公司 Filing system and implement method of distributed configured massive data
CN105744343A (en) * 2016-01-26 2016-07-06 珠海迈科智能科技股份有限公司 Method for rapidly scanning media files and set top box
CN106572280A (en) * 2016-10-12 2017-04-19 曙光信息产业(北京)有限公司 Method and system for scanning file

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060209362A1 (en) * 2005-03-15 2006-09-21 Kabushiki Kaisha Toshiba Scanner system and scanned data storing method
CN102495880A (en) * 2011-12-05 2012-06-13 北京新媒传信科技有限公司 Method and device for synchronizing file
CN103020186A (en) * 2012-11-30 2013-04-03 广东欧珀移动通信有限公司 File searching method, device and equipment based on embedded device
CN103036956A (en) * 2012-11-30 2013-04-10 航天恒星科技有限公司 Filing system and implement method of distributed configured massive data
CN105744343A (en) * 2016-01-26 2016-07-06 珠海迈科智能科技股份有限公司 Method for rapidly scanning media files and set top box
CN106572280A (en) * 2016-10-12 2017-04-19 曙光信息产业(北京)有限公司 Method and system for scanning file

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
董赞强: "《基于网络编码的数据通信技术研究》", 31 December 2015 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108804253A (en) * 2017-05-02 2018-11-13 中国科学院高能物理研究所 A kind of concurrent job backup method for mass data backup
CN108804253B (en) * 2017-05-02 2021-08-06 中国科学院高能物理研究所 Parallel operation backup method for mass data backup
CN109542858A (en) * 2018-10-11 2019-03-29 平安科技(深圳)有限公司 Data archiving method, device based on distributed memory system, computer equipment

Similar Documents

Publication Publication Date Title
US8171474B2 (en) System and method for managing, scheduling, controlling and monitoring execution of jobs by a job scheduler utilizing a publish/subscription interface
CN101227379B (en) Method and system for implementation of data synchronization
CN105610972A (en) Clustered task dispatching system
CN106170016A (en) A kind of method and system processing high concurrent data requests
CN1846202A (en) Mail server based application record synchronization
CN107105009A (en) Job scheduling method and device based on Kubernetes system docking workflow engines
CN101383839A (en) Data distribution system based on data server and implementation method
CN111274052A (en) Data distribution method, server, and computer-readable storage medium
CN101179553A (en) Efficient order-preserving delivery method and device for concurrent messages
CN105959420A (en) Multi-client metadata synchronous update method and device
CN106648967A (en) File scanning method and system
CN107562803A (en) Data supply system and method, terminal
CN103248636A (en) Offline download system and method
CN110570158A (en) power grid standing book electronic transfer method and system and computer readable storage medium
CN113342764A (en) Data synchronization method and device among different cloud servers
CN106572280B (en) Scan the method and system of file
CN112711522B (en) Cloud testing method and system based on docker and electronic equipment
CN112099930A (en) Quantum computer cluster distributed queue scheduling method
CN102342077A (en) Method and system for the real time synthesis of interactions relating to a user
CN111709696A (en) Method and device for generating mail list based on SOA (service oriented architecture)
CN111475333A (en) Openstack-based database backup method and device
CN110928911A (en) System, method and device for processing checking request and computer readable storage medium
US20090164523A1 (en) System and method for sending messages
US20090106781A1 (en) Remote call handling methods and systems
CN207264367U (en) A kind of parallel printing system based on controller

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20170510